Multimodal Model Workflow

You can use multimodal Vision Language Models like GPT-4o, Claude, Gemini, and Florence-2 with Roboflow Workflows.

Roboflow Workflows supports using multimodal models in Workflows.

There are four general-purpose multimodal models that you can use for a wide range of tasks. These are:

• GPT-4o

• Claude

• Gemini

• Florence-2

These models can be used for tasks such as:

• Single- and multi-label image classification

• Zero-shot object detection

• Image captioning generation

• And more

To use a multimodal model in Workflows, you need to:

1. Add the model.

2. Choose a task type.

3. Use a built-in connector that converts the results of the model into a format understood by other Workflows block.

Let's walk through each of these steps.

Add a Multimodal Model

To use Claude, Gemini, or GPT-4o in Workflows, you need to add the block that corresponds with the model that you want to use.

For this guide, let’s walk through an example that uses Claude.

Click “Add Block” to add a block. Then, search for Claude:

A configuration panel will appear in which you can configure the Claude block.

If you use any multimodal model that calls an external API (i.e. GPT-4o), you will need to set your model API key.

You can use Claude (and Gemini and GPT-4o) for several tasks, including:

• Open prompt: Directly passes your prompt to the multimodal model.

• Text recognition (OCR): Reads characters in an image.

• Structured output generation: Returns data in a specified format.

• Single-label and multi-label classification: Returns one or more labels that represent the contents of an image.

• Visual question answering: Answer a specific question about the contents of an image.

• Captioning: Returns an image caption.

• Unprompted object detection: Returns bounding boxes that correspond with the location of objects in an image.

You can choose from these tasks using the Task Type dropdown:

Once you have chosen a task type, the output from the block will be automatically added to your Workflow outputs.

Here is an example configuration for object detection:

Add a Connector

If you want to use the output from a multimodal model in other blocks, you will need to add a connector.

You can use connectors to process:

• Classifications, and;

• Bounding boxes.

For example, you can add a connector to retrieve bounding box values from zero-shot object detection supported by Claude 3.

This connector will process the boxes in such a way that lets you use them in the Visualization blocks such as Bounding Box Visualization and Label Visualization.

When you select a VLM connector, configure it to use:

1. Your input image

2. The output from your multimodal block

3. The classes from your multimodal model

4. The name of the multimodal model you are using (in this example, anthropic-claude)

5. The task type you selected when you set up your multimodal model

Here is an example configuration of the VLM as Detector block:

You can then use the connector output with other blocks.

For example, you can use the connector output to display bounding boxes with a Bounding Box Visualization block. The Bounding Box Visualization block should be configured with your input image and the results from the VLM as Detector block:

Here is an example workflow that uses a multimodal model for object detection: