> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/models/supported-models/florence-2.md).

# Florence 2

We support [Microsoft's Florence 2](https://huggingface.co/microsoft/Florence-2-base), a multimodal vision-language model, via our [Serverless Cloud API](https://docs.roboflow.com/deployment/roboflow-cloud/serverless-api). Florence 2 supports captioning, object detection, segmentation, and OCR through task prompts (such as `<CAPTION>`, `<OD>`, `<OCR>`, `<REFERRING_EXPRESSION_SEGMENTATION>`).

## Florence 2 pretrained aliases

Use the alias as the `model_id` in your request and the runtime resolves it to the corresponding pretrained weights.

<table data-search="false"><thead><tr><th>Alias</th></tr></thead><tbody><tr><td><code>florence-2-base</code></td></tr><tr><td><code>florence-2-large</code></td></tr></tbody></table>

## Florence 2 accuracy

Headline zero-shot metrics from the official model cards ([base](https://huggingface.co/microsoft/Florence-2-base), [large](https://huggingface.co/microsoft/Florence-2-large)):

<table data-header-hidden data-search="false"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>Benchmark</td><td><code>florence-2-base</code></td><td><code>florence-2-large</code></td></tr><tr><td>COCO Caption (CIDEr)</td><td>133.0</td><td>135.6</td></tr><tr><td>COCO detection (mAP)</td><td>34.7</td><td>37.5</td></tr><tr><td>RefCOCO (accuracy)</td><td>53.9</td><td>56.3</td></tr></tbody></table>

## Florence 2 inference speed

Latency measured with [Roboflow Inference](https://docs.roboflow.com/deployment/self-hosted/self-hosted) on 1x NVIDIA L4, batch size 1, generating exactly 128 tokens with greedy decoding from the `<CAPTION>` prompt. Latency scales with output length, so use tokens/sec to estimate other lengths.

<table data-search="false"><thead><tr><th>Alias</th><th>Latency, 128 tokens (ms)</th><th>Tokens/sec</th></tr></thead><tbody><tr><td><code>florence-2-base</code></td><td>652</td><td>198</td></tr><tr><td><code>florence-2-large</code></td><td>1120</td><td>115</td></tr></tbody></table>

## Florence 2 API

Florence 2 runs through the shared `/infer/lmm` endpoint. Call it through the HTTP endpoint directly with `curl`, or with the [`inference-sdk`](https://docs.roboflow.com/reference/inference/inference-sdk) wrapper.

{% tabs %}
{% tab title="HTTP (curl)" icon="webhook" %}
{% stepper %}
{% step %}

### Get your API Key

Create a Roboflow account, find your key on the [Roboflow API settings page](https://app.roboflow.com/settings/api) and make it available to your shell:

```bash
export ROBOFLOW_API_KEY="your-key-here"
```

{% endstep %}

{% step %}

### Run the model

Call the `/infer/lmm` endpoint with a task prompt using `curl`:

```bash
curl --location 'https://serverless.roboflow.com/infer/lmm' \
  --header 'Content-Type: application/json' \
  --data '{
    "api_key": "'"$ROBOFLOW_API_KEY"'",
    "image": {"type": "url", "value": "https://media.roboflow.com/quickstart/dog.jpeg"},
    "model_id": "florence-2-base",
    "prompt": "<CAPTION>"
  }'
```

{% endstep %}
{% endstepper %}
{% endtab %}

{% tab title="SDK (Python)" icon="python" %}
{% stepper %}
{% step %}

### Get your API Key

Create a Roboflow account, find your key on the [Roboflow API settings page](https://app.roboflow.com/settings/api) and make it available to your shell:

```bash
export ROBOFLOW_API_KEY="your-key-here"
```

{% endstep %}

{% step %}

### Install the dependencies

This package calls the model:

```bash
pip install -U inference-sdk supervision
```

{% endstep %}

{% step %}

### Run the model

Call the LMM inference endpoint with a task prompt:

```python
import os
import supervision as sv
from inference_sdk import InferenceHTTPClient

image = sv.load_image_from_url("https://media.roboflow.com/quickstart/dog.jpeg")

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["ROBOFLOW_API_KEY"],
)

result = client.infer_lmm(
    inference_input=image,
    model_id="florence-2-base",
    prompt="<CAPTION>",
)

print(result["response"])  # {'<CAPTION>': 'A man carrying a dog on his back.'}
```

<figure><img src="/files/1HuJlQhHOCgP0Milm6KA" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}
{% endtab %}
{% endtabs %}

{% hint style="info" %}
Set `api_url` to match your deployment target:

* `https://serverless.roboflow.com` for the Serverless Cloud API.
* `http://localhost:9001` for a local [Inference](https://docs.roboflow.com/deployment/self-hosted/self-hosted) server.
* Your [Dedicated Deployment](https://docs.roboflow.com/deployment/roboflow-cloud/dedicated-deployments) URL for a private endpoint.
  {% endhint %}

Swap `<CAPTION>` for any supported task prompt (for example `<DETAILED_CAPTION>`, `<OD>`, `<OCR>`, `<OPEN_VOCABULARY_DETECTION>`, `<REFERRING_EXPRESSION_SEGMENTATION>`) to switch between captioning, detection, OCR, and segmentation tasks.

For self-hosted deployment and the full list of task prompts, see the [Inference documentation](https://docs.roboflow.com/deployment/self-hosted/self-hosted).

## Florence 2 task prompts

Florence 2 switches task by prompt token. Pass one of the following as `prompt`:

| Task                              | Prompt                                                       |
| --------------------------------- | ------------------------------------------------------------ |
| Object detection                  | `<OD>`                                                       |
| Dense region captioning           | `<DENSE_REGION_CAPTION>`                                     |
| Image captioning                  | `<CAPTION>`, `<DETAILED_CAPTION>`, `<MORE_DETAILED_CAPTION>` |
| Region proposal                   | `<REGION_PROPOSAL>`                                          |
| Phrase grounding                  | `<CAPTION_TO_PHRASE_GROUNDING>`                              |
| Referring expression segmentation | `<REFERRING_EXPRESSION_SEGMENTATION>`                        |
| Region to segmentation            | `<REGION_TO_SEGMENTATION>`                                   |
| Open vocabulary detection         | `<OPEN_VOCABULARY_DETECTION>`                                |
| Region to description             | `<REGION_TO_DESCRIPTION>`                                    |
| OCR                               | `<OCR>`                                                      |
| OCR with region                   | `<OCR_WITH_REGION>`                                          |

## Run Florence 2 with self-hosted Inference

Florence 2 can also be loaded directly with the [`inference`](https://docs.roboflow.com/deployment/self-hosted/self-hosted) package.

{% stepper %}
{% step %}

### Install the package

```bash
pip install "inference[transformers]"
```

Use `inference-gpu[transformers]` on a GPU machine.
{% endstep %}

{% step %}

### Run the model

```python
from inference import get_model

model = get_model("florence-2-base", api_key="YOUR_API_KEY")

result = model.infer(
    "https://media.roboflow.com/inference/seawithdock.jpeg",
    prompt="<CAPTION>",
)

print(result[0].response)
```

Swap `<CAPTION>` for any task prompt from the table above.
{% endstep %}
{% endstepper %}

### Execution modes in Workflows

When used in a [Workflow](https://docs.roboflow.com/workflows), Florence 2 runs in one of two modes:

* **Local execution**: the model runs on your Inference server (GPU recommended).
* **Remote execution**: the model is invoked over HTTP on a remote Inference server.