CLIP

Use OpenAI's CLIP model through our Serverless Cloud API

We support OpenAI's CLIP model for generating image and text embeddings, and for zero-shot similarity comparison between them, via our Serverless Cloud API. We expose three endpoints:

• /clip/embed_image, returns an embedding vector for an image

• /clip/embed_text, returns an embedding vector for a string or list of strings

• /clip/compare, returns similarity scores between a subject and a list of prompts

Embeddings can be cached and reused for tasks like classification, retrieval, clustering, and semantic search. For broader usage details, see the Inference documentation.

Code sample

Below is a code sample that compares an image against a list of text labels. Call the HTTP endpoint directly with curl, or use the inference-sdk wrapper.

HTTP (curl)

1

Get your API Key

Create a Roboflow account, find your key on the Roboflow API settings page and make it available to your shell:

export ROBOFLOW_API_KEY="your-key-here"

2

Run the model

Call the /clip/compare endpoint with curl:

curl --location 'https://serverless.roboflow.com/clip/compare' \
  --header 'Content-Type: application/json' \
  --data '{
    "api_key": "'"$ROBOFLOW_API_KEY"'",
    "subject": {"type": "url", "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"},
    "subject_type": "image",
    "prompt": ["a photo of a dog", "a photo of a cat", "a photo of a car"],
    "prompt_type": "text"
  }'

SDK (Python)

1

Get your API Key

Create a Roboflow account, find your key on the Roboflow API settings page and make it available to your shell:

export ROBOFLOW_API_KEY="your-key-here"

2

Install the dependencies

This package calls the model:

pip install -U inference-sdk supervision

3

Run the model

Run the comparison (image from here):

import os
import supervision as sv
from inference_sdk import InferenceHTTPClient

image = sv.load_image_from_url("https://media.roboflow.com/notebooks/examples/dog.jpeg")

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["ROBOFLOW_API_KEY"],
)

result = client.clip_compare(
    subject=image,
    prompt=[
        "a photo of a dog",
        "a photo of a cat",
        "a photo of a car",
    ],
    subject_type="image",
    prompt_type="text",
)

# similarity is a list of cosine similarity scores, one per prompt
print(result["similarity"])

The code above prints inference results to the terminal:

[0.2726989686489105, 0.19865083694458008, 0.20997387170791626]

Inference speed

Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.

Model	Latency (ms)
clip	3.9

Measured with embed_image on the ViT-B-16 checkpoint (image embedding only).

Set api_url to match your deployment target:

• https://serverless.roboflow.com for the Serverless Cloud API.

• http://localhost:9001 for a local Inference server.

• Your Dedicated Deployment URL for a private endpoint.

Use with Inference (self-hosted)

CLIP also runs entirely on your own hardware with the inference Python package. Loading the weights in-process avoids a network round trip per call, which matters when you are embedding large image sets for search, clustering, or dataset cleaning.

1

Install the package

pip install "inference[clip]"

2

Embed and compare locally

The Clip class exposes embed_image, embed_text, and compare. The sample embeds an image and a prompt, then scores their cosine similarity:

from inference.models import Clip
from inference.core.utils.postprocess import cosine_similarity

clip = Clip(model_id="clip/ViT-B-16")

image_embedding = clip.embed_image("https://media.roboflow.com/inference/people-walking.jpg")
text_embedding = clip.embed_text("a crowd of people walking")

print(cosine_similarity(image_embedding[0], text_embedding[0]))

The result is between 0 and 1: the higher the number, the more similar the image and the text.

Available checkpoints

model_id selects the CLIP backbone:

clip/RN50, clip/RN101, clip/RN50x4, clip/RN50x16, clip/RN50x64, clip/ViT-B-32, clip/ViT-B-16, clip/ViT-L-14, clip/ViT-L-14-336px.

The SDK methods clip_compare, get_clip_image_embeddings, and get_clip_text_embeddings accept a clip_version argument to select the same checkpoints when calling a server.

Further reading

• What is CLIP?

• Build an image search engine with CLIP and Faiss

• Analyze and classify video with CLIP