CLIP

Use OpenAI's CLIP model through our Serverless Hosted API

We support OpenAI's CLIP model for generating image and text embeddings, and for zero-shot similarity comparison between them, via our Serverless Hosted API. We expose three endpoints:

• /clip/embed_image, returns an embedding vector for an image

• /clip/embed_text, returns an embedding vector for a string or list of strings

• /clip/compare, returns similarity scores between a subject and a list of prompts

Embeddings can be cached and reused for tasks like classification, retrieval, clustering, and semantic search. For broader usage details, see the Inference documentation.

Code sample

Below is a code sample that compares an image against a list of text labels using the inference-sdk. Pass Roboflow's API Key via the API_KEY env variable.

Call the /clip/compare endpoint directly with curl:

curl --location 'https://serverless.roboflow.com/clip/compare' \
  --header 'Content-Type: application/json' \
  --data '{
    "api_key": "YOUR_API_KEY",
    "subject": {"type": "url", "value": "https://media.roboflow.com/notebooks/examples/dog.jpeg"},
    "subject_type": "image",
    "prompt": ["a photo of a dog", "a photo of a cat", "a photo of a car"],
    "prompt_type": "text"
  }'

The same call through the SDK. Install it:

pip install inference-sdk

Run the comparison (image from here):

import os
import urllib.request
from inference_sdk import InferenceHTTPClient

image_url = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
image_path = "dog.jpeg"
urllib.request.urlretrieve(image_url, image_path)

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["API_KEY"],
)

result = client.clip_compare(
    subject=image_path,
    prompt=[
        "a photo of a dog",
        "a photo of a cat",
        "a photo of a car",
    ],
    subject_type="image",
    prompt_type="text",
)

# similarity is a list of cosine similarity scores, one per prompt
print(result["similarity"])

The code above prints inference results to the terminal:

[0.2726989686489105, 0.19865083694458008, 0.20997387170791626]

Set api_url to match your deployment target:

• https://serverless.roboflow.com for the Serverless Hosted API.

• http://localhost:9001 for a local Inference server.

• Your Dedicated Deployment URL for a private endpoint.