Perception Encoder

Use Meta's Perception Encoder to compute image and text embeddings on a Dedicated Deployment or self-hosted Inference

Perception Encoder is Meta's vision-language embedding model. It maps images and text into a shared embedding space for similarity search, zero-shot classification, and retrieval.

Perception Encoder is not available on the Serverless Cloud API. Run it on a Dedicated Deployment or self-hosted Inference.

We support three Perception Encoder endpoints:

• /perception_encoder/embed_image - embed an image

• /perception_encoder/embed_text - embed a string

• /perception_encoder/compare - compute similarity between an image and a list of text prompts

Code sample

1

Get your API Key

Create a Roboflow account, find your key on the Roboflow API settings page and make it available to your shell:

export ROBOFLOW_API_KEY="your-key-here"

2

Install the dependencies

These packages fetch the image and call the API:

pip install -U requests opencv-python supervision

3

Run the model

The sample below sends an image to /perception_encoder/embed_image and prints the embedding shape. Set URL to your Dedicated Deployment URL or a local Inference server.

import base64
import os
import cv2
import requests
import supervision as sv

URL = "https://your-deployment.roboflow.cloud"

image = sv.load_image_from_url("https://media.roboflow.com/notebooks/examples/dog.jpeg")

_, buffer = cv2.imencode(".jpg", image)
image_base64 = base64.b64encode(buffer).decode("utf-8")

response = requests.post(
    f"{URL}/perception_encoder/embed_image",
    json={
        "api_key": os.environ["ROBOFLOW_API_KEY"],
        "image": {"type": "base64", "value": image_base64},
    },
)
result = response.json()
embedding = result["embeddings"][0]
print(f"Embedding length: {len(embedding)}")
print(f"First values: {embedding[:5]}")

The code above prints the embedding shape to the terminal:

Embedding length: 1024
First values: [0.0545, -0.0338, -0.0355, -0.0062, 0.0154]

Inference speed

Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.

Model	Latency (ms)
perception-encoder	25.2

Measured with embed_image on the PE-Core-L14-336 checkpoint (image embedding only).

Set URL to match your deployment target:

• http://localhost:9001 for a local Inference server.

• Your Dedicated Deployment URL for a private endpoint.

Use with Inference (self-hosted)

Perception Encoder can be loaded directly with the inference package, which is the fastest path when embedding many images or video frames locally.

1

Install the package

pip install "inference[transformers]"

Use inference-gpu[transformers] on a GPU machine.

2

Embed and compare locally

from inference.core.utils.postprocess import cosine_similarity
from inference.models import PerceptionEncoder

pe = PerceptionEncoder(model_id="perception_encoder/PE-Core-B16-224")

image_embedding = pe.embed_image("https://media.roboflow.com/inference/people-walking.jpg")
text_embedding = pe.embed_text("a crowd of people walking")

print(cosine_similarity(image_embedding[0], text_embedding[0]))

Available checkpoints

model_id selects the backbone:

• perception_encoder/PE-Core-B16-224

• perception_encoder/PE-Core-L14-336

• perception_encoder/PE-Core-G14-448

Only the CLIP-style interface is supported; the language-aligned and spatially-aligned Perception Encoder variants are not available yet.

Perception Encoder uses the same API shape as CLIP: embed_image, embed_text, and compare take the same arguments and return the same response format, so code written against CLIP works with Perception Encoder by changing the model.

Use in Workflows

Perception Encoder is available in Workflows through the Perception Encoder Embedding Model block, which generates image or text embeddings without writing code.