SAM3
Use Meta's SAM3 model through our Serverless Cloud API
We support Meta's Segment Anything Model 3 inferencing via our Serverless Cloud API. We offer two different SAM3 endpoints:
Promptable concept segmentation (PCS), which segments every instance of a concept in the image. Concepts are described by text prompts, exemplar boxes, or both.
Promptable visual segmentation (PVS), which interactively segments one object per request from points or a box, in the style of SAM2.
Use this table to pick an endpoint:
A text description (ex: "person")
Masks for every matching instance
/sam3/concept_segment
A box around one example object
Masks for every similar instance
/sam3/concept_segment
Text plus example boxes to include or exclude objects
Masks for every matching instance
/sam3/concept_segment
A click or a box on one specific object
A mask for that object only
/sam3/visual_segment
Send your API key in an Authorization: Bearer header on every request. The legacy api_key query parameter still works but is not recommended.
SAM3 Concept Segmentation (PCS)
POST https://serverless.roboflow.com/sam3/concept_segment
Each entry in prompts describes one concept. The response contains one prompt_results entry per prompt, each holding every instance found. Requests accept at most 16 prompts.
Text prompts
import os
import requests
payload = {
"image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"},
"prompts": [
{"type": "text", "text": "person"},
{"type": "text", "text": "backpack"},
],
"output_prob_thresh": 0.5,
"format": "polygon", # or "rle"
}
response = requests.post(
"https://serverless.roboflow.com/sam3/concept_segment",
headers={"Authorization": f"Bearer {os.environ['ROBOFLOW_API_KEY']}"},
json=payload,
)
for prompt_result in response.json()["prompt_results"]:
print(prompt_result["echo"], len(prompt_result["predictions"]), "instances")Images can also be sent inline as {"type": "base64", "value": "<BASE64_IMAGE>"}.
Exemplar box prompts
Instead of text, you can prompt with an exemplar: a box around one example object. The model finds every instance that matches the example, not just the boxed object.
Boxes use absolute pixel coordinates. Two formats are accepted:
{"x": ..., "y": ..., "width": ..., "height": ...}wherex,yis the top-left corner{"x0": ..., "y0": ..., "x1": ..., "y1": ...}for explicit corners
box_labels is required when boxes is set and must have one entry per box: 1 marks a positive exemplar (find objects like this), 0 marks a negative exemplar (exclude objects like this).
Combined text and exemplar prompts
A single prompt can carry both text and exemplar boxes. This is useful for narrowing a text concept with visual examples, or excluding lookalikes with negative exemplars:
Here the model segments people matching the first (positive) exemplar while suppressing instances similar to the second (negative) exemplar.
SAM3 Visual Segmentation (PVS)
POST https://serverless.roboflow.com/sam3/visual_segment
PVS segments one specific object indicated by clicks or a box. Use it for interactive, human-in-the-loop mask refinement; use PCS when you want every instance of a concept.
A prompt can contain points, a box, or both:
pointsare absolute pixel coordinates."positive": trueincludes the clicked region,falseexcludes it. Add more points to refine the mask.boxuses center-anchored coordinates:x,yis the box center, unlike PCS boxes which are top-left anchored.
The response contains the single highest-confidence mask for the prompt. multimask_output controls how many internal mask proposals the model generates (three when true), but the best proposal is always selected for the response.
For an interactive demo using OpenCV, see this GitHub Gist, which was used in this video:
SAM3 inference speed
Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.
sam3
251.4
Measured with concept segmentation from a single text prompt.
SAM3 API endpoints
Run the SAM3 PCS (promptable concept segmentation) to generate segmentations for image data.
Roboflow API Key that will be passed to the model during initialization for artifact retrieval
The source of the inference request
The detailed source information of the inference request
Roboflow API Key that will be passed to the model during initialization for artifact retrieval
trueInternal stream-pipeline frame pairing id. Not part of the public API.
If true, disables model monitoring for this request
falseThe model ID of SAM3. Use 'sam3/sam3_final' to target the generic base model.
sam3/sam3_finalOne of 'polygon', 'rle'
polygonOptional ID for caching embeddings.
Score threshold for outputs.
0.5IoU threshold for cross-prompt NMS. If None, NMS is disabled. Must be in [0.0, 1.0] when set.
Successful Response
Unique identifier of inference
The frame id of the image used in inference if the input was a video
The time in seconds it took to produce the segmentation including preprocessing
Validation Error
POST /sam3/concept_segment HTTP/1.1
Host: serverless.roboflow.com
Content-Type: application/json
Accept: */*
Content-Length: 474
{
"id": "text",
"api_key": "text",
"usage_billable": true,
"start": 1,
"source": "text",
"source_info": "text",
"stream_pipeline_context_id": "text",
"disable_model_monitoring": false,
"model_id": "sam3/sam3_final",
"format": "polygon",
"image": {
"type": "url",
"value": "http://www.example-image-url.com"
},
"image_id": "text",
"output_prob_thresh": 0.5,
"prompts": [
{
"type": "text",
"text": "text",
"output_prob_thresh": 1,
"boxes": [
{
"x": 1,
"y": 1,
"width": 1,
"height": 1
}
],
"box_labels": [
1
]
}
],
"nms_iou_threshold": 1
}{
"inference_id": "text",
"frame_id": 1,
"time": 1,
"resolved_model": {
"model_id": "text",
"model_package_id": "text",
"backend": "text",
"quantization": "text"
},
"prompt_results": [
{
"prompt_index": 1,
"echo": {
"prompt_index": 1,
"type": "text",
"text": "text",
"num_boxes": 1
},
"predictions": [
{
"masks": [
[
[
1
]
]
],
"confidence": 1,
"format": "polygon"
}
]
}
]
}Run the SAM3 PVS (promptable visual segmentation) to generate segmentations for image data.
Roboflow API Key that will be passed to the model during initialization for artifact retrieval
The source of the inference request
The detailed source information of the inference request
SAM segmentation request.
Attributes: format (Optional[str]): The format of the response. image (InferenceRequestImage): The image to be segmented. image_id (Optional[str]): The ID of the image to be segmented used to retrieve cached embeddings. point_coords (Optional[List[List[float]]]): The coordinates of the interactive points used during decoding. point_labels (Optional[List[float]]): The labels of the interactive points used during decoding.
Roboflow API Key that will be passed to the model during initialization for artifact retrieval
trueInternal stream-pipeline frame pairing id. Not part of the public API.
If true, disables model monitoring for this request
falseThe version ID of SAM to be used for this request. Must be one of hiera_tiny, hiera_small, hiera_large, hiera_b_plus
hiera_largeExample: hiera_largeThe format of the response. Must be one of 'json', 'rle', or 'binary'. If binary, masks are returned as binary numpy arrays. If json, masks are converted to polygons. If rle, masks are converted to RLE format.
jsonExample: jsonThe ID of the image to be segmented used to retrieve cached embeddings. If an embedding is cached, it will be used instead of generating a new embedding. If no embedding is cached, a new embedding will be generated and cached.
image_idIf true, the model will return three masks. For ambiguous input prompts (such as a single click), this will often produce better masks than a single prediction. If only a single mask is needed, the model's predicted quality score can be used to select the best mask. For non-ambiguous prompts, such as multiple input prompts, multimask_output=False can give better results.
trueExample: trueIf True, saves the low-resolution logits to the cache for potential future use. This can speed up subsequent requests with similar prompts on the same image. This feature is ignored if DISABLE_SAM2_LOGITS_CACHE env variable is set True
falseIf True, attempts to load previously cached low-resolution logits for the given image and prompt set. This can significantly speed up inference when making multiple similar requests on the same image. This feature is ignored if DISABLE_SAM2_LOGITS_CACHE env variable is set True
falseSuccessful Response
Unique identifier of inference
The frame id of the image used in inference if the input was a video
The time in seconds it took to produce the segmentation including preprocessing
Validation Error
POST /sam3/visual_segment HTTP/1.1
Host: serverless.roboflow.com
Content-Type: application/json
Accept: */*
Content-Length: 479
{
"id": "text",
"api_key": "text",
"usage_billable": true,
"start": 1,
"source": "text",
"source_info": "text",
"stream_pipeline_context_id": "text",
"disable_model_monitoring": false,
"sam2_version_id": "hiera_large",
"model_id": "text",
"format": "json",
"image": {
"type": "url",
"value": "http://www.example-image-url.com"
},
"image_id": "image_id",
"prompts": [
{
"prompts": [
{
"points": [
{
"positive": true,
"x": 100,
"y": 100
}
]
}
]
}
],
"multimask_output": true,
"save_logits_to_cache": false,
"load_logits_from_cache": false
}{
"inference_id": "text",
"frame_id": 1,
"time": 1,
"resolved_model": {
"model_id": "text",
"model_package_id": "text",
"backend": "text",
"quantization": "text"
},
"predictions": [
{
"masks": [
[
[
1
]
]
],
"confidence": 1,
"format": "polygon"
}
]
}Run SAM3 with self-hosted Inference
SAM3 can also run on your own hardware, either loaded in-process with the inference package or served from a GPU container.
Run in Docker
The server exposes the same /sam3/concept_segment and /sam3/visual_segment endpoints documented above at http://localhost:9001.
Load the model in Python
Weights download automatically on first use.
Interactive segmentation in Python
Sam3ForInteractiveImageSegmentation implements the SAM2-style point and box interface, for human-in-the-loop mask refinement:
Use SAM3 in Workflows
Two SAM3 image blocks are available in Workflows:
SAM 3 runs concept segmentation. Enter the classes you want in
class_names(for example["person", "vehicle"]) and the block outputs instance segmentation predictions that other steps can consume.SAM 3 Interactive runs promptable visual segmentation. Supply labeled points (kind
labeled_points), for example[{"x": 320, "y": 240, "positive": true}], and optionally connect detections from another model to theboxesfield. Each box becomes a separate prompt, and its class name is forwarded to the predicted mask.
Video tracking
The SAM3 Video Tracker block (roboflow_core/sam3_video@v1) runs SAM3's streaming concept tracker frame by frame. You provide concepts as text in class_names, and the model runs fused detection and tracking on every frame. Objects matching a concept keep a stable tracker_id, and, unlike detector-seeded tracking, objects that enter the scene mid-stream are picked up automatically with no re-prompting and no upstream detection model. Each mask carries the concept it matched as its class name and the model's detection score as its confidence (filter with threshold, default 0.5).
Stateful and local-only. One tracking session is kept per
video_metadata.video_identifier. The block requiresWORKFLOWS_STEP_EXECUTION_MODE=local, a GPU, and a persistent WebRTC session.No prompt scheduling. Concept prompts are registered once per session; the session is re-seeded only when the stream restarts or
class_nameschanges. For detector-driven (box-prompted) video tracking, use the SAM2 Video Tracker block on the SAM2 page, which also acceptssam3trackervideoasmodel_id.Model.
model_iddefaults tosam3video, the HuggingFace transformers port of SAM3 video, which exposes the frame-by-frame streaming interface. The nativesam3package's video predictor requires the whole video upfront and cannot be used for live streams.
SAM3-3D (beta)
SAM3-3D turns a 2D image plus masks into 3D assets: meshes and Gaussian splats.
Install the dependencies (Python 3.10 recommended):
Or build and run the 3D-enabled GPU container:
Input. An RGB image plus mask_input, which defines the object regions. Masks are accepted as binary arrays ((H, W) or (N, H, W)), COCO flat polygons, point-pair polygons, RLE dicts, or an sv.Detections object from SAM2 or another segmentation model.
Output. mesh_glb (combined scene mesh, GLB), gaussian_ply (combined Gaussian splat, PLY), objects (per-object mesh_glb, gaussian_ply, and metadata with rotation, translation, and scale), and time.
Setting SPARSE_ATTN_BACKEND and ATTN_BACKEND to flash_attn speeds up the pipeline. In Workflows, SAM3-3D supports local execution and remote execution through the sam3_3d_infer() client method or the /sam3_3d/infer endpoint.
See also
SAM2 - point and box prompted segmentation, plus detector-seeded video tracking.
Segment Anything (SAM) - the original single-object model.
Last updated
Was this helpful?