> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/models/supported-models/sam3.md). # SAM3 Use Meta's SAM3 model through our Serverless Cloud API We support Meta's [Segment Anything Model 3](https://github.com/facebookresearch/sam3) inferencing via our [Serverless Cloud API](https://docs.roboflow.com/deployment/roboflow-cloud/serverless-api). We offer two different SAM3 endpoints: {% hint style="info" %} Training a SAM3 model on Roboflow is available on paid [plans](https://docs.roboflow.com/platform/billing-and-plans#plans) that include [usage-based billing](https://docs.roboflow.com/platform/billing-and-plans#credits). From there, you can request access with the "Request Feature" button on the SAM3 architecture to use the feature [training flow](/models/readme.md). Fine-tuned SAM3 models cannot run on the Serverless Cloud API. Deploy them on a [Dedicated Deployment](https://docs.roboflow.com/deployment/roboflow-cloud/dedicated-deployments) or [self-hosted Inference](https://docs.roboflow.com/deployment/self-hosted/self-hosted). The hosted `sam3` endpoints on this page are unaffected. {% endhint %} * [Promptable concept segmentation](#sam3-concept-segmentation-pcs) (**PCS**), which segments every instance of a concept in the image. Concepts are described by text prompts, exemplar boxes, or both. * [Promptable visual segmentation](#sam3-visual-segmentation-pvs) (**PVS**), which interactively segments one object per request from points or a box, in the style of SAM2. Use this table to pick an endpoint:
You haveYou wantUse
A text description (ex: "person")Masks for every matching instance/sam3/concept_segment
A box around one example objectMasks for every similar instance/sam3/concept_segment
Text plus example boxes to include or exclude objectsMasks for every matching instance/sam3/concept_segment
A click or a box on one specific objectA mask for that object only/sam3/visual_segment
Send your [API key](https://app.roboflow.com/settings/api) in an `Authorization: Bearer` header on every request. The legacy `api_key` query parameter still works but is not recommended. ## SAM3 Concept Segmentation (PCS) `POST https://serverless.roboflow.com/sam3/concept_segment` Each entry in `prompts` describes one concept. The response contains one `prompt_results` entry per prompt, each holding every instance found. Requests accept at most 16 prompts. ### Text prompts ```python import os import requests payload = { "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"}, "prompts": [ {"type": "text", "text": "person"}, {"type": "text", "text": "backpack"}, ], "output_prob_thresh": 0.5, "format": "polygon", # or "rle" } response = requests.post( "https://serverless.roboflow.com/sam3/concept_segment", headers={"Authorization": f"Bearer {os.environ['ROBOFLOW_API_KEY']}"}, json=payload, ) for prompt_result in response.json()["prompt_results"]: print(prompt_result["echo"], len(prompt_result["predictions"]), "instances") ``` Images can also be sent inline as `{"type": "base64", "value": ""}`. ### Exemplar box prompts Instead of text, you can prompt with an exemplar: a box around one example object. The model finds every instance that matches the example, not just the boxed object. ```python payload = { "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"}, "prompts": [ { "type": "visual", "boxes": [{"x": 1409, "y": 705, "width": 112, "height": 183}], "box_labels": [1], } ], "output_prob_thresh": 0.5, "format": "polygon", } ``` Boxes use absolute pixel coordinates. Two formats are accepted: * `{"x": ..., "y": ..., "width": ..., "height": ...}` where `x`, `y` is the top-left corner * `{"x0": ..., "y0": ..., "x1": ..., "y1": ...}` for explicit corners `box_labels` is required when `boxes` is set and must have one entry per box: `1` marks a positive exemplar (find objects like this), `0` marks a negative exemplar (exclude objects like this). ### Combined text and exemplar prompts A single prompt can carry both text and exemplar boxes. This is useful for narrowing a text concept with visual examples, or excluding lookalikes with negative exemplars: ```python payload = { "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"}, "prompts": [ { "type": "visual", "text": "person", "boxes": [ {"x": 1409, "y": 705, "width": 112, "height": 183}, {"x": 1216, "y": 496, "width": 124, "height": 184}, ], "box_labels": [1, 0], } ], "output_prob_thresh": 0.5, "format": "polygon", } ``` Here the model segments people matching the first (positive) exemplar while suppressing instances similar to the second (negative) exemplar. ## SAM3 Visual Segmentation (PVS) `POST https://serverless.roboflow.com/sam3/visual_segment` PVS segments one specific object indicated by clicks or a box. Use it for interactive, human-in-the-loop mask refinement; use PCS when you want every instance of a concept. ```python import os import requests payload = { "image": {"type": "url", "value": "https://media.roboflow.com/inference/people-walking.jpg"}, "prompts": { "prompts": [ { "points": [{"x": 1465, "y": 796, "positive": True}], "box": {"x": 1465, "y": 796, "width": 112, "height": 183}, } ] }, "multimask_output": False, "format": "json", } response = requests.post( "https://serverless.roboflow.com/sam3/visual_segment", headers={"Authorization": f"Bearer {os.environ['ROBOFLOW_API_KEY']}"}, json=payload, ) prediction = response.json()["predictions"][0] print(prediction["confidence"], len(prediction["masks"]), "polygons") ``` A prompt can contain `points`, a `box`, or both: * `points` are absolute pixel coordinates. `"positive": true` includes the clicked region, `false` excludes it. Add more points to refine the mask. * `box` uses center-anchored coordinates: `x`, `y` is the box center, unlike PCS boxes which are top-left anchored. The response contains the single highest-confidence mask for the prompt. `multimask_output` controls how many internal mask proposals the model generates (three when true), but the best proposal is always selected for the response. {% hint style="warning" %} Send one prompt per request. Multiple prompts in one PVS request currently return only one prediction. {% endhint %} For an interactive demo using OpenCV, see this [GitHub Gist](https://gist.github.com/Erol444/4cbc33c6ac52d83c63f6f9d86ca8a7a4), which was used in this video: {% embed url="" %} ## SAM3 inference speed Latency measured with [Roboflow Inference](https://docs.roboflow.com/deployment/self-hosted/self-hosted) on 1x NVIDIA L4, batch size 1, mean after warmup.
ModelLatency (ms)
sam3251.4
Measured with concept segmentation from a single text prompt. ## SAM3 API endpoints {% openapi src="/files/paTKd1Kx4MYL9Z65Fhyv" path="/sam3/concept\_segment" method="post" %} [sam3.json](https://3448603185-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcmo9mhfIjYrvFFy1U7dk%2Fuploads%2Fgit-blob-c718b36dd4197f87a2e72c28664c6d11516aefc8%2Fsam3.json?alt=media) {% endopenapi %} {% openapi src="/files/paTKd1Kx4MYL9Z65Fhyv" path="/sam3/visual\_segment" method="post" %} [sam3.json](https://3448603185-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcmo9mhfIjYrvFFy1U7dk%2Fuploads%2Fgit-blob-c718b36dd4197f87a2e72c28664c6d11516aefc8%2Fsam3.json?alt=media) {% endopenapi %} ## Run SAM3 with self-hosted Inference SAM3 can also run on your own hardware, either loaded in-process with the [`inference`](https://docs.roboflow.com/deployment/self-hosted/self-hosted) package or served from a GPU container. ### Run in Docker ```bash docker run -it --rm -p 9001:9001 --gpus=all roboflow/inference-server:latest ``` The server exposes the same `/sam3/concept_segment` and `/sam3/visual_segment` endpoints documented above at `http://localhost:9001`. ### Load the model in Python ```bash pip install "inference-gpu[sam3]" ``` ```python import os os.environ["API_KEY"] = "YOUR_API_KEY" from inference.core.entities.requests.sam3 import Sam3Prompt from inference.models.sam3 import SegmentAnything3 model = SegmentAnything3(model_id="sam3/sam3_final") prompts = [ # Segment every instance of a concept Sam3Prompt(type="text", text="person"), # Box one example object and segment every similar instance. # box_labels: 1 = positive exemplar, 0 = negative exemplar. Sam3Prompt( type="visual", boxes=[Sam3Prompt.Box(x=1409, y=705, width=112, height=183)], box_labels=[1], ), ] response = model.segment_image( image="path/to/your/image.jpg", prompts=prompts, output_prob_thresh=0.5, format="polygon", # or "rle", "json" ) for prompt_result in response.prompt_results: print(prompt_result.echo.text, len(prompt_result.predictions), "instances") ``` Weights download automatically on first use. ### Interactive segmentation in Python `Sam3ForInteractiveImageSegmentation` implements the SAM2-style point and box interface, for human-in-the-loop mask refinement: ```python from inference.models.sam3 import Sam3ForInteractiveImageSegmentation model = Sam3ForInteractiveImageSegmentation(model_id="sam3/sam3_final") embedding, img_shape, image_id = model.embed_image(image="path/to/image.jpg") masks, scores, logits = model.segment_image( image_id=image_id, prompts={"points": [{"x": 500, "y": 400, "positive": True}]}, ) ``` ## Use SAM3 in Workflows Two SAM3 image blocks are available in [Workflows](https://docs.roboflow.com/workflows): * **SAM 3** runs concept segmentation. Enter the classes you want in `class_names` (for example `["person", "vehicle"]`) and the block outputs instance segmentation predictions that other steps can consume. * **SAM 3 Interactive** runs promptable visual segmentation. Supply labeled points (kind `labeled_points`), for example `[{"x": 320, "y": 240, "positive": true}]`, and optionally connect detections from another model to the `boxes` field. Each box becomes a separate prompt, and its class name is forwarded to the predicted mask. ### Video tracking The **SAM3 Video Tracker** block (`roboflow_core/sam3_video@v1`) runs SAM3's streaming concept tracker frame by frame. You provide concepts as text in `class_names`, and the model runs fused detection and tracking on every frame. Objects matching a concept keep a stable `tracker_id`, and, unlike detector-seeded tracking, objects that enter the scene mid-stream are picked up automatically with no re-prompting and no upstream detection model. Each mask carries the concept it matched as its class name and the model's detection score as its confidence (filter with `threshold`, default `0.5`). * **Stateful and local-only.** One tracking session is kept per `video_metadata.video_identifier`. The block requires `WORKFLOWS_STEP_EXECUTION_MODE=local`, a GPU, and a persistent WebRTC session. * **No prompt scheduling.** Concept prompts are registered once per session; the session is re-seeded only when the stream restarts or `class_names` changes. For detector-driven (box-prompted) video tracking, use the SAM2 Video Tracker block on the [SAM2 page](/models/supported-models/sam2.md), which also accepts `sam3trackervideo` as `model_id`. * **Model.** `model_id` defaults to `sam3video`, the HuggingFace transformers port of SAM3 video, which exposes the frame-by-frame streaming interface. The native `sam3` package's video predictor requires the whole video upfront and cannot be used for live streams. ```python from inference_sdk import InferenceHTTPClient, InferenceConfiguration from inference_sdk.webrtc import StreamConfig, VideoFileSource WORKFLOW = { "version": "1.0", "inputs": [{"type": "InferenceImage", "name": "image"}], "steps": [ { "type": "roboflow_core/sam3_video@v1", "name": "tracker", "images": "$inputs.image", "class_names": ["person", "forklift"], "threshold": 0.5, }, ], "outputs": [ { "type": "JsonField", "name": "predictions", "selector": "$steps.tracker.predictions", } ], } client = InferenceHTTPClient( api_url="http://localhost:9001", api_key="YOUR_API_KEY", ).configure(InferenceConfiguration(api_key_transport="header")) session = client.webrtc.stream( source=VideoFileSource("path/to/video.mp4"), workflow=WORKFLOW, config=StreamConfig(data_output=["predictions"]), ) @session.on_data("predictions") def handle_predictions(predictions, metadata): print(predictions) session.run() ``` ## SAM3-3D (beta) SAM3-3D turns a 2D image plus masks into 3D assets: meshes and Gaussian splats. {% hint style="warning" %} SAM3-3D is in beta. It is available only when the `SAM3_3D_OBJECTS_ENABLED` flag is set, requires a GPU with 32 GB or more of VRAM, and runs through the `inference` package or a local Inference server (it is not on the Serverless Cloud API). {% endhint %} Install the dependencies (Python 3.10 recommended): ```bash pip install --no-cache-dir --no-build-isolation -r requirements/requirements.sam3_3d.txt ``` Or build and run the 3D-enabled GPU container: ```bash docker build -t roboflow/roboflow-inference-server-gpu:dev -f docker/dockerfiles/Dockerfile.onnx.gpu.3d . docker run --gpus all -p 9001:9001 roboflow/roboflow-inference-server-gpu:dev ``` **Input.** An RGB image plus `mask_input`, which defines the object regions. Masks are accepted as binary arrays (`(H, W)` or `(N, H, W)`), COCO flat polygons, point-pair polygons, RLE dicts, or an `sv.Detections` object from SAM2 or another segmentation model. **Output.** `mesh_glb` (combined scene mesh, GLB), `gaussian_ply` (combined Gaussian splat, PLY), `objects` (per-object `mesh_glb`, `gaussian_ply`, and `metadata` with rotation, translation, and scale), and `time`. ```python import os os.environ["SAM3_3D_OBJECTS_ENABLED"] = "true" os.environ["SPARSE_ATTN_BACKEND"] = "flash_attn" os.environ["ATTN_BACKEND"] = "flash_attn" from inference import get_model from inference.core.entities.requests.sam3_3d import Sam3_3D_Objects_InferenceRequest model = get_model("sam3-3d-objects", api_key="YOUR_API_KEY") request = Sam3_3D_Objects_InferenceRequest( image={"type": "file", "value": "image.jpg"}, mask_input=mask_polygons, # polygons, binary masks, or RLE ) response = model.infer_from_request(request) if response.mesh_glb is not None: with open("out_mesh.glb", "wb") as f: f.write(response.mesh_glb) for index, obj in enumerate(response.objects): if obj.gaussian_ply is not None: with open(f"out_object_{index}.ply", "wb") as f: f.write(obj.gaussian_ply) ``` Setting `SPARSE_ATTN_BACKEND` and `ATTN_BACKEND` to `flash_attn` speeds up the pipeline. In Workflows, SAM3-3D supports local execution and remote execution through the `sam3_3d_infer()` client method or the `/sam3_3d/infer` endpoint. ## See also * [SAM2](/models/supported-models/sam2.md) - point and box prompted segmentation, plus detector-seeded video tracking. * [Segment Anything (SAM)](/models/supported-models/sam.md) - the original single-object model.