Run a Model Locally

Run Roboflow models on your own hardware with a Docker inference server, in five minutes.

Roboflow Inference is the free, open-source engine behind Roboflow's hosted models. You can run the same models on your own computer: start the inference server in a Docker container, then send it images over HTTP or with the Python Inference SDK. Running it yourself keeps your images on your own machine and gives you full control over your setup. Local inference does not use credits.

Run a model on your machine

HTTP (Python)

1

Start the inference server

inference server start launches the Inference Server in a Docker container and picks the right version for your hardware: CPU, NVIDIA GPU, or Jetson. Docker must be installed and running.

pip install inference-cli && inference server start

The server listens on http://localhost:9001. For device-specific setup and troubleshooting, see the Inference install guide.

2

Install the dependencies

supervision draws the results and brings in cv2 and numpy:

pip install -U supervision

3

Run the model

Send the image to your local server and run a ready-made RF-DETR model. Ready-made models like this one don't need an API key:

import base64
import cv2
import requests
import supervision as sv

image = sv.load_image_from_url("https://media.roboflow.com/quickstart/cars.jpg")
image_b64 = base64.b64encode(content).decode("utf-8")

result = requests.post(
    "http://localhost:9001/coco/38",  # coco/38 == rfdetr-nano
    data=image_b64,  # raw base64 in the body
    headers={"Content-Type": "application/x-www-form-urlencoded"},
).json()
detections = sv.Detections.from_inference(result)
print(f"Found {len(detections)} objects")

annotated = sv.BoxAnnotator().annotate(image.copy(), detections)
annotated = sv.LabelAnnotator().annotate(annotated, detections)
cv2.imwrite("output.jpg", annotated)

SDK (Python)

1

Start the inference server

inference server start launches the Inference Server in a Docker container and picks the right version for your hardware: CPU, NVIDIA GPU, or Jetson. Docker must be installed and running.

pip install inference-cli && inference server start

The server listens on http://localhost:9001. For device-specific setup and troubleshooting, see the Inference install guide.

2

Install the SDK

supervision draws the results and brings in cv2 and numpy:

pip install -U inference-sdk supervision

3

Run the model

Point the client at your local server and run a ready-made RF-DETR model. Ready-made models like this one don't need an API key:

import cv2
import supervision as sv
from inference_sdk import InferenceHTTPClient

image = sv.load_image_from_url("https://media.roboflow.com/quickstart/cars.jpg")

client = InferenceHTTPClient(api_url="http://localhost:9001")
result = client.infer(image, model_id="rfdetr-nano")

detections = sv.Detections.from_inference(result)
print(f"Found {len(detections)} objects")

annotated = sv.BoxAnnotator().annotate(image.copy(), detections)
annotated = sv.LabelAnnotator().annotate(annotated, detections)
cv2.imwrite("output.jpg", annotated)

The first request downloads the model, so it can take a few seconds. Later requests are fast.

RF-DETR detections from the local inference server

Process a video

To run a model on a video, open a WebRTC session with the server and receive processed frames plus predictions as the video plays. If your computer cannot keep up, the server skips frames to stay in real time. The same SDK can stream one model or a Workflow.

You can also run a model on a video over HTTP by sending one frame at a time, the same way as the image example above (no streaming add-on needed). This is slower: each frame has to be sent, processed, and drawn before the next one starts, so there's a lot of waiting in between. The SDK streams the whole video and works on several frames at once, so it keeps up much better.

1

Install the SDK

Streaming video needs the webrtc add-on:

pip install "inference-sdk[webrtc]"

2

Run the model on a video

import cv2
from inference_sdk import InferenceHTTPClient
from inference_sdk.webrtc import VideoFileSource
import supervision as sv

client = InferenceHTTPClient.init(api_url="http://localhost:9001")

# Download+cache video
source = VideoFileSource("https://media.roboflow.com/quickstart/cars.mp4")
# Uses webrtc for optimized video streaming
session = client.webrtc.stream(
    source=source,
    model_id="rfdetr-nano"
)

# on_frame runs on the main thread, so cv2.imshow is safe
@session.on_frame
def show(frame, data):
    det = sv.Detections.from_inference(data)
    img = sv.BoxAnnotator().annotate(frame, det)
    img = sv.LabelAnnotator().annotate(img, det)
    cv2.imshow("RF-DETR", img)
    if cv2.waitKey(1) == ord("q"):
        session.close()

session.run()
cv2.destroyAllWindows()

Build richer pipelines (tracking, filtering, zones, notifications) visually in the Workflows editor, then run them through the same WebRTC client. See Video processing with Workflows.

Other options

• Prefer a managed endpoint without running hardware? Use a Dedicated Deployment or the Serverless Cloud API.

• Need on-premises, air-gapped, or Kubernetes deployment? See Enterprise Deployment.

• Compare every option in Deploy a Model or Workflow.