Self-Hosted Deployment

Run Roboflow models and Workflows on your own hardware with Inference, the open source computer vision deployment framework.

Inference is an open source computer vision deployment hub. It serves models and Workflows, manages video streams, and optimizes inference for CPUs and GPUs. Self-host it when you need local processing, control over latency and resources, or offline deployment. The Apache 2.0 licensed core also powers Roboflow's hosted APIs.

Self-hosting means you manage the infrastructure. If you would rather Roboflow run the servers, see Dedicated Deployments or the Serverless Cloud API, and the full comparison of options.

Pick a path

There are three ways to run models on your own hardware. Most projects use the Inference Server.

Inference Server

A Docker container that serves models and Workflows over HTTP.

Inference Library

The inference Python package for running models in your process.

Other SDKs

Run models in a web browser, on iOS, or on embedded devices.

Use the server when more than one client or language needs predictions, when you want models isolated from your application dependencies, or when you deploy to edge devices.

Run model locally

For most projects, run Inference Server in Docker and send requests with inference-sdk. The SDK is a Python HTTP client that connects your application to an Inference Server. Use Inference Library when you need to load and run models directly inside your Python process. Both paths accept the same model_id values, so you can switch between them later.

Model IDs

The model_id parameter can be:

A pre-trained model alias, for example rfdetr-small or rfdetr-large
Your own fine-tuned model from Roboflow, for example my-project/1
A Universe model, for example soccer-players-xy9vk/2

Fine-tuned models and Universe models require an API key.

Install

Start the server with the Inference CLI. It detects your hardware and pulls the right Docker image with secure defaults:

pip install inference-cli && inference server start

Then install the HTTP client:

pip install inference-sdk

For hardware requirements, per-device guides, and manual docker run commands, see Install Inference Server. The same client also works against the Serverless Cloud API and Dedicated Deployments: only api_url changes.

Run inference

from inference_sdk import InferenceHTTPClient

image = "https://media.roboflow.com/inference/people-walking.jpg"
client = InferenceHTTPClient(
    api_url="http://localhost:9001",  # your self-hosted server
    api_key="YOUR_API_KEY",
)
results = client.infer(image, model_id="rfdetr-small")

Swap api_url for https://serverless.roboflow.com to use the Serverless Cloud API instead, with no other code changes. See the Inference SDK reference for details.

Visualize results

Install Supervision:

pip install -U supervision

import supervision as sv
from inference_sdk import InferenceHTTPClient

image = sv.load_image_from_url("https://media.roboflow.com/inference/people-walking.jpg")

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="YOUR_API_KEY",
)
results = client.infer(image, model_id="rfdetr-medium")

detections = sv.Detections.from_inference(results)

annotated_image = sv.BoxAnnotator().annotate(scene=image, detections=detections)
annotated_image = sv.LabelAnnotator().annotate(scene=annotated_image, detections=detections)

sv.plot_image(annotated_image)

Install

Install the inference package into your own Python environment:

pip install inference

If you have an NVIDIA GPU, install inference-gpu instead, matching the index URL to the CUDA version installed in your OS:

pip install --extra-index-url https://download.pytorch.org/whl/cu124 inference-gpu

See Inference Library for backend extras and GPU setup details.

Run inference

from inference import get_model

image = "https://media.roboflow.com/inference/people-walking.jpg"
model = get_model(model_id="rfdetr-small")
results = model.infer(image)

get_model() downloads and caches the model weights on first use, then runs inference locally. See the Inference Python Package reference for details.

Visualize results

Install Supervision:

pip install -U supervision

import supervision as sv
from inference import get_model

image = sv.load_image_from_url("https://media.roboflow.com/inference/people-walking.jpg")

model = get_model(model_id="rfdetr-medium")
results = model.infer(image)[0]

detections = sv.Detections.from_inference(results)

annotated_image = sv.BoxAnnotator().annotate(scene=image, detections=detections)
annotated_image = sv.LabelAnnotator().annotate(scene=annotated_image, detections=detections)

sv.plot_image(annotated_image)

Be careful not to expose your API key to external users. Do not embed it in a public-facing frontend app; proxy the request through your own backend instead.

You can run a Workflow the same way, on the server or in your own process: see Deploy a Workflow.

TensorRT-optimized model packages for private models are only available on Enterprise plans when running Inference outside the Roboflow platform. Public models include TensorRT packages on all plans.

PreviousPause and Resume a Dedicated Deployment NextInference Server

Last updated 12 days ago

Was this helpful?