Inference Library

Run models directly in your own Python process with the inference package, with no server and no HTTP hop.

The inference Python package loads models and runs them inside your own process. There is no container to start and no HTTP request between your code and the model, which makes it the lowest-latency way to self-host and the simplest to embed in an existing Python application.

Use it when your application is Python and runs on the same machine as the model. If several clients, languages, or video streams need predictions, or you want models isolated from your application's dependencies, run the Inference Server instead. Both accept the same model_id values, so switching later is a small change.

Install

pip install inference

If you have an NVIDIA GPU, install inference-gpu instead:

pip install --extra-index-url https://download.pytorch.org/whl/cu124 inference-gpu

Match --extra-index-url to the CUDA version installed in your OS: https://download.pytorch.org/whl/cu<major><minor>, for instance https://download.pytorch.org/whl/cu130 for CUDA 13.0. GPU installation requires CUDA in the OS. See the Linux or Windows CUDA installation guide if your environment lacks the dependencies.

Starting with inference 1.2.0, the new inference engine (inference-models) is the default. It supports several model backends, including TensorRT, and picks the fastest one available for your hardware. inference installs what torch and onnx models need; other backends come from package extras:

pip install inference-models[trt10]

On Windows, CUDA setup has extra steps: see Install Bare Metal Inference GPU on Windows.

Run a model

from inference import get_model

image = "https://media.roboflow.com/inference/people-walking.jpg"
model = get_model(model_id="rfdetr-small")
results = model.infer(image)

get_model() downloads and caches the model weights on first use, then runs inference locally. The model_id can be a pre-trained alias, your own fine-tuned model, or a Universe model: see model IDs. Fine-tuned and Universe models require an API key.

Visualize results

Install Supervision to annotate predictions:

pip install -U supervision

import supervision as sv
from inference import get_model

image = sv.load_image_from_url("https://media.roboflow.com/inference/people-walking.jpg")

model = get_model(model_id="rfdetr-medium")
results = model.infer(image)[0]

detections = sv.Detections.from_inference(results)

annotated_image = sv.BoxAnnotator().annotate(scene=image, detections=detections)
annotated_image = sv.LabelAnnotator().annotate(scene=annotated_image, detections=detections)

sv.plot_image(annotated_image)

Video and Workflows

For most video applications, run an Inference Server and stream a model or Workflow to it with the Inference SDK WebRTC client.

If you intentionally run the Inference Library inside your Python process, InferencePipeline can process a webcam, RTSP camera, or video file without a server. This direct-library API gives your process access to frames, custom inference logic, and sinks. See Inference Pipeline.

Going further

• Inference Python Package reference - the full API surface.

• Native Python API - load models and run Workflows without the server.

• Model Weights Download - cache weights for offline and air-gapped hosts.

• Inference Benchmarks - measured throughput by model and hardware.