> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/reference/inference/inference-python.md).

# Inference Python Package

The `inference` Python package is the core open source library that powers Roboflow's computer vision deployment stack. It provides model loading, pre/post-processing, GPU/CPU optimization, and [Workflows](https://docs.roboflow.com/workflows) execution, callable directly from Python.

The [Inference Server](https://docs.roboflow.com/deployment/self-hosted/inference-server) wraps this package and exposes it over HTTP (distributed as a Docker image with all dependencies installed), but you can also use `inference` directly in your own scripts and applications.

## How the pieces fit together

Inference has several components that work together to serve computer vision models:

* [**inference**](/reference/inference/inference-python.md) - core Python package for model loading, inference, and Workflows execution.
* [**inference-sdk**](/reference/inference/inference-sdk.md) - lightweight Python client for communicating with an Inference Server over HTTP.
* [**inference-cli**](/reference/inference/inference-cli.md) - command-line tool for managing the Inference Server and running common tasks.
* [**Inference Server**](https://docs.roboflow.com/deployment/self-hosted/inference-server) - HTTP server (Docker) that wraps the `inference` package as a REST API.

For how these pieces behave at runtime (request routing, parallelization, microservice and appliance patterns), see [Inference Architecture](https://docs.roboflow.com/deployment/self-hosted/inference-server/architecture).

## Multi-backend support

Inference 1.0 supports multiple inference runtime backends: ONNX, TensorRT, Hugging Face, and PyTorch. It automatically selects the fastest available backend for your hardware. For example, if you have an NVIDIA GPU or are running on a Jetson device and a TensorRT engine is available for the model on your platform, Inference uses TensorRT by default.

## Installation

We recommend using a [Python virtual environment (venv)](https://docs.python.org/3/tutorial/venv.html) to isolate the dependencies of Inference.

To install Inference via pip:

```bash
pip install inference
```

If you have an NVIDIA GPU, you can accelerate your inference with:

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu124 inference-gpu
# please adjust the --extra-index-url to the CUDA version installed in your OS
# https://download.pytorch.org/whl/cu<major><minor>, for instance https://download.pytorch.org/whl/cu130 for CUDA 13.0
```

## Quick example

You can run inference either against an Inference Server over HTTP (with `inference-sdk`) or directly in your process (with the native `inference` package). The native `get_model()` call loads the model into your script and returns an object you can call `.infer()` on; the HTTP client sends the image to a server that does the same.

{% tabs %}
{% tab title="inference (native)" %}

```python
from inference import get_model

model = get_model(model_id="rfdetr-small")
results = model.infer("https://media.roboflow.com/inference/people-walking.jpg")
```

{% endtab %}

{% tab title="inference-sdk (HTTP client)" %}

```python
from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    # api_url="http://localhost:9001",  # for self-hosted
    api_url="https://serverless.roboflow.com",
    api_key="ROBOFLOW_API_KEY",
)
results = client.infer(
    "https://media.roboflow.com/inference/people-walking.jpg",
    model_id="rfdetr-small",
)
```

{% endtab %}
{% endtabs %}

To use models that require an API key, set the `ROBOFLOW_API_KEY` environment variable or pass it directly:

```python
model = get_model(model_id="my-project/1", api_key="ROBOFLOW_API_KEY")
```

See the [Native Python API](/reference/inference/inference-python/native-python-api.md) page for a more detailed walkthrough with visualization, and [Run a Model](https://docs.roboflow.com/deployment/self-hosted/self-hosted#run-a-model) for the server-based path.

## Inference Pipeline

`InferencePipeline` runs video inside the same Python process as the `inference` package. Use it only when you choose direct Inference Library execution and need access to its custom logic or sink interfaces. Applications that use an Inference Server or Serverless should use [WebRTC Streaming](/reference/inference/inference-sdk/webrtc.md).

```python
from inference import InferencePipeline
from inference.core.interfaces.stream.sinks import render_boxes

pipeline = InferencePipeline.init(
    model_id="rfdetr-large",
    video_reference="https://storage.googleapis.com/com-roboflow-marketing/inference/people-walking.mp4",
    on_prediction=render_boxes,
    api_key="ROBOFLOW_API_KEY",
)

pipeline.start()
pipeline.join()
```

The code above does object detection annotation directly (through the `render_boxes` sink). For more information, see the [Inference Pipeline](/reference/inference/inference-python/inference-pipeline.md) page.

## Contributing

Inference is open source. The source code, issue tracker, and contribution guide live in the [roboflow/inference](https://github.com/roboflow/inference) repository; see [CONTRIBUTING.md](https://github.com/roboflow/inference/blob/main/CONTRIBUTING.md) to get started.

## Next steps

* [Native Python API](/reference/inference/inference-python/native-python-api.md) - load models and run inference in your own process.
* [Inference Pipeline](/reference/inference/inference-python/inference-pipeline.md) - run models on video streams.
* [Model Weights Download](/reference/inference/inference-python/offline-weights.md) - weight caching and persistent storage.
* [Benchmarks](/reference/inference/inference-python/benchmarks.md) - measured throughput on common hardware.