Inference Server

What the Inference Server is, how to start it with Docker and the Inference CLI, and how to open its built-in JupyterLab notebook.

The Inference Server is a standalone microservice that wraps the inference Python package. It exposes HTTP endpoints for images and a WebRTC endpoint for video streams. One server can serve multiple clients and run the same models and Workflows as Roboflow's hosted APIs. It is the recommended way to self-host: see Pick a path for how it compares to running the library directly.

Where it runs

Self-host the server on your own hardware (Raspberry Pi, NVIDIA GPU, NVIDIA Jetson, or a plain server) with Docker, or in your own AWS, GCP, or Azure account. Roboflow also runs the same server for you as the Serverless Cloud API and Dedicated Deployments: see Choosing a Deployment Option.

Whichever you pick, you talk to it through the Inference SDK, because they share one interface: only the api_url changes.

Running with Docker

Before you begin, make sure Docker is installed on your machine. The easiest way to start the Inference Server is with the Inference CLI:

pip install inference-cli && inference server start

This pulls the appropriate Docker image and publishes port 9001 on 127.0.0.1 by default. Jetson images and --tunnel launches publish on 0.0.0.0; see server security before allowing remote clients.

Check the server status with:

inference server status

Manually setting up a Docker container

inference server start runs docker run under the hood with recommended security settings, caching, and platform-specific options.

If you want to start the container yourself, see the "Manually starting the container" section of your platform's install guide:

• Linux

• Windows

• Mac

• Jetson

• Raspberry Pi

Container settings are controlled with environment variables: see Docker configuration options and the full environment variable reference.

Built-in JupyterLab notebook

Inference Servers ship with a built-in JupyterLab environment, which is the fastest way to experiment during development and testing. It is disabled by default, so start the server with the --dev flag to enable it:

pip install inference-cli
inference server start --dev

Then open http://localhost:9001 in your browser to see the Inference landing page, which links to resources, examples, and the built-in JupyterLab environment. Select "Jump Into an Inference Enabled Notebook" to open JupyterLab in a new tab. It comes preloaded with example notebooks and all the dependencies needed to run Inference.

The --dev notebook environment is meant for local development. Do not enable it on a server that is reachable from an untrusted network: see Securing a Self-Hosted Server.

Stream video

Use the Inference SDK WebRTC client to stream webcams, camera feeds, and video files through a model or Workflow:

pip install "inference-sdk[webrtc]"

Set api_url="http://localhost:9001" when you create InferenceHTTPClient. See WebRTC Streaming for model and Workflow examples, or follow Video processing with Workflows for a task-based guide.

In this section

Install Inference Server

Requirements, per-device install guides, your own cloud, and updating.

Run a Model

Your first request over HTTP, model IDs, and visualization.

Configuration

Container options, environment variables, security, HTTPS, and telemetry.

Architecture

How requests, video, and Workflows flow through the server.