Inference Server
What the Inference Server is, how to start it with Docker and the Inference CLI, and how to open its built-in JupyterLab notebook.
Last updated
Was this helpful?
What the Inference Server is, how to start it with Docker and the Inference CLI, and how to open its built-in JupyterLab notebook.
The Inference Server is a standalone microservice that wraps the inference Python package. It exposes HTTP endpoints for images and a WebRTC endpoint for video streams. One server can serve multiple clients and run the same models and Workflows as Roboflow's hosted APIs. It is the recommended way to self-host: see Pick a path for how it compares to running the library directly.
Self-host the server on your own hardware (Raspberry Pi, NVIDIA GPU, NVIDIA Jetson, or a plain server) with Docker, or in your own AWS, GCP, or Azure account. Roboflow also runs the same server for you as the Serverless Cloud API and Dedicated Deployments: see Choosing a Deployment Option.
Whichever you pick, you talk to it through the Inference SDK, because they share one interface: only the api_url changes.
Before you begin, make sure Docker is installed on your machine. The easiest way to start the Inference Server is with the Inference CLI:
pip install inference-cli && inference server startThis pulls the appropriate Docker image for your machine, with dependencies pre-installed, and starts the Inference Server on port 9001. Check the server status with:
inference server statusinference server start runs docker run under the hood with recommended security settings, caching, and platform-specific options.
If you want to start the container yourself, see the "Manually starting the container" section of your platform's install guide:
Container settings are controlled with environment variables: see Docker configuration options and the full environment variable reference.
Inference Servers ship with a built-in JupyterLab environment, which is the fastest way to experiment during development and testing. It is disabled by default, so start the server with the --dev flag to enable it:
Then open http://localhost:9001 in your browser to see the Inference landing page, which links to resources, examples, and the built-in JupyterLab environment. Select "Jump Into an Inference Enabled Notebook" to open JupyterLab in a new tab. It comes preloaded with example notebooks and all the dependencies needed to run Inference.
The --dev notebook environment is meant for local development. Do not enable it on a server that is reachable from an untrusted network: see Securing a Self-Hosted Server.
Use the Inference SDK WebRTC client to stream webcams, camera feeds, and video files through a model or Workflow:
Set api_url="http://localhost:9001" when you create InferenceHTTPClient. See WebRTC Streaming for model and Workflow examples, or follow Video processing with Workflows for a task-based guide.
Last updated
Was this helpful?
Was this helpful?
pip install inference-cli
inference server start --devpip install "inference-sdk[webrtc]"