Install on Linux
Install and run the Roboflow Inference Server on Linux with the Inference CLI, Docker, or Docker Compose, on CPU, GPU, or TensorRT.
The Docker examples on this page publish port 9001 on localhost. For clients on another machine, deliberately change the mapping to a reachable host address and configure server access.
The easiest way to start the correct container for your machine, with good default settings (a cache volume and a secure, non-privileged execution mode), is to let the CLI choose and start it with inference server start. Install Docker first:
pip install inference-cli
inference server startManually starting the container
If you want more control over the container settings, start it yourself.
The core CPU Docker image includes support for OpenVINO acceleration on x64 CPUs via onnxruntime. Heavy models like SAM2 may run too slowly (dozens of seconds per image) to be practical; if you need them, use a CUDA-capable GPU.
The primary use cases for CPU inference are processing still images (for example NSFW classification of uploads or document verification) or infrequent sampling of frames from a video (for example occupancy tracking of a parking lot).
To get started with CPU inference, use the roboflow/roboflow-inference-server-cpu:latest container.
sudo docker run -d \
--name inference-server \
--read-only \
-p 127.0.0.1:9001:9001 \
--volume ~/.inference/cache:/tmp:rw \
--security-opt="no-new-privileges" \
--cap-drop="ALL" \
--cap-add="NET_BIND_SERVICE" \
roboflow/roboflow-inference-server-cpu:latestThe GPU container adds hardware acceleration on cards that support CUDA via NVIDIA-Docker. Follow the NVIDIA Container Toolkit installation guide, then add --gpus all to the docker run command:
sudo docker run -d \
--name inference-server \
--gpus all \
--read-only \
-p 127.0.0.1:9001:9001 \
--volume ~/.inference/cache:/tmp:rw \
--security-opt="no-new-privileges" \
--cap-drop="ALL" \
--cap-add="NET_BIND_SERVICE" \
roboflow/roboflow-inference-server-gpu:latestWith the GPU container you can optionally enable TensorRT, NVIDIA's model optimization runtime. It greatly increases your models' speed at the expense of a heavy compilation and optimization step (sometimes 15+ minutes) the first time you load each model.
Enable TensorRT by adding TensorrtExecutionProvider to the ONNXRUNTIME_EXECUTION_PROVIDERS environment variable.
Docker Compose
If you use Docker Compose for your application, the equivalent YAML is:
Next steps
Run a model against your new server.
Docker configuration options for ports, caching, and model limits.
Securing a self-hosted server before you expose it beyond localhost.
Last updated
Was this helpful?