Install on NVIDIA Jetson
Install the Roboflow Inference Server on an NVIDIA Jetson device with JetPack-specific containers, TensorRT acceleration, and Docker Compose.
Overview
Jetson is NVIDIA's line of compact, power-efficient modules designed to run AI and deep learning workloads at the edge. They combine a GPU, CPU, and neural accelerators on a single board, which makes them a good fit for robotics, drones, smart cameras, and other embedded applications that need real-time computer vision without a cloud connection. For more details, see NVIDIA's Jetson overview.
Prerequisites
Disk space: allocate at least 10 GB free for the Roboflow Jetson image (8.14 GB).
JetPack version: a supported JetPack (5.x or 6.x).
Recommended hardware: an NVIDIA Orin NX 16 GB or above for best performance.
Docker and the NVIDIA Container Toolkit: containers need the Docker engine plus the NVIDIA runtime to access the GPU. Follow the Docker install guide and the NVIDIA Container Toolkit guide.
Roboflow publishes specialized containers built with hardware acceleration support for JetPack L4T. To detect your JetPack version automatically and start the right container with good defaults, run:
pip install inference-cli
inference server startManually starting the container
If you want more control over the container settings, start it yourself. Jetson devices with NVIDIA JetPack are pre-configured with the NVIDIA container runtime and are hardware accelerated out of the box.
sudo docker run -d \
--name inference-server \
--runtime nvidia \
--read-only \
-p 9001:9001 \
--volume ~/.inference/cache:/tmp:rw \
--security-opt="no-new-privileges" \
--cap-drop="ALL" \
--cap-add="NET_BIND_SERVICE" \
roboflow/roboflow-inference-server-jetson-6.2.0:latestsudo docker run -d \
--name inference-server \
--runtime nvidia \
--read-only \
-p 9001:9001 \
--volume ~/.inference/cache:/tmp:rw \
--security-opt="no-new-privileges" \
--cap-drop="ALL" \
--cap-add="NET_BIND_SERVICE" \
roboflow/roboflow-inference-server-jetson-6.0.0:latestsudo docker run -d \
--name inference-server \
--runtime nvidia \
--read-only \
-p 9001:9001 \
--volume ~/.inference/cache:/tmp:rw \
--security-opt="no-new-privileges" \
--cap-drop="ALL" \
--cap-add="NET_BIND_SERVICE" \
roboflow/roboflow-inference-server-jetson-5.1.1:latestUse the same command as JetPack 5 with the JetPack 4 image tag: roboflow/roboflow-inference-server-jetson-4.6.1:latest for JetPack 4.6, or roboflow/roboflow-inference-server-jetson-4.5.0:latest for JetPack 4.5.
TensorRT
You can optionally enable TensorRT, NVIDIA's model optimization runtime. It greatly increases your models' speed at the expense of a heavy compilation and optimization step (sometimes 15+ minutes) the first time you load each model.
Enable TensorRT by adding TensorrtExecutionProvider to the ONNXRUNTIME_EXECUTION_PROVIDERS environment variable on any of the commands above:
Mounting a persistent cache volume (as in the commands above) keeps the compiled TensorRT engines between restarts, so you only pay the compilation cost once per model.
Docker Compose
If you use Docker Compose for your application, the equivalent YAML is below. Swap the image tag for your JetPack version: jetson-6.2.0, jetson-6.0.0, jetson-5.1.1, jetson-4.6.1, or jetson-4.5.0.
Next steps
Run a model against your new server.
Deployment Manager (Enterprise) to manage a fleet of Jetson devices remotely.
Securing a self-hosted server before you expose it beyond localhost.
Last updated
Was this helpful?