Parallel HTTP API

Run the Roboflow Inference parallel HTTP server to process inference requests asynchronously for higher throughput and lower latency.

Enterprise feature. This feature is only available to Roboflow Enterprise users. Contact the sales team to learn more about Roboflow Enterprise.

You can run multiple models in parallel with parallel processing, a version of Roboflow Inference that processes inference requests asynchronously.

Inference Parallel supports the same features as Roboflow Inference, except that it does not support core models (CLIP and SAM).

With Inference Parallel, preprocessing, auto batching, inference, and post-processing all run in separate threads to increase server FPS throughput. Separate requests to the same model are batched on the fly as allowed by $MAX_BATCH_SIZE, and response handling then occurs independently. Images are passed via Python's SharedMemory module to maximize throughput.

These changes result in as much as a 76% speedup on one measured workload.

How to run Inference with parallel processing

Bash

First, build the parallel server:

./inference/enterprise/parallel/build.sh

Then run the server:

./inference/enterprise/parallel/run.sh

A message appears in the terminal indicating that the server is running and ready for use.

Docker

A container is published on Docker Hub:

docker pull roboflow/roboflow-inference-server-gpu-parallel:latest

If you are pulling a pinned tag, change the $TAG variable in run.sh to match.

Benchmarks

We evaluated the performance of Inference Parallel on a variety of models from Roboflow Universe, comparing it against Inference Server 0.9.5.rc on the same hardware: a computer with eight cores and one GPU. Instance segmentation metrics were calculated with "mask_decode_mode": "fast" in the request body. Requests were posted concurrently with a parallelism of 1000.

Workspace	Model	Model type	Split	0.9.5.rc FPS	0.9.5.parallel FPS
senior-design-project-j9gpp	nbafootage/3	object-detection	train	30.2 fps	44.03 fps
niklas-bommersbach-jyjff	dart-scorer/8	object-detection	train	26.6 fps	47.0 fps
geonu	water-08xpr/1	instance-segmentation	valid	4.7 fps	6.1 fps
university-of-bradford	detecting-drusen_1/2	instance-segmentation	train	6.2 fps	7.2 fps
fy-project-y9ecd	cataract-detection-viwsu/2	classification	train	48.5 fps	65.4 fps
hesunyu	playing-cards-ir0wr/1	classification	train	44.6 fps	57.7 fps

Inference with parallel processing enabled achieved higher FPS on every test. On some models the FPS increase was greater than 10 FPS.