Benchmark Inference

Measure throughput and latency of the Inference Server or the inference Python package with inference benchmark.

inference benchmark offers an easy way to check the performance of Inference in your setup. The command can benchmark both the Inference Server and the inference Python package.

To see the details of the command, run:

inference benchmark --help

A help guide is also available for each sub-command:

inference benchmark api-speed --help

Benchmarking the inference Python package

Running this command requires the inference package to be installed.

pip install inference

A basic benchmark can be run using the following command:

inference benchmark python-package-speed \
  -m {your_model_id} \
  -d {pre-configured dataset name or path to directory with images} \
  -o {output_directory}

The command runs a specified number of inferences using the given model and saves statistics (including benchmark parameters, throughput, latency, errors, and platform details) in the given directory.

Benchmarking the Inference Server

Before running an API benchmark of your local Inference Server, make sure the server is up and running:

inference server start

A basic benchmark can be run using the following command:

inference benchmark api-speed \
  -m {your_model_id} \
  -d {pre-configured dataset name or path to directory with images} \
  -o {output_directory}

The command runs a specified number of inferences using the given model and saves statistics (including benchmark parameters, throughput, latency, errors, and platform details) in the given directory.

This benchmark has more configuration options to support different ways of HTTP API profiling. In default mode, a single client is spawned and it sends one request after another sequentially. This may be suboptimal in specific cases, so you may specify the number of concurrent clients using the -c {number_of_clients} option. Each client sends the next request once the previous one is handled.

This option still does not cover all test scenarios. For instance, you may want to send x requests each second, which is closer to a production environment where multiple clients send requests concurrently. In this scenario, the --rps {value} option can be used (and -c is ignored). The value provided in --rps specifies how many requests are spawned each second without waiting for previous requests to be handled. In I/O intensive benchmark scenarios, we suggest running the command from multiple separate processes and possibly multiple hosts.

See Inference Benchmarks for published results measured with this command.