> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/reference/inference/inference-sdk.md).

# Inference SDK

The `inference-sdk` Python package provides `InferenceHTTPClient`, a client for talking to an [Inference Server](https://docs.roboflow.com/deployment/self-hosted/inference-server) over HTTP. The same client works against the Roboflow [Serverless Hosted API](https://docs.roboflow.com/deployment/roboflow-cloud/serverless-api), a [Dedicated Deployment](https://docs.roboflow.com/deployment/roboflow-cloud/dedicated-deployments), a self-hosted server, or a server running on an edge device - only the `api_url` changes.

```bash
pip install inference-sdk
```

{% hint style="info" %}
`inference-sdk` is a thin HTTP client and does not run models itself. To load and run models inside your own Python process, use the [`inference` package](/reference/inference/inference-python.md).
{% endhint %}

## Quickstart

You can run inference on images from URLs, file paths, PIL images, and NumPy arrays.

{% tabs %}
{% tab title="URL" %}

```python
from inference_sdk import InferenceHTTPClient
import os

image_url = "https://media.roboflow.com/inference/soccer.jpg"

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["API_KEY"],
)

results = client.infer(image_url, model_id="soccer-players-5fuqs/1")
print(results)
```

{% endtab %}

{% tab title="NumPy Array" %}

```python
from inference_sdk import InferenceHTTPClient
import cv2
import os

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["API_KEY"],
)

numpy_image = cv2.imread("path/to/local/image.jpg")
results = client.infer(numpy_image, model_id="soccer-players-5fuqs/1")
print(results)
```

{% endtab %}

{% tab title="PIL Image" %}

```python
from inference_sdk import InferenceHTTPClient
from PIL import Image
import os

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["API_KEY"],
)

pil_image = Image.open("path/to/local/image.jpg")
results = client.infer(pil_image, model_id="soccer-players-5fuqs/1")
print(results)
```

{% endtab %}
{% endtabs %}

On the first request against a self-hosted server, the model weights are downloaded and set up. This request may take some time depending on your network connection and the size of the model. Once the model has downloaded, subsequent requests are much faster. You can also [pre-load models and manage loaded weights](/reference/inference/inference-sdk/model-management.md) to control this process.

{% hint style="info" %}
The model ID is composed of the string `<project_id>/<version_id>`. See [Workspace and Project IDs](/reference/authentication/authentication/workspace-and-project-ids.md) to find yours.
{% endhint %}

### Self-hosted server

You can also self-host the Inference Server (see the [Inference CLI](/reference/inference/inference-cli.md)), and then change `api_url` in the `InferenceHTTPClient`:

```python
client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key=os.environ["API_KEY"],
)
```

### AsyncIO client

```python
import asyncio
from inference_sdk import InferenceHTTPClient

CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)

image_url = "https://source.roboflow.com/pwYAXv9BTpqLyFfgQoPZ/u48G0UpWfk8giSw7wrU8/original.jpg"
loop = asyncio.get_event_loop()
result = loop.run_until_complete(
  CLIENT.infer_async(image_url, model_id="soccer-players-5fuqs/1")
)
```

## Parallel and batch inference

You may want to predict against multiple images in a single call. Two parameters of [`InferenceConfiguration`](/reference/inference/inference-sdk/configuration.md) control batching and parallelism:

* `max_concurrent_requests` - max number of concurrent requests that can be started
* `max_batch_size` - max number of elements that can be injected into a single request

This enables the following improvements:

* if you run the inference container on a powerful on-prem GPU machine, setting `max_batch_size` properly may bring throughput benefits
* if you run inference against the hosted Roboflow API, setting `max_concurrent_requests` causes multiple images to be served at once, bringing throughput benefits
* a combination of both options can be beneficial for clients running the inference container on a cluster of machines: the load of a single node can be optimised and parallel requests to different nodes can be made at a time

```python
from inference_sdk import InferenceHTTPClient

image_url = "https://source.roboflow.com/pwYAXv9BTpqLyFfgQoPZ/u48G0UpWfk8giSw7wrU8/original.jpg"

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
predictions = CLIENT.infer([image_url] * 5, model_id="soccer-players-5fuqs/1")

print(predictions)
```

Methods that support batching and parallelism:

* `infer(...)` and `infer_async(...)`
* `ocr_image(...)` and `ocr_image_async(...)` (enforcing `max_batch_size=1`)
* `detect_gazes(...)` and `detect_gazes_async(...)` - **deprecated**, always raises `inference_sdk.http.errors.FeatureDeprecatedError`
* `get_clip_image_embeddings(...)` and `get_clip_image_embeddings_async(...)`

The client also supports [core foundation models](/reference/inference/inference-sdk/core-models.md) (CLIP, DocTR), [running Workflows](/reference/inference/inference-sdk/workflows.md) for multi-step pipelines, and [WebRTC streaming](/reference/inference/inference-sdk/webrtc.md) for real-time video inference. Use WebRTC to process webcams, camera streams, and video files with either a model or a Workflow.

## What is actually returned as a prediction?

`InferenceHTTPClient` returns plain Python dictionaries that are the responses from the model serving API. Modification is done only in the context of the `visualization` key, which keeps the server-generated prediction visualisation and can be transcoded to the format of choice. Client-side rescaling only adjusts the input size.

## Next steps

* [Configuration](/reference/inference/inference-sdk/configuration.md) - client and model parameters, context managers, and defaults.
* [Model Management](/reference/inference/inference-sdk/model-management.md) - pre-load, list, and unload models on a server.
* [Core Models](/reference/inference/inference-sdk/core-models.md) - CLIP and DocTR endpoints.
* [Workflows](/reference/inference/inference-sdk/workflows.md) - run a Workflow through the client.
* [WebRTC Streaming](/reference/inference/inference-sdk/webrtc.md) - stream video through a model or Workflow.