Model Management

Pre-load, inspect, and unload models on an Inference Server with the inference-sdk client.

Model weights download

When using a self-hosted Inference Server, you can pre-load models to download and cache weights before running inference:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="YOUR_ROBOFLOW_API_KEY"
)

# Pre-load the model (downloads weights to server cache)
client.load_model(model_id="rfdetr-base")

Alternatively, running a first inference will trigger the download automatically.

For Workflows, you should also pre-load all models used in the Workflow and run the Workflow once to cache its definition.

You can verify which models are loaded on the server:

loaded_models = client.list_loaded_models()
print(f"Loaded models: {loaded_models}")

Read more about weights caching, persistent storage, and Docker configuration.

Methods to control the Inference Server

Getting server info

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.get_server_info()

Listing loaded models

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.list_loaded_models()

Async equivalent: list_loaded_models_async()

Getting a specific model description

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.get_model_description(model_id="some/1", allow_loading=True)

If allow_loading is set to True, the model is loaded as a side effect if it is not already loaded. Default: True.

Async equivalent: get_model_description_async()

Loading a model

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.load_model(model_id="some/1", set_as_default=True)

The pointed model is loaded. If set_as_default is set to True, after a successful load the model is used as the default model for the client. Default value: False.

Async equivalent: load_model_async()

Unloading a model

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.unload_model(model_id="some/1")

Sometimes, to avoid OOM on the server side, unloading a model is required.

Async equivalent: unload_model_async()

Unloading all models

from inference_sdk import InferenceHTTPClient

# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
CLIENT.unload_all_models()

Async equivalent: unload_all_models_async()