Ctrlk
Sign InBook a DemoGet Started
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Configuration

Configure InferenceHTTPClient defaults, per-task model parameters, and Workflows execution options with InferenceConfiguration.

Configuration options

Configuring with context managers

The methods use_configuration(...) and use_model(...) are designed to work in context managers. Once the context manager is left, old config values are restored.

from inference_sdk import InferenceHTTPClient, InferenceConfiguration

image_url = "https://source.roboflow.com/pwYAXv9BTpqLyFfgQoPZ/u48G0UpWfk8giSw7wrU8/original.jpg"

custom_configuration = InferenceConfiguration(confidence_threshold=0.8)
# Replace ROBOFLOW_API_KEY with your Roboflow API Key
CLIENT = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)

with CLIENT.use_configuration(custom_configuration):
    _ = CLIENT.infer(image_url, model_id="soccer-players-5fuqs/1")

with CLIENT.use_model("soccer-players-5fuqs/1"):
    _ = CLIENT.infer(image_url)

# after leaving context manager - changes are reverted and `model_id` is still required
_ = CLIENT.infer(image_url, model_id="soccer-players-5fuqs/1")

As you can see, model_id is required for a prediction method only when a default model is not configured.

The model ID is composed of the string <project_id>/<version_id>. See Workspace and Project IDs to find these pieces of information.

Setting the configuration once and using it until the next change

The methods configure(...) and select_model(...) alter the client state and the change is preserved until the next change.

You may also initialise in chain mode:

Overriding model_id for a specific call

model_id can be overridden for a specific call:

Details about client configuration

InferenceHTTPClient provides the InferenceConfiguration dataclass to hold the full configuration.

Overriding fields in this config changes the behaviour of the client (and of the API serving the model). Specific fields are used in specific contexts. In particular:

Classification model

  • visualize_predictions: flag to enable / disable visualisation

  • confidence_threshold as confidence

  • stroke_width: width of stroke in visualisation

  • disable_preproc_auto_orientation, disable_preproc_contrast, disable_preproc_grayscale, disable_preproc_static_crop to alter server-side pre-processing

  • disable_active_learning to prevent the Active Learning feature from registering the datapoint (can be useful, for instance, while testing a model)

  • active_learning_target_dataset - when making inference from a specific model (let's say project_a/1) and you want to save data in another project project_b, the latter should be pointed to by this parameter. Note that you cannot use different types of models in project_a and project_b; if that is the case, data will not be registered.

  • source: optional string that sets a "source" attribute on the inference call. If using model monitoring, this is logged with the inference request so you can filter or query inference requests coming from a particular source, for example to identify which application, system, or deployment is making the request.

  • source_info: optional string that sets an additional "source_info" attribute on the inference call, for example to identify a sub-component in an app.

Object detection model

  • visualize_predictions: flag to enable / disable visualisation

  • visualize_labels: flag to enable / disable label visualisation if visualisation is enabled

  • confidence_threshold as confidence

  • class_filter to filter out a list of classes

  • class_agnostic_nms: flag to control whether NMS is class-agnostic

  • fix_batch_size

  • iou_threshold: to dictate the NMS IoU threshold

  • stroke_width: width of stroke in visualisation

  • max_detections: max detections to return from the model

  • max_candidates: max candidates for post-processing from the model

  • disable_preproc_auto_orientation, disable_preproc_contrast, disable_preproc_grayscale, disable_preproc_static_crop to alter server-side pre-processing

  • disable_active_learning, active_learning_target_dataset, source, source_info - as described above

Keypoint detection model

  • visualize_predictions: flag to enable / disable visualisation

  • visualize_labels: flag to enable / disable label visualisation if visualisation is enabled

  • confidence_threshold as confidence

  • keypoint_confidence_threshold (as keypoint_confidence) to filter out detected keypoints based on model confidence

  • class_filter to filter out a list of object classes

  • class_agnostic_nms: flag to control whether NMS is class-agnostic

  • fix_batch_size

  • iou_threshold: to dictate the NMS IoU threshold

  • stroke_width: width of stroke in visualisation

  • max_detections: max detections to return from the model

  • max_candidates: max candidates for post-processing from the model

  • disable_preproc_auto_orientation, disable_preproc_contrast, disable_preproc_grayscale, disable_preproc_static_crop to alter server-side pre-processing

  • disable_active_learning, active_learning_target_dataset, source, source_info - as described above

Instance segmentation model

  • visualize_predictions: flag to enable / disable visualisation

  • visualize_labels: flag to enable / disable label visualisation if visualisation is enabled

  • confidence_threshold as confidence

  • class_filter to filter out a list of classes

  • class_agnostic_nms: flag to control whether NMS is class-agnostic

  • fix_batch_size

  • iou_threshold: to dictate the NMS IoU threshold

  • stroke_width: width of stroke in visualisation

  • max_detections: max detections to return from the model

  • max_candidates: max candidates for post-processing from the model

  • disable_preproc_auto_orientation, disable_preproc_contrast, disable_preproc_grayscale, disable_preproc_static_crop to alter server-side pre-processing

  • mask_decode_mode

  • tradeoff_factor

  • disable_active_learning, active_learning_target_dataset, source, source_info - as described above

Configuration of the client

  • output_visualisation_format: one of VisualisationResponseFormat.BASE64, VisualisationResponseFormat.NUMPY, VisualisationResponseFormat.PILLOW. Given that server-side visualisation is enabled, you may choose which format should be used in the output.

  • client_downsizing_disabled: set to False if you want to perform client-side downsizing. Default True. Client-side scaling is only supposed to down-scale (keeping aspect ratio) the input for inference, to utilise the internet connection more efficiently (at the price of image manipulation / transcoding). Model input size information is used to determine the target size; if not available, default_max_input_size is used.

  • max_concurrent_requests: max number of concurrent requests that can be started

  • max_batch_size: max number of elements that can be injected into a single request

  • workflow_run_retries_enabled: flag that decides if transient errors in Workflows executions should be retried. Defaults to true and the default can be altered with the environment variable WORKFLOW_RUN_RETRIES_ENABLED.

Configuration of Workflows execution

  • profiling_directory: specifies the location where Workflows profiler traces are saved. By default, it is the ./inference_profiling directory.

PreviousInference SDKNextCore Models

Last updated

Was this helpful?