Detections Classes Replacement

Replace classes of detections with classes predicted by a chained classification model.

Replace class labels of detection bounding boxes with classes predicted by a classification model applied to cropped regions, combining generic detection results with specialized classification predictions to enable two-stage detection workflows, fine-grained classification, and class refinement workflows where generic detections are refined with specific class labels from specialized classifiers.

How This Block Works

This block combines results from a detection model (with bounding boxes and generic classes) with classification predictions (from a specialized classifier applied to cropped regions) to replace generic class labels with specific ones. The block:

1. Receives two inputs with different dimensionality levels:

   • object_detection_predictions: Detection results (dimensionality level 1) containing bounding boxes with generic classes (e.g., "dog", "person", "vehicle")

   • classification_predictions: Classification results (dimensionality level 2) from a classifier applied to cropped regions of each detection (e.g., "Golden Retriever", "Labrador" for dog detections). Can also be a list of strings (e.g. from OCR).

2. Matches classifications to detections:

   • Uses PARENT_ID_KEY (detection_id) in classification predictions to link each classification result to its source detection, OR

   • Uses positional mapping (order-based) if predictions are raw strings/lists without parent IDs.

3. Extracts leading class from each classification prediction:

   For single-label classifications:

   • Uses the "top" class (predicted class) from the classification result

   • Extracts class name, class ID, and confidence from the classification prediction

   For multi-label classifications:

   • Finds the class with the highest confidence score

   • Uses the most confident label as the replacement class

   • Extracts class name, class ID, and confidence from the highest-confidence prediction

   For string predictions:

   • Uses the string as the class name

   • Assigns a default confidence of 1.0 and class ID of 0

4. Handles missing classifications:

   • Detections without corresponding classification predictions are discarded by default

   • If fallback_class_name is provided, detections without classifications use the fallback class instead of being discarded

   • Fallback class ID is set to the provided value, or sys.maxsize if not specified or negative

5. Filters detections:

   • Keeps only detections that have classification results (or fallback if specified)

   • Removes detections that cannot be matched to classification predictions

6. Replaces class information:

   • Replaces class names in detections with classification class names

   • Replaces class IDs in detections with classification class IDs

   • Replaces confidence scores in detections with classification confidence scores

   • Updates all detection metadata to reflect the new class information

7. Generates new detection IDs:

   • Creates new unique detection IDs for updated detections (prevents ID conflicts)

   • Ensures detection IDs are unique after class replacement

8. Returns updated detections:

   • Outputs detections with replaced classes, maintaining bounding box coordinates and other properties

   • Output dimensionality matches input detection predictions (dimensionality level 1)

The block enables two-stage detection workflows where a generic detection model locates objects and a specialized classification model provides fine-grained labels. This is useful when you need generic localization (e.g., "dog") combined with specific classification (e.g., "Golden Retriever", "German Shepherd") without losing spatial information.

Common Use Cases

• Two-Stage Detection and Classification: Combine generic detection with specialized classification for fine-grained labeling (e.g., detect "dog" then classify breed, detect "vehicle" then classify type, detect "person" then classify age group), enabling two-stage detection workflows

• Class Refinement: Refine generic class labels with specific classifications from specialized models (e.g., refine "animal" to specific species, refine "vehicle" to specific models, refine "food" to specific dishes), enabling class refinement workflows

• Multi-Model Workflows: Combine detection and classification models to leverage the strengths of both (e.g., use generic detector for localization and specialist classifier for identification, combine coarse and fine-grained models, leverage specialized classifiers with general detectors), enabling multi-model workflows

• Hierarchical Classification: Apply hierarchical classification where detection provides high-level classes and classification provides detailed sub-classes (e.g., detect "mammal" then classify species, detect "plant" then classify variety, detect "structure" then classify type), enabling hierarchical classification workflows

• Crop-Based Classification: Use classification results from cropped regions to enhance detection results (e.g., classify crops to improve detection labels, apply specialized classifiers to detected regions, refine detections with crop classifications), enabling crop-based classification workflows

• Fine-Grained Object Recognition: Enable fine-grained recognition by combining localization and detailed classification (e.g., recognize specific product models, identify specific animal breeds, classify specific vehicle types), enabling fine-grained recognition workflows

Connecting to Other Blocks

This block receives detection and classification predictions and produces detections with replaced classes:

• After detection and classification model blocks to combine generic detection with specialized classification (e.g., object detection + classification to refined detections, detection model + classifier to labeled detections), enabling detection-classification fusion workflows

• After crop blocks that create crops from detections for classification (e.g., crop detections then classify crops, create crops for classification then replace classes), enabling crop-classification workflows

• Before visualization blocks to display detections with refined classes (e.g., visualize refined detections, display detections with specific labels, show classification-enhanced detections), enabling refined detection visualization workflows

• Before filtering blocks to filter detections with refined classes (e.g., filter by specific classes, filter refined detections, apply filters to classified detections), enabling refined detection filtering workflows

• Before analytics blocks to perform analytics on refined detections (e.g., analyze specific classes, perform analytics on classified detections, track refined detection metrics), enabling refined detection analytics workflows

• In workflow outputs to provide refined detections as final output (e.g., two-stage detection outputs, classification-enhanced detection outputs, refined detection results), enabling refined detection output workflows

Requirements

This block requires object detection predictions (with bounding boxes) and classification predictions from crops of those bounding boxes. The classification predictions must have PARENT_ID_KEY (detection_id) to link classifications to their source detections. The block accepts different dimensionality levels: detection predictions at level 1 and classification predictions at level 2 (from crops). For single-label classifications, the "top" class is used. For multi-label classifications, the most confident class is selected. Detections without classification results are discarded unless fallback_class_name is provided. The block outputs detections with replaced classes, class IDs, and confidences, with new detection IDs generated. Output dimensionality matches input detection predictions (level 1).

Type identifier

Use the following identifier in step "type" field: roboflow_core/detections_classes_replacement@v1 to add the block as a step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
fallback_class_name	str	Optional class name to use for detections that don't have corresponding classification predictions. If not provided (default None), detections without classifications are discarded. If provided, detections without classifications use this fallback class name instead of being removed. Useful for preserving detections when classification fails or is unavailable..	✅
fallback_class_id	int	Optional class ID to use with fallback_class_name for detections without classification predictions. If not specified or negative, the class ID is set to sys.maxsize. Only used when fallback_class_name is provided. Should match the class ID mapping used in your model..	✅

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds Detections Classes Replacement in version v1 has.

Input and output bindings

• input

  • object_detection_predictions (Union[instance_segmentation_prediction, keypoint_detection_prediction, object_detection_prediction]): Detection predictions (object detection, instance segmentation, or keypoint detection) containing bounding boxes with generic class labels that will be replaced with classification results. These detections should correspond to the regions that were cropped and classified. Detections must have detection IDs that match the PARENT_ID_KEY in classification predictions. Detections at dimensionality level 1..

  • classification_predictions (Union[classification_prediction, list_of_values, string]): Labels to replace detection class names with. Accepts classification predictions (linked via parent_id), plain strings, or lists of strings (e.g. OCR/LMM output like Gemini). String inputs are matched to detections positionally (1:1 by index). Classification inputs support single-label ('top' class) and multi-label (most confident class)..

  • fallback_class_name (string): Optional class name to use for detections that don't have corresponding classification predictions. If not provided (default None), detections without classifications are discarded. If provided, detections without classifications use this fallback class name instead of being removed. Useful for preserving detections when classification fails or is unavailable..

  • fallback_class_id (integer): Optional class ID to use with fallback_class_name for detections without classification predictions. If not specified or negative, the class ID is set to sys.maxsize. Only used when fallback_class_name is provided. Should match the class ID mapping used in your model..

• output

  • predictions (Union[object_detection_prediction, instance_segmentation_prediction, keypoint_detection_prediction]): Prediction with detected bounding boxes in form of sv.Detections(...) object if object_detection_prediction or Prediction with detected bounding boxes and segmentation masks in form of sv.Detections(...) object if instance_segmentation_prediction or Prediction with detected bounding boxes and detected keypoints in form of sv.Detections(...) object if keypoint_detection_prediction.

Example JSON definition

{
	    "name": "<your_step_name_here>",
	    "type": "roboflow_core/detections_classes_replacement@v1",
	    "object_detection_predictions": "$steps.object_detection_model.predictions",
	    "classification_predictions": "$steps.classification_model.predictions",
	    "fallback_class_name": null,
	    "fallback_class_id": null
	}