Detect Small Objects

Use SAHI (slicing aided hyper inference) in a Workflow to detect small objects by slicing images, running the model per tile, and stitching detections.

The best way to improve the performance of a machine learning system is often peripheral to improving the model. Collecting more data, labeling datasets more accurately, adding redundancy to detections, or breaking down the problem into smaller pieces are usually higher-leverage endeavors than iterating on model architecture.

One common shortcoming of model performance occurs when the objects of interest are very small relative to the size of the image. One approach is tiling the image into smaller chunks and feeding them through the model in parallel. This method is called SAHI (Slicing Aided Hyper Inference). A large image is divided into overlapping slices, each slice is run through a model, and a single prediction is reconstructed as the output.

Using SAHI in Workflows is easy with the Image Slicer and Detections Stitch blocks.

Difficulty: Easy Time to complete: 10 minutes

Starting point

We will start with an output similar to the Hello World tutorial. Follow that guide to create a simple Workflow that visualizes predictions from an object detection model:

Starting Point

But when the objects are small, this model fails to find them.

Testing Without SAHI

Even though cars are one of the most widely represented classes in the MS COCO dataset that this model was trained on, it still only finds one of them, because when the image is scaled down to the model's input resolution the cars become only a few pixels wide.

Original Predictions

Adding SAHI

We will fix this by slicing the image into smaller tiles so that the model gets more pixels of information to work from.

Start SAHI Path

The first step is to add the Image Slicer block to the Workflow. We will do this in a parallel execution path so that we can compare the outputs to the original non-SAHI predictions at the end.

Add Image Slicer Block

Then we will add an object detection model that runs on each slice. Everything between an Image Slicer and an Image Stitch block operates element-wise on the set of slices. Workflows automatically parallelizes work (such as batching predictions through the GPU) where it can.

Add Model

Next, we'll add the Detections Stitch block to aggregate the predictions back into the dimensionality of the original image.

Add Detections Stitch

Now, we can visualize these aggregated predictions in the same way as we visualized outputs of a regular model.

Add Visualizations

We will add a Background Color Visualization to dim regions of the image that were not predicted by the model, to make the predicted areas stand out.

Add Background Color

And a bounding box around predicted objects of interest.

Add Bounding Box

Testing SAHI

Now we're ready to see how our revised approach is working. We'll simplify the output to predict only the two visualizations we're interested in.

Adjust Outputs

When we run the Workflow we see that now all of the cars are detected.

Test SAHI

Output with SAHI

Filter unwanted predictions

But the SAHI approach worked a little bit too well. The model is now picking up some objects from the background that we weren't interested in. Let's fix this by modifying the predictions from the model.

Add Filter

Add a Detections Filter block. This block lets us exclude predictions that don't match a set of criteria that we define.

Select Detections Filter

To set our criteria, click the Operations configuration in the block's sidebar.

Choose Operations

Then choose "Filter By Class & Confidence".

Select Class Confidence

And include only the car class.

Filter By Class

Now, when we test our model again we see boxes around the cars and no extraneous detections in the background.

Test With Filter

Output with SAHI and Filter

Next steps

In this tutorial we learned how to use SAHI to detect small objects, how to perform operations on a batch of images, and how to transform detections.

Next, we will run a Workflow on a live video stream.