For the complete documentation index, see llms.txt. This page is also available as Markdown.

Batch Processing

Run Workflows on large batches of images and stored videos with cloud infrastructure provisioned for you.

About

Batch Processing is a cost-effective way to run Workflows on batches of images and stored videos. It's ideal for asynchronously processing large amounts of data.

Batch Processing automatically provisions the infrastructure needed to run a large batch.

Batch Processing is available on Growth and Enterprise plans. You can start a job from the "Batch Processing" tab, or run a Workflow on a selection in the Asset Library.

You can configure a Batch Processing job through the Roboflow web interface or through our API (via the CLI).

When you start a job, machines will be provisioned in the cloud to process your data. You will then receive a JSON file with the output from the Workflow you chose to run on your data.

The following video explains Batch Processing in depth:

Web App

Create a Batch Processing Job

To create a Batch Processing job, click Deployments in the left sidebar of your Roboflow dashboard. Then, click on the "Batch Processing" tab:

Click "New Batch Job" to create a Batch Processing job.

A window will open in which you can configure your job:

Choose a Workflow

To start configuring a job, first select a Workflow. If you do not already have a Workflow, refer to our Workflows documentation to get started.

Upload Images or Videos

Next, you need to upload the images or videos on which you want to run your Workflow.

Configure Hardware

You can run your Batch Processing job on a CPU or a GPU. GPU jobs are faster but more expensive.

For pricing information, refer to the Roboflow pricing documentation.

Select either a CPU or GPU for your job:

Several advanced configuration options are also available under the "Advanced Options" tab. We recommend leaving these options as the default.

Start the Job

To start the Batch Processing job, click "Create Batch Job".

The infrastructure for your job will be provisioned and processing will begin.

Monitor Job Progress

When you start your job, a status indicator will appear indicating when processing is being configured, when the batch data is being processed, and when the job is complete.

You can monitor how much of a batch has been processed in real time.

The amount of time it will take to process your data depends on how many images or videos you are processing, the complexity of your Workflow, and whether you selected CPU or GPU hardware.

Open a job to view its details, including the "Input Source" that shows which images the job ran on: the Asset Library search query used to select them (with a link to reopen that selection), or the number of images picked manually.

Run a Workflow from the App

Besides the API and CLI, you can start a Batch Processing job directly from the Roboflow app to run a Workflow over large sets of stored images. There are two ways to do this:

  • On demand, from the Asset Library.

  • Automatically, each time a Datasource mirrors new images from a cloud bucket.

From the Asset Library

The Asset Library lets you run a Workflow on the images you select, on demand.

Select images manually, or select all images matching your current search, then click "Run Workflow". For the full flow, including how to write results back onto your images, see Running a Workflow.

Automatically When a Datasource Mirrors

A Datasource mirrors images and metadata from a cloud bucket into your Workspace. You can automatically run a Workflow over the new images each mirror imports. This keeps enrichment such as tagging, quality scoring, or pre-labeling up to date as new data arrives, with no manual step.

You configure these automations in the "Workflow runs" section of your Datasources page.

Managing Workflow runs requires a Workspace role with permission to manage batch automations. If you do not see the "Workflow runs" section, ask a Workspace admin.

To add an automation:

  1. In the "Workflow runs" section, click "Add workflow run".

  2. Enter a Name for the automation.

  3. Under "Run when", choose "On Sync" and select the Datasources that should trigger it.

  4. Select the Workflow to run. It must have exactly one image input.

  5. Choose a Machine type (CPU or GPU).

  6. Click "Create".

Each time one of the selected Datasources mirrors, the automation runs the Workflow as a Batch Processing job over the images that mirror imported. Track progress in the Activity Center and on the "Batch Processing" tab under Deployments, the same as any other Batch Processing job.

To have a Workflow write its results back onto your images so you can search for them in the Asset Library, see Writing results back to the Asset Library.

Run a Job with the API or CLI

To create and run a Batch Processing job programmatically, see the HTTP API and CLI sections below. For debugging common issues, see Troubleshooting.

HTTP API

Quick Links:

Ingest Data

Upload Video

Upload a video

post

Request a signed URL to upload a video file. After receiving the response, PUT the video to the uploadURL with the provided extensionHeaders.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

fileNamestringRequired

Name of the video file (e.g. my_video.mp4).

Responses
200

Signed URL details for uploading the video.

application/json
statusstringOptionalExample: ok
post/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/video
POST /data-staging/v1/external/{workspace}/batches/{batch_id}/upload/video?api_key=text&fileName=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Signed URL details for uploading the video.

{
  "status": "ok",
  "signedURLDetails": {
    "uploadURL": "text",
    "method": "PUT",
    "extensionHeaders": {
      "Content-Type": "application/octet-stream",
      "x-goog-content-length-range": "1,1073741824"
    },
    "maxFileSize": 1073741824
  }
}

Upload Image

Upload a single image

post

Upload a single image via multipart form data. Best for batches up to 5,000 images.

Note: Single-image and bulk uploads cannot be combined for the same batch.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

fileNamestringRequired

Name of the image file.

Body
filestring ยท binaryOptional

The image file to upload.

Responses
200

Image uploaded successfully.

application/json
statusstringOptionalExample: ok
post/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/image
POST /data-staging/v1/external/{workspace}/batches/{batch_id}/upload/image?api_key=text&fileName=text HTTP/1.1
Host: api.roboflow.com
Content-Type: multipart/form-data
Accept: */*
Content-Length: 17

{
  "file": "binary"
}
200

Image uploaded successfully.

{
  "status": "ok"
}

Bulk Upload Images

Bulk upload images

post

Request a signed URL for uploading a .tar archive of images. Recommended for batches exceeding 5,000 images. Bundle up to 500 images per archive.

The response contains a signed URL and extension headers. Pack images into a .tar archive and PUT it to the signed URL.

Note: Bulk and single-image uploads cannot be combined for the same batch.

When performing bulk ingestion, data is indexed in the background. There may be a short delay before all data is available.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Responses
200

Signed URL details for uploading a tar archive.

application/json
statusstringOptionalExample: ok
post/data-staging/v1/external/{workspace}/batches/{batch_id}/bulk-upload/image-files
POST /data-staging/v1/external/{workspace}/batches/{batch_id}/bulk-upload/image-files?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Signed URL details for uploading a tar archive.

{
  "status": "ok",
  "signedURLDetails": {
    "shardId": "123e4567-e89b-12d3-a456-426614174000",
    "uploadURL": "text",
    "method": "PUT",
    "extensionHeaders": {
      "Content-Type": "application/x-tar",
      "x-goog-content-length-range": "1,536870912"
    },
    "maxNumberOfImages": 500,
    "maxShardSize": 536870912
  }
}

Check Batch Status

Get batch item count

get

Returns the count of ingested items in a batch. Use this to verify all data has been ingested before starting a job.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Responses
200

Batch item count.

application/json
statusstringOptionalExample: ok
countintegerOptional

Number of items in the batch.

Example: 42
get/data-staging/v1/external/{workspace}/batches/{batch_id}/count
GET /data-staging/v1/external/{workspace}/batches/{batch_id}/count?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Batch item count.

{
  "status": "ok",
  "count": 42
}

Check Shard Upload Details

List batch shards

get

Returns shard details for a bulk-upload batch. Paginated โ€” use nextPageToken from the response to fetch subsequent pages.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

nextPageTokenstringOptional

Pagination token from a previous response.

Responses
200

Paginated list of batch shards.

application/json
statusstringOptionalExample: ok
shardsobject[]Optional

List of shard objects.

nextPageTokenstring ยท nullableOptional

Token for fetching the next page of results. null if no more pages.

get/data-staging/v1/external/{workspace}/batches/{batch_id}/shards
GET /data-staging/v1/external/{workspace}/batches/{batch_id}/shards?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Paginated list of batch shards.

{
  "status": "ok",
  "shards": [
    {}
  ],
  "nextPageToken": "text"
}

Start a Job

Start a batch processing job

post

Start a batch processing job that runs a Workflow against staged data.

Job ID constraints: Lowercase letters, digits, hyphens, and underscores only. Maximum 20 characters.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

job_idstring ยท max: 20Required

Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Body
typestring ยท enumRequired

Job type.

Possible values:
processingTimeoutSecondsintegerOptional

Maximum cumulative machine runtime in seconds across all parallel workers.

Default: 3600
notificationsURLstring ยท uriOptional

Webhook URL for job completion notifications. Custom webhook headers are not yet supported. The only header sent is Authorization: Bearer rf_{workspace_id}.

Responses
200

Job started successfully.

application/json
statusstringOptionalExample: ok
post/batch-processing/v1/external/{workspace}/jobs/{job_id}
POST /batch-processing/v1/external/{workspace}/jobs/{job_id}?api_key=text HTTP/1.1
Host: api.roboflow.com
Content-Type: application/json
Accept: */*
Content-Length: 415

{
  "type": "simple-image-processing-v1",
  "jobInput": {
    "type": "staging-batch-input-v1",
    "batchId": "text"
  },
  "computeConfiguration": {
    "type": "compute-configuration-v2",
    "machineType": "cpu",
    "workersPerMachine": 4
  },
  "processingTimeoutSeconds": 3600,
  "processingSpecification": {
    "type": "workflows-processing-specification-v1",
    "workspace": "text",
    "workflowId": "text",
    "aggregationFormat": "jsonl"
  },
  "notificationsURL": "https://example.com"
}
200

Job started successfully.

{
  "status": "ok"
}

Monitor Job Progress

Get Job Status

Get job status

get

Returns the current status of a batch processing job.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

job_idstring ยท max: 20Required

Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Responses
200

Job status details.

application/json
statusstringOptionalExample: ok
jobStatusstringOptional

Current job status (e.g. pending, processing, completed, failed).

progressnumberOptional

Processing progress as a fraction between 0 and 1.

get/batch-processing/v1/external/{workspace}/jobs/{job_id}
GET /batch-processing/v1/external/{workspace}/jobs/{job_id}?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Job status details.

{
  "status": "ok",
  "jobStatus": "text",
  "progress": 1
}

List Job Stages

List job stages

get

Returns the list of stages for a job. Each job typically has processing and export stages, each producing an output batch.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

job_idstring ยท max: 20Required

Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Responses
200

List of job stages.

application/json
statusstringOptionalExample: ok
stagesobject[]Optional

List of stage objects. Each stage has an ID and an output batch ID.

get/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages
GET /batch-processing/v1/external/{workspace}/jobs/{job_id}/stages?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

List of job stages.

{
  "status": "ok",
  "stages": [
    {}
  ]
}

List Stage Tasks

List tasks for a stage

get

Returns the list of tasks for a specific job stage. Paginated โ€” use nextPageToken from the response to fetch subsequent pages.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

job_idstring ยท max: 20Required

Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
stage_idstringRequired

The stage identifier.

Query parameters
api_keystringRequired

Your Roboflow API key.

nextPageTokenstringOptional

Pagination token from a previous response.

Responses
200

Paginated list of tasks.

application/json
statusstringOptionalExample: ok
tasksobject[]Optional

List of task objects.

nextPageTokenstring ยท nullableOptional

Token for fetching the next page of results. null if no more pages.

get/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages/{stage_id}/tasks
GET /batch-processing/v1/external/{workspace}/jobs/{job_id}/stages/{stage_id}/tasks?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Paginated list of tasks.

{
  "status": "ok",
  "tasks": [
    {}
  ],
  "nextPageToken": "text"
}

Export Results

List Output Parts

List output batch parts

get

Lists the parts of an output batch. Use the export stage output batch for compressed results.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

Responses
200

List of batch parts.

application/json
statusstringOptionalExample: ok
get/data-staging/v1/external/{workspace}/batches/{batch_id}/parts
GET /data-staging/v1/external/{workspace}/batches/{batch_id}/parts?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

List of batch parts.

{
  "status": "ok",
  "parts": [
    {
      "partName": "text"
    }
  ]
}

List Download URLs

List download URLs

get

Returns paginated download URLs for files in a batch part.

Path parameters
workspacestringRequired

Your Roboflow workspace identifier.

batch_idstring ยท max: 64Required

Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores.

Pattern: ^[a-z0-9_-]+$
Query parameters
api_keystringRequired

Your Roboflow API key.

nextPageTokenstringOptional

Pagination token from a previous response.

partNamestringOptional

Filter by part name (from the list parts response).

Responses
200

Paginated list of download URLs.

application/json
statusstringOptionalExample: ok
nextPageTokenstring ยท nullableOptional

Token for fetching the next page of results. null if no more pages.

get/data-staging/v1/external/{workspace}/batches/{batch_id}/list
GET /data-staging/v1/external/{workspace}/batches/{batch_id}/list?api_key=text HTTP/1.1
Host: api.roboflow.com
Accept: */*
200

Paginated list of download URLs.

{
  "status": "ok",
  "filesMetadata": [
    {
      "downloadURL": "text",
      "fileName": "text",
      "partName": "text",
      "shardId": "text",
      "contentType": "text",
      "nestedContentType": "text"
    }
  ],
  "nextPageToken": "text"
}

Webhook Notifications

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes. See CLI Usage for webhook configuration and payload formats.

CLI

By installing inference-cli you gain access to the inference rf-cloud command, which allows you to interact with Batch Processing and Data Staging - the core components of Roboflow Batch Processing.

Setup

For cloud storage support:

If you need help finding your API key, see our authentication guide.

Ingest Data

Images

Videos

Batch ID format: Must be lowercase, at most 64 characters, with only letters, digits, hyphens (-), and underscores (_).

Cloud Storage

If your data is already in cloud storage (S3, Google Cloud Storage, or Azure), you can process it directly without downloading files locally.

For images:

For videos:

The --bucket-path parameter supports the following providers. Glob patterns filter which files are ingested:

Provider
Path format
Glob example

S3

s3://bucket-name/path/

s3://my-bucket/training-data/**/*.jpg - all JPGs recursively

Google Cloud Storage

gs://bucket-name/path/

gs://my-bucket/videos/2024-*/*.mp4 - MP4s in 2024-* folders

Azure Blob Storage

az://container-name/path/

az://container/images/*.png - PNGs in the images folder

Your cloud storage credentials are used only locally by the CLI to generate presigned URLs. They are never uploaded to Roboflow servers.

For large datasets, the system automatically splits images into chunks of 20,000 files each. Videos work best in batches under 1,000.

Signed URL Ingestion

For advanced automation, you can ingest data via signed URLs instead of local files:

Flag
Description

--data-source references-file

Process files referenced via signed URLs.

--references <path_or_url>

Path to a JSONL file containing file URLs, or a signed URL pointing to such a file.

Reference File Format (JSONL):

Signed URL ingestion is available to Growth Plan and Enterprise customers.

Inspect Staged Data

Start a Job

Process Images

Process Videos

Finding your Workflow ID: Open the Workflow Editor in the Roboflow App, click "Deploy", and find the identifier in the code snippet.

By default, processing runs on CPU. Use --machine-type gpu for Workflows with multiple or large models.

Monitor Job Progress

The start command outputs a Job ID. Use it to check status:

Export Results

The job details will include the output batch ID. Use it to export results:

Webhook Automation

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes.

Data Ingestion Webhooks

The CLI commands create-batch-of-images and create-batch-of-videos support:

Flag
Description

--notifications-url <webhook_url>

Webhook endpoint for notifications.

--notification-category ingest-status

Overall ingestion process status. Default.

--notification-category files-status

Individual file processing status.

Notifications are delivered via HTTP POST with an Authorization header containing your Roboflow Publishable Key.

Ingest Status Notification

File Status Notification

Job Completion Webhooks

Add --notifications-url when starting a job:

Job Completion Notification

Cloud Storage Authentication

AWS S3 and S3-Compatible Storage

Credentials are detected automatically from:

  1. Environment variables:

  1. AWS credential files (~/.aws/credentials, ~/.aws/config)

  2. IAM roles (EC2, ECS, Lambda)

Named profiles:

S3-compatible services (Cloudflare R2, MinIO, etc.):

Google Cloud Storage

Credentials are detected from:

  1. Service account key file (recommended for automation):

  1. User credentials from gcloud CLI (gcloud auth login)

  2. GCP metadata service (when running on Google Cloud Platform)

Azure Blob Storage

SAS Token (recommended):

Account Key:

Generate a SAS token via Azure CLI:

Custom Scripts

For advanced use cases, reference scripts for generating signed URL files:

Discover All Options

Last updated

Was this helpful?