> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/deployment/roboflow-cloud/batch-processing.md).

# Batch Processing

## About

Batch Processing is a cost-effective way to run [Workflows](https://docs.roboflow.com/workflows) on batches of images and stored videos. It's ideal for asynchronously processing large amounts of data.

Batch Processing automatically provisions the infrastructure needed to run a large batch.

{% hint style="info" %}
Batch Processing is available on Growth and Enterprise plans. You can start a job from the "Batch Processing" tab, or run a Workflow on a selection in the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library).
{% endhint %}

You can configure a Batch Processing job through the Roboflow web interface or through our API (via the CLI).

When you start a job, machines will be provisioned in the cloud to process your data. You will then receive a JSON file with the output from the Workflow you chose to run on your data.

The following video explains Batch Processing in depth:

{% embed url="<https://www.youtube.com/watch?v=S7K2j2IeQrM>" %}

## Web App

### Create a Batch Processing Job

To create a Batch Processing job, click Deployments in the left sidebar of your Roboflow dashboard. Then, click on the "Batch Processing" tab:

<figure><img src="/files/mMsiAqydMiK2FezgoQwA" alt=""><figcaption></figcaption></figure>

Click "New Batch Job" to create a Batch Processing job.

A window will open in which you can configure your job:

<figure><img src="/files/sadh7iEmJFvIsd3jkApM" alt=""><figcaption></figcaption></figure>

#### Choose a Workflow

To start configuring a job, first select a Workflow. If you do not already have a Workflow, refer to our Workflows documentation to get started.

#### Upload Images or Videos

Next, you need to upload the images or videos on which you want to run your Workflow.

#### Configure Hardware

You can run your Batch Processing job on a CPU or a GPU. GPU jobs are faster but more expensive.

For pricing information, refer to the Roboflow pricing documentation.

Select either a CPU or GPU for your job:

<figure><img src="/files/c9feJZm73QN5CsPkKUNn" alt=""><figcaption></figcaption></figure>

Several advanced configuration options are also available under the "Advanced Options" tab. We recommend leaving these options as the default.

#### Start the Job

To start the Batch Processing job, click "Create Batch Job".

The infrastructure for your job will be provisioned and processing will begin.

### Monitor Job Progress

When you start your job, a status indicator will appear indicating when processing is being configured, when the batch data is being processed, and when the job is complete.

You can monitor how much of a batch has been processed in real time.

The amount of time it will take to process your data depends on how many images or videos you are processing, the complexity of your Workflow, and whether you selected CPU or GPU hardware.

Open a job to view its details, including the "Input Source" that shows which images the job ran on: the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library) search query used to select them (with a link to reopen that selection), or the number of images picked manually.

### Run a Workflow from the App

Besides the [API](#http-api) and [CLI](#cli), you can start a Batch Processing job directly from the Roboflow app to run a [Workflow](https://docs.roboflow.com/workflows) over large sets of stored images. There are two ways to do this:

* On demand, from the Asset Library.
* Automatically, each time a [Datasource](https://docs.roboflow.com/datasets/create-and-upload/adding-data/datasources) mirrors new images from a cloud bucket.

#### From the Asset Library

The [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library) lets you run a Workflow on the images you select, on demand.

Select images manually, or select all images matching your current search, then click "Run Workflow". For the full flow, including how to write results back onto your images, see [Running a Workflow](https://docs.roboflow.com/platform/workspaces/asset-library#running-a-workflow).

#### Automatically When a Datasource Mirrors

A [Datasource](https://docs.roboflow.com/datasets/create-and-upload/adding-data/datasources) mirrors images and metadata from a cloud bucket into your Workspace. You can automatically run a Workflow over the new images each mirror imports. This keeps enrichment such as tagging, quality scoring, or pre-labeling up to date as new data arrives, with no manual step.

You configure these automations in the "Workflow runs" section of your [Datasources](https://app.roboflow.com/settings/datasources) page.

{% hint style="info" %}
Managing Workflow runs requires a Workspace role with permission to manage batch automations. If you do not see the "Workflow runs" section, ask a Workspace admin.
{% endhint %}

To add an automation:

1. In the "Workflow runs" section, click "Add workflow run".
2. Enter a Name for the automation.
3. Under "Run when", choose "On Sync" and select the [Datasources](https://docs.roboflow.com/datasets/create-and-upload/adding-data/datasources) that should trigger it.
4. Select the Workflow to run. It must have exactly one `image` input.
5. Choose a Machine type (CPU or GPU).
6. Click "Create".

Each time one of the selected Datasources mirrors, the automation runs the Workflow as a Batch Processing job over the images that mirror imported. Track progress in the Activity Center and on the "Batch Processing" tab under Deployments, the same as any other Batch Processing job.

To have a Workflow write its results back onto your images so you can search for them in the Asset Library, see [Writing results back to the Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library#writing-results-back-to-the-asset-library).

### Run a Job with the API or CLI

To create and run a Batch Processing job programmatically, see the [HTTP API](#http-api) and [CLI](#cli) sections below. For debugging common issues, see [Troubleshooting](/deployment/roboflow-cloud/batch-processing/troubleshooting.md).

## HTTP API

**Quick Links:**

* [Ingest Data](#ingest-data) (video, single image, images)
* [Check Batch Status](#check-batch-status) (item count, shard details)
* [Start a Job](#start-a-job)
* [Monitor Job Progress](#monitor-job-progress) (job status, stages, tasks)
* [Export Results](#export-results) (output parts, download URLs)
* [Webhook Notifications](#webhook-notifications)

### Ingest Data

#### Upload Video

## Upload a video

> Request a signed URL to upload a video file. After receiving the response, PUT the video to the \`uploadURL\` with the provided \`extensionHeaders\`.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/video":{"post":{"operationId":"uploadVideo","tags":["Data Ingestion"],"summary":"Upload a video","description":"Request a signed URL to upload a video file. After receiving the response, PUT the video to the `uploadURL` with the provided `extensionHeaders`.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"name":"fileName","in":"query","required":true,"schema":{"type":"string"},"description":"Name of the video file (e.g. `my_video.mp4`)."}],"responses":{"200":{"description":"Signed URL details for uploading the video.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"signedURLDetails":{"type":"object","properties":{"uploadURL":{"type":"string","description":"PUT the video file to this URL."},"method":{"type":"string","description":"HTTP method to use for the upload."},"extensionHeaders":{"type":"object","additionalProperties":{"type":"string"},"description":"Include these headers in the PUT request."},"maxFileSize":{"type":"integer","description":"Maximum file size in bytes."}}}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### Upload Image

## Upload a single image

> Upload a single image via multipart form data. Best for batches up to 5,000 images.\
> \
> \*\*Note:\*\* Single-image and bulk uploads cannot be combined for the same batch.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/image":{"post":{"operationId":"uploadImage","tags":["Data Ingestion"],"summary":"Upload a single image","description":"Upload a single image via multipart form data. Best for batches up to 5,000 images.\n\n**Note:** Single-image and bulk uploads cannot be combined for the same batch.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"name":"fileName","in":"query","required":true,"schema":{"type":"string"},"description":"Name of the image file."}],"requestBody":{"required":true,"content":{"multipart/form-data":{"schema":{"type":"object","properties":{"file":{"type":"string","format":"binary","description":"The image file to upload."}}}}}},"responses":{"200":{"description":"Image uploaded successfully.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### Bulk Upload Images

## Bulk upload images

> Request a signed URL for uploading a \`.tar\` archive of images. Recommended for batches exceeding 5,000 images. Bundle up to 500 images per archive.\
> \
> The response contains a signed URL and extension headers. Pack images into a \`.tar\` archive and PUT it to the signed URL.\
> \
> \*\*Note:\*\* Bulk and single-image uploads cannot be combined for the same batch.\
> \
> When performing bulk ingestion, data is indexed in the background. There may be a short delay before all data is available.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/bulk-upload/image-files":{"post":{"operationId":"bulkUploadImages","tags":["Data Ingestion"],"summary":"Bulk upload images","description":"Request a signed URL for uploading a `.tar` archive of images. Recommended for batches exceeding 5,000 images. Bundle up to 500 images per archive.\n\nThe response contains a signed URL and extension headers. Pack images into a `.tar` archive and PUT it to the signed URL.\n\n**Note:** Bulk and single-image uploads cannot be combined for the same batch.\n\nWhen performing bulk ingestion, data is indexed in the background. There may be a short delay before all data is available.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Signed URL details for uploading a tar archive.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"signedURLDetails":{"type":"object","properties":{"shardId":{"type":"string","format":"uuid","description":"Unique identifier for this shard upload."},"uploadURL":{"type":"string","description":"PUT the tar archive to this URL."},"method":{"type":"string","description":"HTTP method to use for the upload."},"extensionHeaders":{"type":"object","additionalProperties":{"type":"string"},"description":"Include these headers in the PUT request."},"maxNumberOfImages":{"type":"integer","description":"Maximum number of images per tar archive."},"maxShardSize":{"type":"integer","description":"Maximum tar archive size in bytes."}}}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

### Check Batch Status

## Get batch item count

> Returns the count of ingested items in a batch. Use this to verify all data has been ingested before starting a job.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Batch Status","description":"Inspect staged data and batch contents."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/count":{"get":{"operationId":"getBatchCount","tags":["Batch Status"],"summary":"Get batch item count","description":"Returns the count of ingested items in a batch. Use this to verify all data has been ingested before starting a job.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Batch item count.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"count":{"type":"integer","description":"Number of items in the batch."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### Check Shard Upload Details

## List batch shards

> Returns shard details for a bulk-upload batch. Paginated — use \`nextPageToken\` from the response to fetch subsequent pages.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Batch Status","description":"Inspect staged data and batch contents."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/shards":{"get":{"operationId":"getBatchShards","tags":["Batch Status"],"summary":"List batch shards","description":"Returns shard details for a bulk-upload batch. Paginated — use `nextPageToken` from the response to fetch subsequent pages.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"}],"responses":{"200":{"description":"Paginated list of batch shards.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"shards":{"type":"array","items":{"type":"object"},"description":"List of shard objects."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}}
```

### Start a Job

## Start a batch processing job

> Start a batch processing job that runs a Workflow against staged data.\
> \
> \*\*Job ID constraints:\*\* Lowercase letters, digits, hyphens, and underscores only. Maximum 20 characters.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}":{"post":{"operationId":"startJob","tags":["Processing"],"summary":"Start a batch processing job","description":"Start a batch processing job that runs a Workflow against staged data.\n\n**Job ID constraints:** Lowercase letters, digits, hyphens, and underscores only. Maximum 20 characters.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/JobStartRequest"}}}},"responses":{"200":{"description":"Job started successfully.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}},"schemas":{"JobStartRequest":{"type":"object","required":["type","jobInput","computeConfiguration","processingSpecification"],"properties":{"type":{"type":"string","enum":["simple-image-processing-v1"],"description":"Job type."},"jobInput":{"type":"object","required":["type","batchId"],"properties":{"type":{"type":"string","enum":["staging-batch-input-v1"],"description":"Input type."},"batchId":{"type":"string","description":"The batch ID containing the data to process."}}},"computeConfiguration":{"type":"object","required":["type","machineType"],"properties":{"type":{"type":"string","enum":["compute-configuration-v2"],"description":"Configuration type."},"machineType":{"type":"string","enum":["cpu","gpu"],"description":"Machine type. Use `gpu` for Workflows with multiple or large models."},"workersPerMachine":{"type":"integer","default":4,"description":"Number of parallel workers per machine. Reduce for memory-intensive Workflows."}}},"processingTimeoutSeconds":{"type":"integer","default":3600,"description":"Maximum cumulative machine runtime in seconds across all parallel workers."},"processingSpecification":{"type":"object","required":["type","workspace","workflowId"],"properties":{"type":{"type":"string","enum":["workflows-processing-specification-v1"],"description":"Processing specification type."},"workspace":{"type":"string","description":"Workspace containing the Workflow."},"workflowId":{"type":"string","description":"The Workflow to run. Find this in the Workflow Editor under \"Deploy\"."},"aggregationFormat":{"type":"string","enum":["jsonl","csv"],"default":"jsonl","description":"Output format for aggregated results."}}},"notificationsURL":{"type":"string","format":"uri","description":"Webhook URL for job completion notifications. Custom webhook headers are not yet supported. The only header sent is `Authorization: Bearer rf_{workspace_id}`."}}}}}}
```

### Monitor Job Progress

#### Get Job Status

## Get job status

> Returns the current status of a batch processing job.

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}":{"get":{"operationId":"getJobStatus","tags":["Processing"],"summary":"Get job status","description":"Returns the current status of a batch processing job.","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Job status details.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"jobStatus":{"type":"string","description":"Current job status (e.g. `pending`, `processing`, `completed`, `failed`)."},"progress":{"type":"number","description":"Processing progress as a fraction between 0 and 1."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### List Job Stages

## List job stages

> Returns the list of stages for a job. Each job typically has \`processing\` and \`export\` stages, each producing an output batch.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages":{"get":{"operationId":"getJobStages","tags":["Processing"],"summary":"List job stages","description":"Returns the list of stages for a job. Each job typically has `processing` and `export` stages, each producing an output batch.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"List of job stages.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"stages":{"type":"array","items":{"type":"object"},"description":"List of stage objects. Each stage has an ID and an output batch ID."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### List Stage Tasks

## List tasks for a stage

> Returns the list of tasks for a specific job stage. Paginated — use \`nextPageToken\` from the response to fetch subsequent pages.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages/{stage_id}/tasks":{"get":{"operationId":"getStageTasks","tags":["Processing"],"summary":"List tasks for a stage","description":"Returns the list of tasks for a specific job stage. Paginated — use `nextPageToken` from the response to fetch subsequent pages.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"name":"stage_id","in":"path","required":true,"schema":{"type":"string"},"description":"The stage identifier."},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"}],"responses":{"200":{"description":"Paginated list of tasks.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"tasks":{"type":"array","items":{"type":"object"},"description":"List of task objects."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}}
```

### Export Results

#### List Output Parts

## List output batch parts

> Lists the parts of an output batch. Use the \`export\` stage output batch for compressed results.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Export","description":"Download results after processing completes."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/parts":{"get":{"operationId":"listBatchParts","tags":["Data Export"],"summary":"List output batch parts","description":"Lists the parts of an output batch. Use the `export` stage output batch for compressed results.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"List of batch parts.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"parts":{"type":"array","items":{"type":"object","properties":{"partName":{"type":"string","description":"Name of the batch part."}}},"description":"List of batch part objects."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}}
```

#### List Download URLs

## List download URLs

> Returns paginated download URLs for files in a batch part.<br>

```json
{"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Export","description":"Download results after processing completes."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/list":{"get":{"operationId":"listDownloadUrls","tags":["Data Export"],"summary":"List download URLs","description":"Returns paginated download URLs for files in a batch part.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"},{"name":"partName","in":"query","schema":{"type":"string"},"description":"Filter by part name (from the list parts response)."}],"responses":{"200":{"description":"Paginated list of download URLs.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"filesMetadata":{"type":"array","items":{"type":"object","properties":{"downloadURL":{"type":"string","description":"Signed URL to download the file."},"fileName":{"type":"string","description":"Original file name."},"partName":{"type":"string","nullable":true,"description":"Part name this file belongs to."},"shardId":{"type":"string","nullable":true,"description":"Shard ID (for bulk-upload batches)."},"contentType":{"type":"string","description":"Content type (e.g. `image`, `video`)."},"nestedContentType":{"type":"string","nullable":true,"description":"Nested content type, if applicable."}}},"description":"List of file metadata objects with download URLs."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}}
```

### Webhook Notifications

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes. See [CLI Usage](#webhook-automation) for webhook configuration and payload formats.

## CLI

By installing `inference-cli` you gain access to the `inference rf-cloud` command, which allows you to interact with Batch Processing and Data Staging - the core components of Roboflow Batch Processing.

### Setup

```bash
pip install inference-cli
export ROBOFLOW_API_KEY="YOUR-API-KEY-GOES-HERE"
```

For cloud storage support:

```bash
pip install 'inference-cli[cloud-storage]'
```

If you need help finding your API key, see our [authentication guide](https://docs.roboflow.com/reference/authentication/authentication/find-your-roboflow-api-key).

### Ingest Data

#### Images

```bash
inference rf-cloud data-staging create-batch-of-images \
  --images-dir <your-images-dir-path> \
  --batch-id <your-batch-id>
```

#### Videos

```bash
inference rf-cloud data-staging create-batch-of-videos \
  --videos-dir <your-videos-dir-path> \
  --batch-id <your-batch-id>
```

{% hint style="info" %}
**Batch ID format:** Must be lowercase, at most 64 characters, with only letters, digits, hyphens (`-`), and underscores (`_`).
{% endhint %}

#### Cloud Storage

If your data is already in cloud storage (S3, Google Cloud Storage, or Azure), you can process it directly without downloading files locally.

**For images:**

```bash
inference rf-cloud data-staging create-batch-of-images \
  --data-source cloud-storage \
  --bucket-path <cloud-path> \
  --batch-id <your-batch-id>
```

**For videos:**

```bash
inference rf-cloud data-staging create-batch-of-videos \
  --data-source cloud-storage \
  --bucket-path <cloud-path> \
  --batch-id <your-batch-id>
```

The `--bucket-path` parameter supports the following providers. Glob patterns filter which files are ingested:

| Provider             | Path format                 | Glob example                                                    |
| -------------------- | --------------------------- | --------------------------------------------------------------- |
| S3                   | `s3://bucket-name/path/`    | `s3://my-bucket/training-data/**/*.jpg` - all JPGs recursively  |
| Google Cloud Storage | `gs://bucket-name/path/`    | `gs://my-bucket/videos/2024-*/*.mp4` - MP4s in `2024-*` folders |
| Azure Blob Storage   | `az://container-name/path/` | `az://container/images/*.png` - PNGs in the images folder       |

{% hint style="info" %}
Your cloud storage credentials are used **only locally** by the CLI to generate presigned URLs. They are **never uploaded** to Roboflow servers.
{% endhint %}

{% hint style="warning" %}
Generated presigned URLs are valid for 24 hours. Ensure your batch processing job completes within this timeframe.
{% endhint %}

For large datasets, the system automatically splits images into chunks of 20,000 files each. Videos work best in batches under 1,000.

#### Signed URL Ingestion

For advanced automation, you can ingest data via signed URLs instead of local files:

| Flag                            | Description                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------- |
| `--data-source references-file` | Process files referenced via signed URLs.                                           |
| `--references <path_or_url>`    | Path to a JSONL file containing file URLs, or a signed URL pointing to such a file. |

**Reference File Format (JSONL):**

```
{"name": "<unique-file-name-1>", "url": "https://<signed-url>"}
{"name": "<unique-file-name-2>", "url": "https://<signed-url>"}
```

{% hint style="info" %}
Signed URL ingestion is available to Growth Plan and Enterprise customers.
{% endhint %}

### Inspect Staged Data

```bash
inference rf-cloud data-staging show-batch-details --batch-id <your-batch-id>
```

### Start a Job

#### Process Images

```bash
inference rf-cloud batch-processing process-images-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --machine-type gpu
```

#### Process Videos

```bash
inference rf-cloud batch-processing process-videos-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --machine-type gpu \
  --max-video-fps <your-desired-fps>
```

{% hint style="info" %}
**Finding your Workflow ID:** Open the Workflow Editor in the Roboflow App, click "Deploy", and find the identifier in the code snippet.
{% endhint %}

{% hint style="info" %}
By default, processing runs on CPU. Use `--machine-type gpu` for Workflows with multiple or large models.
{% endhint %}

### Monitor Job Progress

The start command outputs a **Job ID**. Use it to check status:

```bash
inference rf-cloud batch-processing show-job-details --job-id <your-job-id>
```

### Export Results

The job details will include the **output batch ID**. Use it to export results:

```bash
inference rf-cloud data-staging export-batch \
  --target-dir <dir-to-export-result> \
  --batch-id <output-batch-of-a-job>
```

### Webhook Automation

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes.

#### Data Ingestion Webhooks

The CLI commands `create-batch-of-images` and `create-batch-of-videos` support:

| Flag                                    | Description                                |
| --------------------------------------- | ------------------------------------------ |
| `--notifications-url <webhook_url>`     | Webhook endpoint for notifications.        |
| `--notification-category ingest-status` | Overall ingestion process status. Default. |
| `--notification-category files-status`  | Individual file processing status.         |

Notifications are delivered via HTTP POST with an `Authorization` header containing your Roboflow Publishable Key.

**Ingest Status Notification**

```json
{
    "type": "roboflow-data-staging-notification-v1",
    "event_id": "8c20f970-fe10-41e1-9ef2-e057c63c07ff",
    "ingest_id": "8cd48813430f2be70b492db67e07cc86",
    "batch_id": "test-batch-117",
    "shard_id": null,
    "notification": {
        "type": "ingest-status-notification-v1",
        "success": false,
        "error_details": {
            "type": "unsafe-url-detected",
            "reason": "Untrusted domain found: https://example.com/image.png"
        }
    },
    "delivery_attempt": 1
}
```

**File Status Notification**

```json
{
    "type": "roboflow-data-staging-notification-v1",
    "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81",
    "ingest_id": "d5cb69aa-b2d1-4202-a1c1-0231f180bda9",
    "batch_id": "prod-batch-1",
    "shard_id": "0d40fa12-349e-439f-83f8-42b9b7987b33",
    "notification": {
        "type": "ingest-files-status-notification-v1",
        "success": true,
        "ingested_files": [
            "000000494869.jpg",
            "000000186042.jpg"
        ],
        "failed_files": [
            {
                "type": "file-size-limit-exceeded",
                "file_name": "big_image.png",
                "reason": "Max size of single image is 20971520B."
            }
        ],
        "content_truncated": false
    },
    "delivery_attempt": 1
}
```

#### Job Completion Webhooks

Add `--notifications-url` when starting a job:

```bash
inference rf-cloud batch-processing process-images-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --notifications-url <webhook_url>
```

**Job Completion Notification**

```json
{
  "type": "roboflow-batch-job-notification-v1",
  "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81",
  "job_id": "<your-batch-job-id>",
  "job_state": "success | fail",
  "delivery_attempt": 1
}
```

### Cloud Storage Authentication

#### AWS S3 and S3-Compatible Storage

Credentials are detected automatically from:

1. **Environment variables:**

```bash
export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key
export AWS_SESSION_TOKEN=your-session-token  # Optional
```

2. **AWS credential files** (`~/.aws/credentials`, `~/.aws/config`)
3. **IAM roles** (EC2, ECS, Lambda)

**Named profiles:**

```bash
export AWS_PROFILE=production
```

**S3-compatible services (Cloudflare R2, MinIO, etc.):**

```bash
export AWS_ENDPOINT_URL=https://account-id.r2.cloudflarestorage.com
export AWS_REGION=auto  # R2 requires region='auto'
export AWS_ACCESS_KEY_ID=your-r2-access-key
export AWS_SECRET_ACCESS_KEY=your-r2-secret-key
```

#### Google Cloud Storage

Credentials are detected from:

1. **Service account key file** (recommended for automation):

```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
```

2. **User credentials** from gcloud CLI (`gcloud auth login`)
3. **GCP metadata service** (when running on Google Cloud Platform)

#### Azure Blob Storage

**SAS Token (recommended):**

```bash
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_SAS_TOKEN="sv=2021-06-08&ss=b&srt=sco&sp=rl&se=2024-12-31"
```

**Account Key:**

```bash
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=your-account-key
```

Generate a SAS token via Azure CLI:

```bash
az storage container generate-sas \
  --account-name mystorageaccount \
  --name my-container \
  --permissions rl \
  --expiry 2024-12-31T23:59:59Z
```

#### Custom Scripts

For advanced use cases, reference scripts for generating signed URL files:

* **AWS S3:** [generateS3SignedUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateS3SignedUrls.sh)
* **Google Cloud Storage:** [generateGCSSignedUrls.sh](https://github.com/roboflow/roboflow-python/blob/main/scripts/generateGCSSignedUrls.sh)
* **Azure Blob Storage:** [generateAzureSasUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateAzureSasUrls.sh)

### Discover All Options

```bash
inference rf-cloud --help
inference rf-cloud data-staging --help
inference rf-cloud batch-processing --help
```