> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/deployment/roboflow-cloud/batch-processing.md).

# Batch Processing

Run Workflows on large batches of images and stored videos with cloud infrastructure provisioned for you.

## About

Batch Processing is a cost-effective way to run [Workflows](https://docs.roboflow.com/workflows) on batches of images and stored videos. It's ideal for asynchronously processing large amounts of data.

Batch Processing automatically provisions the infrastructure needed to run a large batch.

{% hint style="info" %}
Batch Processing is available to every workspace on Roboflow's credit system or Enterprise plans. You can start a job from the "Batch Processing" tab, or run a Workflow on a selection in the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library).
{% endhint %}

You can configure a Batch Processing job through the Roboflow web interface or through our API (via the CLI).

When you start a job, machines will be provisioned in the cloud to process your data. You will then receive a JSON file with the output from the Workflow you chose to run on your data.

Jobs run on CPU or GPU machines. GPU jobs are faster but cost more. See the [credits page](https://roboflow.com/credits) for CPU and GPU rates. How long a job takes depends on how many images or videos it processes, how complex the Workflow is, and the machine type. For common issues, see [Troubleshooting](https://docs.roboflow.com/deployment/roboflow-cloud/batch-processing/troubleshooting.md).

The following video explains Batch Processing in depth:

{% embed url="<https://www.youtube.com/watch?v=S7K2j2IeQrM>" %}

## Web App

Open "Deployments" in the left sidebar and select the "Batch Processing" tab.

<figure><img src="https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-1193fa943b014f1b279c123ad5ccc88fac10f551%2FScreenshot%202025-05-19%20at%2012.14.51.png?alt=media" alt=""><figcaption></figcaption></figure>

Click "New Batch Job" and follow these steps:

1. Select a [Workflow](https://docs.roboflow.com/workflows).
2. Upload the images or videos to process.
3. Choose CPU or GPU hardware. Leave "Advanced Options" at their defaults unless you need to change them.
4. Click "Create Batch Job".

<figure><img src="https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-10667231b1e727c9344565e52171e0d1cb26b90c%2FScreenshot%202025-05-19%20at%2012.16.27.png?alt=media" alt=""><figcaption></figcaption></figure>

The job list shows each job's progress in real time. Jobs you start from this tab also appear in your Activity Center, where only you see them. Click a job to open its details. "Source and type" shows which images it ran on: the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library) search query that selected them, with an "Open in Asset Library" link, or the number of images picked manually.

### From the Asset Library

In the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library), select images, or select all images that match your search, then click "Run Workflow". See [Running a Workflow](https://docs.roboflow.com/platform/workspaces/asset-library#running-a-workflow) for the full flow, including how to write results back onto your images.

### From the Agent

Ask [Roboflow Agent](https://docs.roboflow.com/agents/roboflow-agent) to run a published Workflow on Asset Library images: images you pick, images that match a search, or the whole library. It can also report on your jobs, and it tells you in the chat when a job finishes. Jobs it starts also appear in Background Tasks.

The Agent can also open Batch Processing as a chat tab. In its new job form, the Agent fills in the Workflow and settings you agreed on, but you pick files from your computer yourself.

### Automatically When a Datasource Mirrors

Each time a [Datasource](https://docs.roboflow.com/datasets/create-and-upload/adding-data/datasources) mirrors, a Batch Processing job can run a Workflow over the new images it imported (ex: to keep tags or pre-labels current). In the "Workflow runs" section of the [Datasources](https://app.roboflow.com/settings/datasources) page, click "Add workflow run". Under "Run when", choose "On Sync" and select the Datasources that trigger it. The Workflow must have exactly one `image` input.

{% hint style="info" %}
If you do not see the "Workflow runs" section, ask a Workspace admin for a role that can manage batch automations.
{% endhint %}

To make the results searchable, see [Writing results back to the Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library#writing-results-back-to-the-asset-library).

## HTTP API

**Quick Links:**

* [Ingest Data](#ingest-data) (video, single image, images)
* [Check Batch Status](#check-batch-status) (item count, shard details)
* [Start a Job](#start-a-job)
* [Monitor Job Progress](#monitor-job-progress) (job status, stages, tasks)
* [Export Results](#export-results) (output parts, download URLs)
* [Webhook Notifications](#webhook-notifications)

### Ingest Data

#### Upload Video

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/upload/video" method="post" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### Upload Image

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/upload/image" method="post" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### Bulk Upload Images

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/bulk-upload/image-files" method="post" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

### Check Batch Status

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/count" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### Check Shard Upload Details

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/shards" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

### Start a Job

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/batch-processing/v1/external/{workspace}/jobs/{job\_id}" method="post" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

### Monitor Job Progress

#### Get Job Status

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/batch-processing/v1/external/{workspace}/jobs/{job\_id}" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### List Job Stages

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/batch-processing/v1/external/{workspace}/jobs/{job\_id}/stages" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### List Stage Tasks

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/batch-processing/v1/external/{workspace}/jobs/{job\_id}/stages/{stage\_id}/tasks" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

### Export Results

#### List Output Parts

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/parts" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

#### List Download URLs

{% openapi src="/files/wQobgR9eCvl08xtNitXf" path="/data-staging/v1/external/{workspace}/batches/{batch\_id}/list" method="get" %}
[batch-processing.yaml](https://1583372177-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoksNmwps1HmYr8TqHDnX%2Fuploads%2Fgit-blob-c01faa8d2b7705ff92e611d69a086a8d11d1c26a%2Fbatch-processing.yaml?alt=media)
{% endopenapi %}

### Webhook Notifications

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes. See [CLI Usage](#webhook-automation) for webhook configuration and payload formats.

## CLI

Use `inference rf-cloud` to stage and process local or cloud images and videos. For images already in your [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library), use `roboflow batch`.

### Local and Cloud Files

#### Setup

For local or cloud files, install the Inference CLI:

```bash
pip install inference-cli
export ROBOFLOW_API_KEY="YOUR-API-KEY-GOES-HERE"
```

For cloud storage support:

```bash
pip install 'inference-cli[cloud-storage]'
```

If you need help finding your API key, see our [authentication guide](https://docs.roboflow.com/reference/authentication/authentication/find-your-roboflow-api-key).

#### Ingest Data

**Images**

```bash
inference rf-cloud data-staging create-batch-of-images \
  --images-dir <your-images-dir-path> \
  --batch-id <your-batch-id>
```

**Videos**

```bash
inference rf-cloud data-staging create-batch-of-videos \
  --videos-dir <your-videos-dir-path> \
  --batch-id <your-batch-id>
```

{% hint style="info" %}
Use a lowercase batch ID with at most 64 characters: letters, digits, hyphens (`-`), and underscores (`_`).
{% endhint %}

**Cloud Storage**

If your data is already in cloud storage (S3, Google Cloud Storage, or Azure), you can process it directly without downloading files locally.

For images:

```bash
inference rf-cloud data-staging create-batch-of-images \
  --data-source cloud-storage \
  --bucket-path <cloud-path> \
  --batch-id <your-batch-id>
```

For videos:

```bash
inference rf-cloud data-staging create-batch-of-videos \
  --data-source cloud-storage \
  --bucket-path <cloud-path> \
  --batch-id <your-batch-id>
```

The `--bucket-path` parameter supports the following providers. Glob patterns filter which files are ingested:

<table data-search="false"><thead><tr><th>Provider</th><th>Path format</th><th>Glob example</th></tr></thead><tbody><tr><td>S3</td><td><code>s3://bucket-name/path/</code></td><td><code>s3://my-bucket/training-data/**/*.jpg</code>: all JPGs recursively</td></tr><tr><td>Google Cloud Storage</td><td><code>gs://bucket-name/path/</code></td><td><code>gs://my-bucket/videos/2024-*/*.mp4</code>: MP4s in <code>2024-*</code> folders</td></tr><tr><td>Azure Blob Storage</td><td><code>az://container-name/path/</code></td><td><code>az://container/images/*.png</code>: PNGs in the images folder</td></tr></tbody></table>

{% hint style="info" %}
Your cloud storage credentials are used only locally by the CLI to generate presigned URLs. They are never uploaded to Roboflow servers.
{% endhint %}

{% hint style="warning" %}
Generated presigned URLs are valid for 24 hours. Ensure your batch processing job completes within this timeframe.
{% endhint %}

For large datasets, the system automatically splits images into chunks of 20,000 files each. Videos work best in batches under 1,000.

**Signed URL Ingestion**

For advanced automation, you can ingest data via signed URLs instead of local files:

<table data-search="false"><thead><tr><th>Flag</th><th>Description</th></tr></thead><tbody><tr><td><code>--data-source references-file</code></td><td>Process files referenced via signed URLs.</td></tr><tr><td><code>--references &#x3C;path_or_url></code></td><td>Path to a JSONL file containing file URLs, or a signed URL pointing to such a file.</td></tr></tbody></table>

Reference files use `JSONL`:

```
{"name": "<unique-file-name-1>", "url": "https://<signed-url>"}
{"name": "<unique-file-name-2>", "url": "https://<signed-url>"}
```

{% hint style="info" %}
Signed URL ingestion is available on Enterprise and the earlier Growth plan.
{% endhint %}

#### Inspect Staged Data

```bash
inference rf-cloud data-staging show-batch-details --batch-id <your-batch-id>
```

#### Start a Job

**Process Images**

```bash
inference rf-cloud batch-processing process-images-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --machine-type gpu
```

**Process Videos**

```bash
inference rf-cloud batch-processing process-videos-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --machine-type gpu \
  --max-video-fps <your-desired-fps>
```

{% hint style="info" %}
To find your Workflow ID, open the Workflow Editor in the Roboflow App, click "Deploy", and find the identifier in the code snippet.
{% endhint %}

{% hint style="info" %}
By default, processing runs on CPU. Use `--machine-type gpu` for Workflows with multiple or large models.
{% endhint %}

#### Monitor Job Progress

The start command outputs a job ID. Use it to check status:

```bash
inference rf-cloud batch-processing show-job-details --job-id <your-job-id>
```

#### Export Results

The job details will include the output batch ID. Use it to export results:

```bash
inference rf-cloud data-staging export-batch \
  --target-dir <dir-to-export-result> \
  --batch-id <output-batch-of-a-job>
```

#### Webhook Automation

Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes.

**Data Ingestion Webhooks**

The CLI commands `create-batch-of-images` and `create-batch-of-videos` support:

<table data-search="false"><thead><tr><th>Flag</th><th>Description</th></tr></thead><tbody><tr><td><code>--notifications-url &#x3C;webhook_url></code></td><td>Webhook endpoint for notifications.</td></tr><tr><td><code>--notification-category ingest-status</code></td><td>Overall ingestion process status. Default.</td></tr><tr><td><code>--notification-category files-status</code></td><td>Individual file processing status.</td></tr></tbody></table>

Notifications are delivered via HTTP POST with an `Authorization` header containing your Roboflow Publishable Key.

**Ingest Status Notification**

```json
{
    "type": "roboflow-data-staging-notification-v1",
    "event_id": "8c20f970-fe10-41e1-9ef2-e057c63c07ff",
    "ingest_id": "8cd48813430f2be70b492db67e07cc86",
    "batch_id": "test-batch-117",
    "shard_id": null,
    "notification": {
        "type": "ingest-status-notification-v1",
        "success": false,
        "error_details": {
            "type": "unsafe-url-detected",
            "reason": "Untrusted domain found: https://example.com/image.png"
        }
    },
    "delivery_attempt": 1
}
```

**File Status Notification**

```json
{
    "type": "roboflow-data-staging-notification-v1",
    "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81",
    "ingest_id": "d5cb69aa-b2d1-4202-a1c1-0231f180bda9",
    "batch_id": "prod-batch-1",
    "shard_id": "0d40fa12-349e-439f-83f8-42b9b7987b33",
    "notification": {
        "type": "ingest-files-status-notification-v1",
        "success": true,
        "ingested_files": [
            "000000494869.jpg",
            "000000186042.jpg"
        ],
        "failed_files": [
            {
                "type": "file-size-limit-exceeded",
                "file_name": "big_image.png",
                "reason": "Max size of single image is 20971520B."
            }
        ],
        "content_truncated": false
    },
    "delivery_attempt": 1
}
```

**Job Completion Webhooks**

Add `--notifications-url` when starting a job:

```bash
inference rf-cloud batch-processing process-images-with-workflow \
  --workflow-id <workflow-id> \
  --batch-id <batch-id> \
  --notifications-url <webhook_url>
```

**Job Completion Notification**

```json
{
  "type": "roboflow-batch-job-notification-v1",
  "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81",
  "job_id": "<your-batch-job-id>",
  "job_state": "success | fail",
  "delivery_attempt": 1
}
```

#### Cloud Storage Authentication

**AWS S3 and S3-Compatible Storage**

Credentials are detected automatically from:

1. Environment variables:

```bash
export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key
export AWS_SESSION_TOKEN=your-session-token  # Optional
```

2. AWS credential files (`~/.aws/credentials`, `~/.aws/config`)
3. IAM roles (EC2, ECS, Lambda)

Named profiles:

```bash
export AWS_PROFILE=production
```

For S3-compatible services (ex: Cloudflare R2, MinIO):

```bash
export AWS_ENDPOINT_URL=https://account-id.r2.cloudflarestorage.com
export AWS_REGION=auto  # R2 requires region='auto'
export AWS_ACCESS_KEY_ID=your-r2-access-key
export AWS_SECRET_ACCESS_KEY=your-r2-secret-key
```

**Google Cloud Storage**

Credentials are detected from:

1. Service account key file (recommended for automation):

```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
```

2. User credentials from gcloud CLI (`gcloud auth login`)
3. GCP metadata service (when running on Google Cloud Platform)

**Azure Blob Storage**

SAS token (recommended):

```bash
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_SAS_TOKEN="YOUR_SAS_TOKEN"
```

Account key:

```bash
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=your-account-key
```

Generate a SAS token with the Azure CLI. Replace `<expiry-utc>` with a future UTC time in `YYYY-MM-DDTHH:MMZ` format:

```bash
az storage container generate-sas \
  --account-name mystorageaccount \
  --name my-container \
  --permissions rl \
  --expiry <expiry-utc>
```

**Custom Scripts**

For advanced use cases, reference scripts for generating signed URL files:

* AWS S3: [generateS3SignedUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateS3SignedUrls.sh)
* Google Cloud Storage: [generateGCSSignedUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateGCSSignedUrls.sh)
* Azure Blob Storage: [generateAzureSasUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateAzureSasUrls.sh)

#### Discover All Options

```bash
inference rf-cloud --help
inference rf-cloud data-staging --help
inference rf-cloud batch-processing --help
```

### Asset Library Images

Replace `<workspace-url>` with your Workspace slug (ex: `my-workspace`). Choose a published Workflow with one `image` input. Give other required inputs defaults, except `source_id`, which Batch Processing fills for each image.

Install or update the [Roboflow CLI](https://docs.roboflow.com/reference/platform/cli), sign in, and list your Workflows:

```bash
pip install --upgrade roboflow
roboflow auth login
roboflow --workspace <workspace-url> --json workflow list
```

Use a published Workflow's `id` from the JSON output. Find the images to process with a [RoboQL](https://docs.roboflow.com/platform/workspaces/asset-library) search:

```bash
roboflow --workspace <workspace-url> --json image search "tag:night-shift" \
  --fields id,filename
```

Pass the image IDs from the search results to `--image-ids`:

```bash
roboflow --workspace <workspace-url> batch create \
  --workflow <workflow-id> \
  --image-ids <image-id-1>,<image-id-2>
```

To process all images that match a structured filter, replace `--image-ids` with `--query "tag:night-shift"`. Use `--all` to process the entire Asset Library. Choose exactly one selection option.

The create response includes `taskId`, `jobId`, and `requestId`. Use the task ID to wait for completion and the job ID to inspect the job:

```bash
roboflow --workspace <workspace-url> asynctasks wait <task-id>
roboflow --workspace <workspace-url> batch status <job-id>
roboflow --workspace <workspace-url> batch list
```

{% hint style="info" %}
A job keeps running if you close the terminal or the wait command times out. After a network error, retry with the same options. Pass the `requestId` from the response or error hint as `--request-id <request-id>` to avoid creating a duplicate job.
{% endhint %}

Use `batch abort <job-id>` to stop a job or `batch restart <job-id>` to retry it with its existing configuration. Both commands ask for confirmation; add `--yes` for scripts. A restart can consume credits.
