> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/deployment/roboflow-cloud/batch-processing.md). # Batch Processing Run Workflows on large batches of images and stored videos with cloud infrastructure provisioned for you. ## About Batch Processing is a cost-effective way to run [Workflows](https://docs.roboflow.com/workflows) on batches of images and stored videos. It's ideal for asynchronously processing large amounts of data. Batch Processing automatically provisions the infrastructure needed to run a large batch. {% hint style="info" %} Batch Processing is available to every workspace on Roboflow's credit system or Enterprise plans. You can start a job from the "Batch Processing" tab, or run a Workflow on a selection in the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library). {% endhint %} You can configure a Batch Processing job through the Roboflow web interface or through our API (via the CLI). When you start a job, machines will be provisioned in the cloud to process your data. You will then receive a JSON file with the output from the Workflow you chose to run on your data. Jobs run on CPU or GPU machines. GPU jobs are faster but cost more; see [Roboflow pricing](https://roboflow.com/pricing). How long a job takes depends on how many images or videos it processes, how complex the Workflow is, and the machine type. For common issues, see [Troubleshooting](/deployment/roboflow-cloud/batch-processing/troubleshooting.md). The following video explains Batch Processing in depth: {% embed url="" %} ## Web App Open "Deployments" in the left sidebar and select the "Batch Processing" tab.
Click "New Batch Job" and follow these steps: 1. Select a [Workflow](https://docs.roboflow.com/workflows). 2. Upload the images or videos to process. 3. Choose CPU or GPU hardware. Leave "Advanced Options" at their defaults unless you need to change them. 4. Click "Create Batch Job".
The job list shows each job's progress in real time. Jobs you start from this tab also appear in your Activity Center, where only you see them. A job's "Input Source" shows which images it ran on: the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library) search query that selected them, with a link to reopen that selection, or the number of images picked manually. ### From the Asset Library In the [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library), select images, or select all images that match your search, then click "Run Workflow". See [Running a Workflow](https://docs.roboflow.com/platform/workspaces/asset-library#running-a-workflow) for the full flow, including how to write results back onto your images. ### From the Agent Ask [Roboflow Agent](https://docs.roboflow.com/agents/roboflow-agent) to run a published Workflow on Asset Library images: images you pick, images that match a search, or the whole library. It can also report on your jobs, and it tells you in the chat when a job finishes. Jobs it starts also appear in Background Tasks. The Agent can also open Batch Processing as a chat tab. In its new job form, the Agent fills in the Workflow and settings you agreed on, but you pick files from your computer yourself. ### Automatically When a Datasource Mirrors Each time a [Datasource](https://docs.roboflow.com/datasets/create-and-upload/adding-data/datasources) mirrors, a Batch Processing job can run a Workflow over the new images it imported (ex: to keep tags or pre-labels current). In the "Workflow runs" section of the [Datasources](https://app.roboflow.com/settings/datasources) page, click "Add workflow run". Under "Run when", choose "On Sync" and select the Datasources that trigger it. The Workflow must have exactly one `image` input. {% hint style="info" %} If you do not see the "Workflow runs" section, ask a Workspace admin for a role that can manage batch automations. {% endhint %} To make the results searchable, see [Writing results back to the Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library#writing-results-back-to-the-asset-library). ## HTTP API **Quick Links:** * [Ingest Data](#ingest-data) (video, single image, images) * [Check Batch Status](#check-batch-status) (item count, shard details) * [Start a Job](#start-a-job) * [Monitor Job Progress](#monitor-job-progress) (job status, stages, tasks) * [Export Results](#export-results) (output parts, download URLs) * [Webhook Notifications](#webhook-notifications) ### Ingest Data #### Upload Video ## Upload a video > Request a signed URL to upload a video file. After receiving the response, PUT the video to the \`uploadURL\` with the provided \`extensionHeaders\`.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/video":{"post":{"operationId":"uploadVideo","tags":["Data Ingestion"],"summary":"Upload a video","description":"Request a signed URL to upload a video file. After receiving the response, PUT the video to the `uploadURL` with the provided `extensionHeaders`.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"name":"fileName","in":"query","required":true,"schema":{"type":"string"},"description":"Name of the video file (e.g. `my_video.mp4`)."}],"responses":{"200":{"description":"Signed URL details for uploading the video.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"signedURLDetails":{"type":"object","properties":{"uploadURL":{"type":"string","description":"PUT the video file to this URL."},"method":{"type":"string","description":"HTTP method to use for the upload."},"extensionHeaders":{"type":"object","additionalProperties":{"type":"string"},"description":"Include these headers in the PUT request."},"maxFileSize":{"type":"integer","description":"Maximum file size in bytes."}}}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### Upload Image ## Upload a single image > Upload a single image via multipart form data. Best for batches up to 5,000 images.\ > \ > \*\*Note:\*\* Single-image and bulk uploads cannot be combined for the same batch.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/upload/image":{"post":{"operationId":"uploadImage","tags":["Data Ingestion"],"summary":"Upload a single image","description":"Upload a single image via multipart form data. Best for batches up to 5,000 images.\n\n**Note:** Single-image and bulk uploads cannot be combined for the same batch.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"name":"fileName","in":"query","required":true,"schema":{"type":"string"},"description":"Name of the image file."}],"requestBody":{"required":true,"content":{"multipart/form-data":{"schema":{"type":"object","properties":{"file":{"type":"string","format":"binary","description":"The image file to upload."}}}}}},"responses":{"200":{"description":"Image uploaded successfully.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### Bulk Upload Images ## Bulk upload images > Request a signed URL for uploading a \`.tar\` archive of images. Recommended for batches exceeding 5,000 images. Bundle up to 500 images per archive.\ > \ > The response contains a signed URL and extension headers. Pack images into a \`.tar\` archive and PUT it to the signed URL.\ > \ > \*\*Note:\*\* Bulk and single-image uploads cannot be combined for the same batch.\ > \ > When performing bulk ingestion, data is indexed in the background. There may be a short delay before all data is available.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Ingestion","description":"Upload images and videos to Data Staging before processing."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/bulk-upload/image-files":{"post":{"operationId":"bulkUploadImages","tags":["Data Ingestion"],"summary":"Bulk upload images","description":"Request a signed URL for uploading a `.tar` archive of images. Recommended for batches exceeding 5,000 images. Bundle up to 500 images per archive.\n\nThe response contains a signed URL and extension headers. Pack images into a `.tar` archive and PUT it to the signed URL.\n\n**Note:** Bulk and single-image uploads cannot be combined for the same batch.\n\nWhen performing bulk ingestion, data is indexed in the background. There may be a short delay before all data is available.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Signed URL details for uploading a tar archive.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"signedURLDetails":{"type":"object","properties":{"shardId":{"type":"string","format":"uuid","description":"Unique identifier for this shard upload."},"uploadURL":{"type":"string","description":"PUT the tar archive to this URL."},"method":{"type":"string","description":"HTTP method to use for the upload."},"extensionHeaders":{"type":"object","additionalProperties":{"type":"string"},"description":"Include these headers in the PUT request."},"maxNumberOfImages":{"type":"integer","description":"Maximum number of images per tar archive."},"maxShardSize":{"type":"integer","description":"Maximum tar archive size in bytes."}}}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` ### Check Batch Status ## Get batch item count > Returns the count of ingested items in a batch. Use this to verify all data has been ingested before starting a job.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Batch Status","description":"Inspect staged data and batch contents."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/count":{"get":{"operationId":"getBatchCount","tags":["Batch Status"],"summary":"Get batch item count","description":"Returns the count of ingested items in a batch. Use this to verify all data has been ingested before starting a job.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Batch item count.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"count":{"type":"integer","description":"Number of items in the batch."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### Check Shard Upload Details ## List batch shards > Returns shard details for a bulk-upload batch. Paginated — use \`nextPageToken\` from the response to fetch subsequent pages.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Batch Status","description":"Inspect staged data and batch contents."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/shards":{"get":{"operationId":"getBatchShards","tags":["Batch Status"],"summary":"List batch shards","description":"Returns shard details for a bulk-upload batch. Paginated — use `nextPageToken` from the response to fetch subsequent pages.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"}],"responses":{"200":{"description":"Paginated list of batch shards.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"shards":{"type":"array","items":{"type":"object"},"description":"List of shard objects."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}} ``` ### Start a Job ## Start a batch processing job > Start a batch processing job that runs a Workflow against staged data.\ > \ > \*\*Job ID constraints:\*\* Lowercase letters, digits, hyphens, and underscores only. Maximum 20 characters.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}":{"post":{"operationId":"startJob","tags":["Processing"],"summary":"Start a batch processing job","description":"Start a batch processing job that runs a Workflow against staged data.\n\n**Job ID constraints:** Lowercase letters, digits, hyphens, and underscores only. Maximum 20 characters.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/JobStartRequest"}}}},"responses":{"200":{"description":"Job started successfully.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}},"schemas":{"JobStartRequest":{"type":"object","required":["type","jobInput","computeConfiguration","processingSpecification"],"properties":{"type":{"type":"string","enum":["simple-image-processing-v1"],"description":"Job type."},"jobInput":{"type":"object","required":["type","batchId"],"properties":{"type":{"type":"string","enum":["staging-batch-input-v1"],"description":"Input type."},"batchId":{"type":"string","description":"The batch ID containing the data to process."}}},"computeConfiguration":{"type":"object","required":["type","machineType"],"properties":{"type":{"type":"string","enum":["compute-configuration-v2"],"description":"Configuration type."},"machineType":{"type":"string","enum":["cpu","gpu"],"description":"Machine type. Use `gpu` for Workflows with multiple or large models."},"workersPerMachine":{"type":"integer","default":4,"description":"Number of parallel workers per machine. Reduce for memory-intensive Workflows."}}},"processingTimeoutSeconds":{"type":"integer","default":3600,"description":"Maximum cumulative machine runtime in seconds across all parallel workers."},"processingSpecification":{"type":"object","required":["type","workspace","workflowId"],"properties":{"type":{"type":"string","enum":["workflows-processing-specification-v1"],"description":"Processing specification type."},"workspace":{"type":"string","description":"Workspace containing the Workflow."},"workflowId":{"type":"string","description":"The Workflow to run. Find this in the Workflow Editor under \"Deploy\"."},"aggregationFormat":{"type":"string","enum":["jsonl","csv"],"default":"jsonl","description":"Output format for aggregated results."}}},"notificationsURL":{"type":"string","format":"uri","description":"Webhook URL for job completion notifications. Custom webhook headers are not yet supported. The only header sent is `Authorization: Bearer rf_{workspace_id}`."}}}}}} ``` ### Monitor Job Progress #### Get Job Status ## Get job status > Returns the current status of a batch processing job. ```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}":{"get":{"operationId":"getJobStatus","tags":["Processing"],"summary":"Get job status","description":"Returns the current status of a batch processing job.","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"Job status details.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"jobStatus":{"type":"string","description":"Current job status (e.g. `pending`, `processing`, `completed`, `failed`)."},"progress":{"type":"number","description":"Processing progress as a fraction between 0 and 1."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### List Job Stages ## List job stages > Returns the list of stages for a job. Each job typically has \`processing\` and \`export\` stages, each producing an output batch.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages":{"get":{"operationId":"getJobStages","tags":["Processing"],"summary":"List job stages","description":"Returns the list of stages for a job. Each job typically has `processing` and `export` stages, each producing an output batch.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"List of job stages.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"stages":{"type":"array","items":{"type":"object"},"description":"List of stage objects. Each stage has an ID and an output batch ID."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### List Stage Tasks ## List tasks for a stage > Returns the list of tasks for a specific job stage. Paginated — use \`nextPageToken\` from the response to fetch subsequent pages.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Processing","description":"Start and monitor batch processing jobs."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/batch-processing/v1/external/{workspace}/jobs/{job_id}/stages/{stage_id}/tasks":{"get":{"operationId":"getStageTasks","tags":["Processing"],"summary":"List tasks for a stage","description":"Returns the list of tasks for a specific job stage. Paginated — use `nextPageToken` from the response to fetch subsequent pages.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/job_id"},{"name":"stage_id","in":"path","required":true,"schema":{"type":"string"},"description":"The stage identifier."},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"}],"responses":{"200":{"description":"Paginated list of tasks.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"tasks":{"type":"array","items":{"type":"object"},"description":"List of task objects."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"job_id":{"name":"job_id","in":"path","required":true,"schema":{"type":"string","maxLength":20,"pattern":"^[a-z0-9_-]+$"},"description":"Job identifier. Lowercase, max 20 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}} ``` ### Export Results #### List Output Parts ## List output batch parts > Lists the parts of an output batch. Use the \`export\` stage output batch for compressed results.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Export","description":"Download results after processing completes."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/parts":{"get":{"operationId":"listBatchParts","tags":["Data Export"],"summary":"List output batch parts","description":"Lists the parts of an output batch. Use the `export` stage output batch for compressed results.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"}],"responses":{"200":{"description":"List of batch parts.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"parts":{"type":"array","items":{"type":"object","properties":{"partName":{"type":"string","description":"Name of the batch part."}}},"description":"List of batch part objects."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."}}}} ``` #### List Download URLs ## List download URLs > Returns paginated download URLs for files in a batch part.
```json {"openapi":"3.0.3","info":{"title":"Roboflow Batch Processing API","version":"1.0"},"tags":[{"name":"Data Export","description":"Download results after processing completes."}],"servers":[{"url":"https://api.roboflow.com"}],"paths":{"/data-staging/v1/external/{workspace}/batches/{batch_id}/list":{"get":{"operationId":"listDownloadUrls","tags":["Data Export"],"summary":"List download URLs","description":"Returns paginated download URLs for files in a batch part.\n","parameters":[{"$ref":"#/components/parameters/workspace"},{"$ref":"#/components/parameters/batch_id"},{"$ref":"#/components/parameters/api_key"},{"$ref":"#/components/parameters/nextPageToken"},{"name":"partName","in":"query","schema":{"type":"string"},"description":"Filter by part name (from the list parts response)."}],"responses":{"200":{"description":"Paginated list of download URLs.","content":{"application/json":{"schema":{"type":"object","properties":{"status":{"type":"string"},"filesMetadata":{"type":"array","items":{"type":"object","properties":{"downloadURL":{"type":"string","description":"Signed URL to download the file."},"fileName":{"type":"string","description":"Original file name."},"partName":{"type":"string","nullable":true,"description":"Part name this file belongs to."},"shardId":{"type":"string","nullable":true,"description":"Shard ID (for bulk-upload batches)."},"contentType":{"type":"string","description":"Content type (e.g. `image`, `video`)."},"nestedContentType":{"type":"string","nullable":true,"description":"Nested content type, if applicable."}}},"description":"List of file metadata objects with download URLs."},"nextPageToken":{"type":"string","nullable":true,"description":"Token for fetching the next page of results. `null` if no more pages."}}}}}}}}}},"components":{"parameters":{"workspace":{"name":"workspace","in":"path","required":true,"schema":{"type":"string"},"description":"Your Roboflow workspace identifier."},"batch_id":{"name":"batch_id","in":"path","required":true,"schema":{"type":"string","maxLength":64,"pattern":"^[a-z0-9_-]+$"},"description":"Batch identifier. Lowercase, max 64 chars: letters, digits, hyphens, underscores."},"api_key":{"name":"api_key","in":"query","required":true,"schema":{"type":"string"},"description":"Your Roboflow API key."},"nextPageToken":{"name":"nextPageToken","in":"query","required":false,"schema":{"type":"string"},"description":"Pagination token from a previous response."}}}} ``` ### Webhook Notifications Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes. See [CLI Usage](#webhook-automation) for webhook configuration and payload formats. ## CLI Use `inference rf-cloud` to stage and process local or cloud images and videos. For images already in your [Asset Library](https://docs.roboflow.com/platform/workspaces/asset-library), use `roboflow batch`. ### Local and Cloud Files #### Setup For local or cloud files, install the Inference CLI: ```bash pip install inference-cli export ROBOFLOW_API_KEY="YOUR-API-KEY-GOES-HERE" ``` For cloud storage support: ```bash pip install 'inference-cli[cloud-storage]' ``` If you need help finding your API key, see our [authentication guide](https://docs.roboflow.com/reference/authentication/authentication/find-your-roboflow-api-key). #### Ingest Data **Images** ```bash inference rf-cloud data-staging create-batch-of-images \ --images-dir \ --batch-id ``` **Videos** ```bash inference rf-cloud data-staging create-batch-of-videos \ --videos-dir \ --batch-id ``` {% hint style="info" %} Use a lowercase batch ID with at most 64 characters: letters, digits, hyphens (`-`), and underscores (`_`). {% endhint %} **Cloud Storage** If your data is already in cloud storage (S3, Google Cloud Storage, or Azure), you can process it directly without downloading files locally. For images: ```bash inference rf-cloud data-staging create-batch-of-images \ --data-source cloud-storage \ --bucket-path \ --batch-id ``` For videos: ```bash inference rf-cloud data-staging create-batch-of-videos \ --data-source cloud-storage \ --bucket-path \ --batch-id ``` The `--bucket-path` parameter supports the following providers. Glob patterns filter which files are ingested:
ProviderPath formatGlob example
S3s3://bucket-name/path/s3://my-bucket/training-data/**/*.jpg: all JPGs recursively
Google Cloud Storagegs://bucket-name/path/gs://my-bucket/videos/2024-*/*.mp4: MP4s in 2024-* folders
Azure Blob Storageaz://container-name/path/az://container/images/*.png: PNGs in the images folder
{% hint style="info" %} Your cloud storage credentials are used only locally by the CLI to generate presigned URLs. They are never uploaded to Roboflow servers. {% endhint %} {% hint style="warning" %} Generated presigned URLs are valid for 24 hours. Ensure your batch processing job completes within this timeframe. {% endhint %} For large datasets, the system automatically splits images into chunks of 20,000 files each. Videos work best in batches under 1,000. **Signed URL Ingestion** For advanced automation, you can ingest data via signed URLs instead of local files:
FlagDescription
--data-source references-fileProcess files referenced via signed URLs.
--references <path_or_url>Path to a JSONL file containing file URLs, or a signed URL pointing to such a file.
Reference files use `JSONL`: ``` {"name": "", "url": "https://"} {"name": "", "url": "https://"} ``` {% hint style="info" %} Signed URL ingestion is available to Growth Plan and Enterprise customers. {% endhint %} #### Inspect Staged Data ```bash inference rf-cloud data-staging show-batch-details --batch-id ``` #### Start a Job **Process Images** ```bash inference rf-cloud batch-processing process-images-with-workflow \ --workflow-id \ --batch-id \ --machine-type gpu ``` **Process Videos** ```bash inference rf-cloud batch-processing process-videos-with-workflow \ --workflow-id \ --batch-id \ --machine-type gpu \ --max-video-fps ``` {% hint style="info" %} To find your Workflow ID, open the Workflow Editor in the Roboflow App, click "Deploy", and find the identifier in the code snippet. {% endhint %} {% hint style="info" %} By default, processing runs on CPU. Use `--machine-type gpu` for Workflows with multiple or large models. {% endhint %} #### Monitor Job Progress The start command outputs a job ID. Use it to check status: ```bash inference rf-cloud batch-processing show-job-details --job-id ``` #### Export Results The job details will include the output batch ID. Use it to export results: ```bash inference rf-cloud data-staging export-batch \ --target-dir \ --batch-id ``` #### Webhook Automation Instead of polling for status, you can use webhooks to get notified when ingestion or processing completes. **Data Ingestion Webhooks** The CLI commands `create-batch-of-images` and `create-batch-of-videos` support:
FlagDescription
--notifications-url <webhook_url>Webhook endpoint for notifications.
--notification-category ingest-statusOverall ingestion process status. Default.
--notification-category files-statusIndividual file processing status.
Notifications are delivered via HTTP POST with an `Authorization` header containing your Roboflow Publishable Key. **Ingest Status Notification** ```json { "type": "roboflow-data-staging-notification-v1", "event_id": "8c20f970-fe10-41e1-9ef2-e057c63c07ff", "ingest_id": "8cd48813430f2be70b492db67e07cc86", "batch_id": "test-batch-117", "shard_id": null, "notification": { "type": "ingest-status-notification-v1", "success": false, "error_details": { "type": "unsafe-url-detected", "reason": "Untrusted domain found: https://example.com/image.png" } }, "delivery_attempt": 1 } ``` **File Status Notification** ```json { "type": "roboflow-data-staging-notification-v1", "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81", "ingest_id": "d5cb69aa-b2d1-4202-a1c1-0231f180bda9", "batch_id": "prod-batch-1", "shard_id": "0d40fa12-349e-439f-83f8-42b9b7987b33", "notification": { "type": "ingest-files-status-notification-v1", "success": true, "ingested_files": [ "000000494869.jpg", "000000186042.jpg" ], "failed_files": [ { "type": "file-size-limit-exceeded", "file_name": "big_image.png", "reason": "Max size of single image is 20971520B." } ], "content_truncated": false }, "delivery_attempt": 1 } ``` **Job Completion Webhooks** Add `--notifications-url` when starting a job: ```bash inference rf-cloud batch-processing process-images-with-workflow \ --workflow-id \ --batch-id \ --notifications-url ``` **Job Completion Notification** ```json { "type": "roboflow-batch-job-notification-v1", "event_id": "8f42708b-aeb7-4b73-9d83-cf18518b6d81", "job_id": "", "job_state": "success | fail", "delivery_attempt": 1 } ``` #### Cloud Storage Authentication **AWS S3 and S3-Compatible Storage** Credentials are detected automatically from: 1. Environment variables: ```bash export AWS_ACCESS_KEY_ID=your-access-key-id export AWS_SECRET_ACCESS_KEY=your-secret-access-key export AWS_SESSION_TOKEN=your-session-token # Optional ``` 2. AWS credential files (`~/.aws/credentials`, `~/.aws/config`) 3. IAM roles (EC2, ECS, Lambda) Named profiles: ```bash export AWS_PROFILE=production ``` For S3-compatible services (ex: Cloudflare R2, MinIO): ```bash export AWS_ENDPOINT_URL=https://account-id.r2.cloudflarestorage.com export AWS_REGION=auto # R2 requires region='auto' export AWS_ACCESS_KEY_ID=your-r2-access-key export AWS_SECRET_ACCESS_KEY=your-r2-secret-key ``` **Google Cloud Storage** Credentials are detected from: 1. Service account key file (recommended for automation): ```bash export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json ``` 2. User credentials from gcloud CLI (`gcloud auth login`) 3. GCP metadata service (when running on Google Cloud Platform) **Azure Blob Storage** SAS token (recommended): ```bash export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount export AZURE_STORAGE_SAS_TOKEN="YOUR_SAS_TOKEN" ``` Account key: ```bash export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount export AZURE_STORAGE_ACCOUNT_KEY=your-account-key ``` Generate a SAS token with the Azure CLI. Replace `` with a future UTC time in `YYYY-MM-DDTHH:MMZ` format: ```bash az storage container generate-sas \ --account-name mystorageaccount \ --name my-container \ --permissions rl \ --expiry ``` **Custom Scripts** For advanced use cases, reference scripts for generating signed URL files: * AWS S3: [generateS3SignedUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateS3SignedUrls.sh) * Google Cloud Storage: [generateGCSSignedUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateGCSSignedUrls.sh) * Azure Blob Storage: [generateAzureSasUrls.sh](https://raw.githubusercontent.com/roboflow/roboflow-python/main/scripts/generateAzureSasUrls.sh) #### Discover All Options ```bash inference rf-cloud --help inference rf-cloud data-staging --help inference rf-cloud batch-processing --help ``` ### Asset Library Images Replace `` with your Workspace slug (ex: `my-workspace`). Choose a published Workflow with one `image` input. Give other required inputs defaults, except `source_id`, which Batch Processing fills for each image. Install or update the [Roboflow CLI](https://docs.roboflow.com/reference/platform/cli), sign in, and list your Workflows: ```bash pip install --upgrade roboflow roboflow auth login roboflow --workspace --json workflow list ``` Use a published Workflow's `id` from the JSON output. Find the images to process with a [RoboQL](https://docs.roboflow.com/platform/workspaces/asset-library) search: ```bash roboflow --workspace --json image search "tag:night-shift" \ --fields id,filename ``` Pass the image IDs from the search results to `--image-ids`: ```bash roboflow --workspace batch create \ --workflow \ --image-ids , ``` To process all images that match a structured filter, replace `--image-ids` with `--query "tag:night-shift"`. Use `--all` to process the entire Asset Library. Choose exactly one selection option. The create response includes `taskId`, `jobId`, and `requestId`. Use the task ID to wait for completion and the job ID to inspect the job: ```bash roboflow --workspace asynctasks wait roboflow --workspace batch status roboflow --workspace batch list ``` {% hint style="info" %} A job keeps running if you close the terminal or the wait command times out. After a network error, retry with the same options. Pass the `requestId` from the response or error hint as `--request-id ` to avoid creating a duplicate job. {% endhint %} Use `batch abort ` to stop a job or `batch restart ` to retry it with its existing configuration. Both commands ask for confirmation; add `--yes` for scripts. A restart can consume credits.