Datasources

Automatically sync images and metadata from external storage into your Roboflow workspace.

Datasources let you continuously mirror images and metadata from cloud storage into your Roboflow asset library. Once mirrored, images are searchable by semantics, custom metadata, tags, or image similarity, and can be added to any Project for labeling and training.

Currently, AWS S3 and S3-compatible storage bucket mirroring is supported. Azure Blob Storage and Google Cloud Storage support is coming soon.

If your source data is in cloud storage such as AWS S3, use Datasources and Bucket Mirror as the default path into Roboflow. Use signed URL uploads or local download workflows only for one-time or ad hoc imports.

How Bucket Mirror Works

When you configure a Datasource, Roboflow crawls your S3 bucket and imports all matching image files into your Workspace's Asset Library.

• Supported image formats: JPEG, PNG, BMP, WebP, AVIF

• Files already present in your workspace (matched by their S3 location and hash) are not re-imported, reducing egress costs

• If a .json sidecar file exists alongside an image with the same base name, its metadata is imported; nested keys are flattened using dot notation (e.g., capture.temperature) - see Metadata Sidecars

• Files that disappear from the bucket are kept by default; enable orphan removal to delete them instead (see Removing Orphaned Files)

Mirror your Bucket To Roboflow

Prerequisites

1. An AWS S3 bucket containing your image data

2. A reusable Roboflow credential that can read from that bucket. See Datasource Credentials, then follow AWS S3 Credentials.

Add a Credential in Roboflow

Roboflow stores your bucket access securely and encrypted as a reusable credential. For AWS setup steps and least-privilege guidance, use AWS S3 Credentials.

Go to Credentials in your Workspace settings and click on Add Credential.

Configure a Datasource for Bucket Mirroring

Create a new Datasource from your workspace settings. The form has two tabs:

• "Connection" holds the bucket details and access: name, provider, bucket, region, and Credential. Select your saved Credential from the "Credential" dropdown, or use the "+" next to it to add one without leaving the form.

• "Mirror Configuration" holds the import destination, file filters, and mirror behavior.

Choosing an Import Destination

Under "Mirror Configuration", the "Import Destination" section controls where mirrored files land. Each Datasource imports to a single destination; add another Datasource to import elsewhere.

• "Workspace" mirrors into the Asset Library. Use the "Import into" dropdown to keep files at the workspace root, or select a Project to also add them to that Project.

• "Folder" scopes the mirrored images to a project folder so only that folder's team can see them. This option is available on plans with Project Folder Permissions.

Filtering with Glob Patterns

By default, all supported image files in the bucket are imported. You can restrict which files are imported using glob patterns, either specified directly or via a .txt file stored in the bucket.

You may also provide an explicit whitelist of file paths instead of glob patterns.

Pattern semantics

• * matches any characters except / (single directory level)

• ** matches any characters including / (multiple directory levels)

Examples

Match by prefix:

harvest**

Matches: harvest, harvest2024, harvest/sun/file.jpg, harvest-data.png Does not match: Harvest, my-harvest

Match everything in a folder:

/harvest/sun/**

Matches: /harvest/sun/file.txt, /harvest/sun/subfolder/image.jpg, /harvest/sun/deep/nested/path/data.png Does not match: /harvest/moon/file.txt, /other/sun/file.txt

Match by suffix within a subtree:

/planting/**/*crops.png

Matches: /planting/wheat-crops.png, /planting/subfolder/rice-crops.png Does not match: /planting/wheat.png, /other/wheat-crops.png

Match at a specific directory level with a name pattern:

/*/a/**/*weed*2025-10-27.png

Matches: /farm/a/field/weed-2025-10-27.png, /garden/a/plot/seaweed-data-2025-10-27.png Does not match: /farm/b/field/weed-2025-10-27.png

Exact path:

/exact/path/to/file.jpg

Matches only that specific file.

Literal wildcards in filenames: Wrap the pattern in quotes to treat * as a literal character:

"/path/to/file*.jpg"

Removing Orphaned Files

Orphan removal is off by default, so files that disappear from your bucket are retained. When removeOrphanedSourcesWhenDisappeared is enabled, files that are no longer present in your S3 bucket (or no longer matched by your glob patterns) are removed from your Roboflow workspace, provided they are not referenced by any Project or another Datasource configuration.

This also applies when you delete a Datasource. If orphan removal is enabled and the bucket has been mirrored at least once, images originating from that bucket that are not used by any other Project may be removed by the cleanup worker. The delete confirmation dialog will warn you about this and require explicit acknowledgement before proceeding. To avoid this, disable orphan removal on the Datasource's mirror configs before deleting it.

File Naming

The namingStrategy setting controls how imported files are named and displayed in Roboflow:

Strategy	Description
fullPath	Uses the full S3 key path as the filename (default)
fileName	Uses only the filename portion of the S3 key
eTag	Uses the S3 object ETag
metadata	Uses a value from the image's metadata, specified by namingStrategyMetadataKey (required)

Image Updates

When an image in S3 is modified, Roboflow can update the copy in your workspace:

• updateImageWhenNewer (default: true) - re-imports the image when the S3 object is newer than the stored version

• updateImageStrategy - controls how the update is applied; currently overwrite (replaces the existing image) is supported

Metadata Sidecars

Attach metadata to images by placing a .json sidecar file in the bucket alongside each image, using the same base name:

my-bucket/
  images/
    photo_001.jpg
    photo_001.json      # metadata for photo_001.jpg
    photo_002.jpg
    photo_002.json      # metadata for photo_002.jpg

The sidecar file contains key-value pairs:

{
  "camera_id": "cam001",
  "location": "warehouse-3",
  "capture": { "temperature": 72.5, "humidity": 45 }
}

Nested objects are flattened using dot notation. The example above produces:

Key	Value
camera_id	"cam001"
location	"warehouse-3"
capture.temperature	72.5
capture.humidity	45

Sidecar file constraints:

• Maximum file size: 256 KB

• Must be valid JSON

• null and undefined values are filtered out

Metadata Sync Strategies

When an image's metadata sidecar .json file is updated in S3, two settings control how the update is applied:

• updateMetadataWhenNewer (default: true) - re-syncs metadata when the sidecar file is newer than the stored version

• updateMetadataStrategy - controls how the synced metadata interacts with metadata you have set manually via the UI or API:

Strategy	Behavior
mergeBucketWins (default)	Merges both sources; on key conflicts, the bucket value wins
mergeUserWins	Merges both sources; on key conflicts, the user-set value wins
overwrite	Bucket metadata completely replaces all existing metadata
untilFirstChange	Syncs from bucket until a user manually edits any metadata field, then stops
append	Only adds new keys from the bucket; never overwrites existing keys

Triggering a Mirroring

You can trigger a mirror manually at any time from the Datasources list by clicking the play button next to a Datasource.

A manual trigger is subject to the following guards:

• In-progress: If a sync is already running, you cannot start another one until it finishes.

• Cooldown: After a sync completes, manual re-triggers are blocked for 15 minutes. The button tooltip shows how many minutes remain. The cooldown is skipped when:

  • The previous sync found nothing new to import (zero files enqueued, or all files failed).

  • You edited the Datasource configuration since the last run.

  • The last run completed with errors.

• Hourly cap: A Datasource can be synced at most 10 times per rolling hour. This cap applies even when a cooldown skip would otherwise allow an immediate retrigger.

Scheduled (cron) syncs bypass both the cooldown and the hourly cap.

Running on a Daily Schedule

To mirror automatically, open the "Scheduling" section under the "Mirror Configuration" tab and check "Run automatically on a daily schedule". The datasource then syncs every 24 hours. Scheduling is off by default.

Viewing Synced Assets

Each of your datasource entries has an eye icon that opens the Asset Library filtered to images and videos from that specific Datasource. The icon is disabled until the Datasource has completed at least one sync.

To view all images synced from any Datasource, click "View Datasource Assets" at the bottom of the Datasources list. This link appears once at least one Datasource has run.

Both links navigate to the Asset Library with a pre-filled tag filter so you can browse, search, and manage only the bucket-mirrored subset of your Workspace images.

S3-compatible storage

Datasources work with S3-compatible storage providers that implement the required S3 API operations.

To configure one of these providers:

1. In the Datasource "Connection" tab, select S3 as the provider.

2. Enter the bucket name and credentials as usual.

3. Define the provider's custom endpoint URL.

4. Set the region to auto or the provider-specific region value.

Use the provider's S3 API endpoint for endpoint. Do not use a CDN URL, a public bucket URL, or a browser download URL.

The same glob pattern filtering, metadata sidecar behavior, and mirror settings also work with these providers.

Supported S3-compatible storage providers include:

Provider	Example endpoint hostname
Cloudflare R2	<account-id>.r2.cloudflarestorage.com
Backblaze B2	s3.<region>.backblazeb2.com
DigitalOcean Spaces	<region>.digitaloceanspaces.com
Akamai Linode Object Storage	<region>.linodeobjects.com
Wasabi	s3.<region>.wasabisys.com
Vultr Object Storage	<region>.vultrobjects.com
OVHcloud Object Storage	s3.<region>.io.cloud.ovh.net
Scaleway Object Storage	s3.<region>.scw.cloud
Open Telekom Cloud	obs.<region>.otc.t-systems.com
Exoscale SOS	sos-<region>.exo.io
IONOS Cloud Object Storage	s3-<region>.ionoscloud.com
IBM Cloud Object Storage	s3.<region>.cloud-object-storage.appdomain.cloud
Oracle Cloud Infrastructure Object Storage S3 Compatibility	compat.objectstorage.<region>.oraclecloud.com
Seagate Lyve Cloud	s3.<region>.lyvecloud.seagate.com
Huawei Cloud OBS	obs.<region>.myhuaweicloud.com
Alibaba Cloud OSS	oss-<region>.aliyuncs.com
Tencent Cloud COS	cos.<region>.myqcloud.com
Yandex Object Storage	storage.yandexcloud.net
Storj Hosted S3 Gateway	gateway.storjshare.io