> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/datasets/adding-data/datasources.md). # Datasources Datasources let you continuously mirror images and metadata from cloud storage into your Roboflow asset library. Once mirrored, images are searchable by semantics, custom metadata, tags, or image similarity, and can be added to any Project for labeling and training. Currently, AWS S3 and S3-compatible storage bucket mirroring is supported. Azure Blob Storage and Google Cloud Storage support is coming soon. {% hint style="warning" %} If your source data is in cloud storage such as AWS S3, use Datasources and Bucket Mirror as the default path into Roboflow. Use signed URL uploads or local download workflows only for one-time or ad hoc imports. {% endhint %} ## How Bucket Mirror Works When you configure a Datasource, Roboflow crawls your S3 bucket and imports all matching image files into your Workspace's [Asset Library](/workspaces/asset-library.md). * Supported image formats: JPEG, PNG, BMP, WebP, AVIF * Files already present in your workspace (matched by their S3 location and hash) are not re-imported, reducing egress costs * If a `.json` sidecar file exists alongside an image with the same base name, its metadata is imported; nested keys are flattened using dot notation (e.g., `capture.temperature`) — see [Metadata Sidecars](#metadata-sidecars) * Files that disappear from the bucket are kept by default; enable orphan removal to delete them instead (see [Removing Orphaned Files](#removing-orphaned-files)) ## Mirror your Bucket To Roboflow ### Prerequisites 1. An AWS S3 bucket containing your image data 2. A reusable Roboflow credential that can read from that bucket. See [Datasource Credentials](/datasets/adding-data/datasources/datasource-credentials.md), then follow [AWS S3 Credentials](/datasets/adding-data/datasources/datasource-credentials/aws-s3.md). ### Add a Credential in Roboflow Roboflow stores your bucket access securely and encrypted as a reusable credential. For AWS setup steps and least-privilege guidance, use [AWS S3 Credentials](/datasets/adding-data/datasources/datasource-credentials/aws-s3.md). Go to [Credentials](https://app.roboflow.com/settings/thirdpartykeys) in your Workspace settings and click on [Add Credential](https://app.roboflow.com/settings/thirdpartykeys#create). ### Configure a Datasource for Bucket Mirroring [Create a new Datasource](https://app.roboflow.com/settings/datasources) from your workspace settings. The form has two tabs: * "Connection" holds the bucket details and access: name, provider, bucket, region, and Credential. Select your saved Credential from the "Credential" dropdown, or use the "+" next to it to add one without leaving the form. * "Mirror Configuration" holds the import destination, file filters, and mirror behavior. ### Choosing an Import Destination Under "Mirror Configuration", the "Import Destination" section controls where mirrored files land. Each Datasource imports to a single destination; add another Datasource to import elsewhere. * "Workspace" mirrors into the [Asset Library](/workspaces/asset-library.md). Use the "Import into" dropdown to keep files at the workspace root, or select a Project to also add them to that Project. * "Folder" scopes the mirrored images to a project folder so only that folder's team can see them. This option is available on plans with [Project Folder Permissions](/datasets/project-folders/project-folder-permissions.md). ### Filtering with Glob Patterns By default, all supported image files in the bucket are imported. You can restrict which files are imported using glob patterns, either specified directly or via a `.txt` file stored in the bucket. You may also provide an explicit whitelist of file paths instead of glob patterns. ### Pattern semantics * `*` matches any characters except `/` (single directory level) * `**` matches any characters including `/` (multiple directory levels) ### Examples **Match by prefix:** ``` harvest** ``` Matches: `harvest`, `harvest2024`, `harvest/sun/file.jpg`, `harvest-data.png`\ Does not match: `Harvest`, `my-harvest` **Match everything in a folder:** ``` /harvest/sun/** ``` Matches: `/harvest/sun/file.txt`, `/harvest/sun/subfolder/image.jpg`, `/harvest/sun/deep/nested/path/data.png`\ Does not match: `/harvest/moon/file.txt`, `/other/sun/file.txt` **Match by suffix within a subtree:** ``` /planting/**/*crops.png ``` Matches: `/planting/wheat-crops.png`, `/planting/subfolder/rice-crops.png`\ Does not match: `/planting/wheat.png`, `/other/wheat-crops.png` **Match at a specific directory level with a name pattern:** ``` /*/a/**/*weed*2025-10-27.png ``` Matches: `/farm/a/field/weed-2025-10-27.png`, `/garden/a/plot/seaweed-data-2025-10-27.png`\ Does not match: `/farm/b/field/weed-2025-10-27.png` **Exact path:** ``` /exact/path/to/file.jpg ``` Matches only that specific file. **Literal wildcards in filenames:**\ Wrap the pattern in quotes to treat `*` as a literal character: ``` "/path/to/file*.jpg" ``` ### Removing Orphaned Files Orphan removal is off by default, so files that disappear from your bucket are retained. When `removeOrphanedSourcesWhenDisappeared` is enabled, files that are no longer present in your S3 bucket (or no longer matched by your glob patterns) are removed from your Roboflow workspace, provided they are not referenced by any Project or another Datasource configuration. This also applies when you delete a Datasource. If orphan removal is enabled and the bucket has been mirrored at least once, images originating from that bucket that are not used by any other Project may be removed by the cleanup worker. The delete confirmation dialog will warn you about this and require explicit acknowledgement before proceeding. To avoid this, disable orphan removal on the Datasource's mirror configs before deleting it. ### File Naming The `namingStrategy` setting controls how imported files are named and displayed in Roboflow: | Strategy | Description | | ---------- | ------------------------------------------------------------------------------------------- | | `fullPath` | Uses the full S3 key path as the filename (default) | | `fileName` | Uses only the filename portion of the S3 key | | `eTag` | Uses the S3 object ETag | | `metadata` | Uses a value from the image's metadata, specified by `namingStrategyMetadataKey` (required) | ### Image Updates When an image in S3 is modified, Roboflow can update the copy in your workspace: * `updateImageWhenNewer` (default: `true`) — re-imports the image when the S3 object is newer than the stored version * `updateImageStrategy` — controls how the update is applied; currently `overwrite` (replaces the existing image) is supported ### Metadata Sidecars Attach metadata to images by placing a `.json` sidecar file in the bucket alongside each image, using the same base name: ``` my-bucket/ images/ photo_001.jpg photo_001.json # metadata for photo_001.jpg photo_002.jpg photo_002.json # metadata for photo_002.jpg ``` The sidecar file contains key-value pairs: ```json { "camera_id": "cam001", "location": "warehouse-3", "capture": { "temperature": 72.5, "humidity": 45 } } ``` Nested objects are flattened using dot notation. The example above produces: | Key | Value | | --------------------- | --------------- | | `camera_id` | `"cam001"` | | `location` | `"warehouse-3"` | | `capture.temperature` | `72.5` | | `capture.humidity` | `45` | Sidecar file constraints: * Maximum file size: 256 KB * Must be valid JSON * `null` and `undefined` values are filtered out ### Metadata Sync Strategies When an image's metadata sidecar `.json` file is updated in S3, two settings control how the update is applied: * `updateMetadataWhenNewer` (default: `true`) — re-syncs metadata when the sidecar file is newer than the stored version * `updateMetadataStrategy` — controls how the synced metadata interacts with metadata you have set manually via the UI or API: | Strategy | Behavior | | --------------------------- | ---------------------------------------------------------------------------- | | `mergeBucketWins` (default) | Merges both sources; on key conflicts, the bucket value wins | | `mergeUserWins` | Merges both sources; on key conflicts, the user-set value wins | | `overwrite` | Bucket metadata completely replaces all existing metadata | | `untilFirstChange` | Syncs from bucket until a user manually edits any metadata field, then stops | | `append` | Only adds new keys from the bucket; never overwrites existing keys | ## Triggering a Mirroring You can trigger a mirror manually at any time from the [Datasources list](https://app.roboflow.com/settings/datasources) by clicking the play button next to a Datasource. A manual trigger is subject to the following guards: * **In-progress**: If a sync is already running, you cannot start another one until it finishes. * **Cooldown**: After a sync completes, manual re-triggers are blocked for 15 minutes. The button tooltip shows how many minutes remain. The cooldown is skipped when: * The previous sync found nothing new to import (zero files enqueued, or all files failed). * You edited the Datasource configuration since the last run. * The last run completed with errors. * **Hourly cap**: A Datasource can be synced at most 10 times per rolling hour. This cap applies even when a cooldown skip would otherwise allow an immediate retrigger. Scheduled (cron) syncs bypass both the cooldown and the hourly cap. ### Running on a Daily Schedule To mirror automatically, open the "Scheduling" section under the "Mirror Configuration" tab and check "Run automatically on a daily schedule". The datasource then syncs every 24 hours. Scheduling is off by default. ## Viewing Synced Assets Each of your [datasource entries](https://app.roboflow.com/settings/datasources) has an eye icon that opens the [Asset Library](/workspaces/asset-library.md) filtered to images and videos from that specific Datasource. The icon is disabled until the Datasource has completed at least one sync. To view all images synced from any Datasource, click "View Datasource Assets" at the bottom of the Datasources list. This link appears once at least one Datasource has run. Both links navigate to the Asset Library with a pre-filled tag filter so you can browse, search, and manage only the bucket-mirrored subset of your Workspace images. ## S3-compatible storage Datasources work with S3-compatible storage providers that implement the required S3 API operations. To configure one of these providers: 1. In the Datasource "Connection" tab, select `S3` as the provider. 2. Enter the bucket name and credentials as usual. 3. Define the provider's custom `endpoint` URL. 4. Set the region to `auto` or the provider-specific region value. Use the provider's S3 API endpoint for `endpoint`. Do not use a CDN URL, a public bucket URL, or a browser download URL. The same glob pattern filtering, metadata sidecar behavior, and mirror settings also work with these providers. Supported S3-compatible storage providers include: | Provider | Example endpoint hostname | | ----------------------------------------------------------- | -------------------------------------------------- | | Cloudflare R2 | `.r2.cloudflarestorage.com` | | Backblaze B2 | `s3..backblazeb2.com` | | DigitalOcean Spaces | `.digitaloceanspaces.com` | | Akamai Linode Object Storage | `.linodeobjects.com` | | Wasabi | `s3..wasabisys.com` | | Vultr Object Storage | `.vultrobjects.com` | | OVHcloud Object Storage | `s3..io.cloud.ovh.net` | | Scaleway Object Storage | `s3..scw.cloud` | | Open Telekom Cloud | `obs..otc.t-systems.com` | | Exoscale SOS | `sos-.exo.io` | | IONOS Cloud Object Storage | `s3-.ionoscloud.com` | | IBM Cloud Object Storage | `s3..cloud-object-storage.appdomain.cloud` | | Oracle Cloud Infrastructure Object Storage S3 Compatibility | `compat.objectstorage..oraclecloud.com` | | Seagate Lyve Cloud | `s3..lyvecloud.seagate.com` | | Huawei Cloud OBS | `obs..myhuaweicloud.com` | | Alibaba Cloud OSS | `oss-.aliyuncs.com` | | Tencent Cloud COS | `cos..myqcloud.com` | | Yandex Object Storage | `storage.yandexcloud.net` | | Storj Hosted S3 Gateway | `gateway.storjshare.io` |