Collaborate on Labeling
Have the entire team help label your datasets.
About
Whether you have a small team working on labeling hundreds of images or a large team working on millions, creating a dataset is about more than drawing boxes. A big part of labeling is in the process of getting an image from the real world into a trained model's stored knowledge that involves image collection, storage, organization, selection, assignment, labeling, and review.
Roboflow offers collaborative features that allow you to:
Divide work between multiple team members by assigning labeling jobs to anyone on the team
Organize images into batches as you upload them
Provide image labeling instructions to help guide work and ensure consistency
Get an at-a-glance view of labeling work in progress
See a historical timeline of all labeling work
Revert changes
Add comments to images and view image comment history
Send images with labeling issues back to team members for changes
Approve or reject labels before including them in a dataset
Assign Labeling Jobs
You can divide labeling work among a team or assign it to specific people responsible for labeling. Assigning jobs to individual team members means you won't have to worry about stepping on each others' work if you're online at the same time.
You can choose to assign jobs to one or more people on your team and if you haven’t included a team member to your workspace yet, you can also invite them and assign an labeling job to be completed at the same time.
Provide Labeling Instructions
You can provide instructions to labelers from the Assign Images tab. Click "Add Instructions" to add instructions to a batch before assigning the batch to an labeler. When you have set your instructions, click "Assign Images".
Click "Edit" to assign instructions:
Click "Save Instructions" to save the labeling instructions.
Job Notifications
Once a labeling job has been assigned, a notification will alert your team members when there's work assigned to them.
Labeling Jobs Board
The labeling jobs board gives an at-a-glance view of the current state of your individually assigned jobs as they go through the labeling process.
To view statistics for a particular labeler, specify a value in the Labeler dropdown.
Job Details
Clicking on individual jobs on the labeling jobs board gives a more detailed view of the individual job and its progress. You can quickly see images that still need to be labeled and reassign jobs to different team members as needed.
Managing Individual Images
On the Annotated tab within a job, you can select individual images using checkboxes, drag-to-select, or the "Select all" checkbox in the header. Once you have a selection:
Click "Send N Images to Unannotated" to move selected images back to the Unannotated tab for re-labeling.
Click "Add N Images to Dataset" to add only the selected images to the dataset. Unselected annotated images remain in the job.
When adding a partial selection to the dataset, the job stays in its current state with updated counts so labelers can continue working on the remaining images.
These controls are also available in the annotation editor. When viewing an annotated image in an assigned job:
Use the checkmark button to add the current image to the dataset.
Use the "Send to Unannotated" button (arrow icon) next to the image switcher to send the current image back for re-labeling.
After either action, the editor advances to the next image automatically.
Review Mode
You can individually approve or reject labeled images and send them back to the labeler for rework when necessary. To do so, click on a batch of images. Then, navigate between the Approved, Rejected, and To Do tabs to view the state of images in a job.
Review Mode is on if your plan includes it. To turn it off, open your project's Annotate page and click "Disable Review Mode", which removes the Review column and sends a finished job straight to your Dataset. Click "Enable Review Mode" to turn it back on at any time, since turning it off does not remove Review Mode from your plan.
Browse Labeling Work From the Agent
Open a project tab in Roboflow Agent and select "Dataset" to browse the project's images by labeling stage. A chip per stage ("Unassigned", "Annotating", "Review", "Dataset") shows its image count. Pick a chip, then a batch or job card, to narrow the grid below.
Click an image to open it in the annotation editor. Images open read-only in "Unassigned", or when the job has left the annotation board or is no longer assigned to you. You need the Search Workspace Images permission, plus Update Annotation to edit labels.
Move a Job Through Its Stages
When you open a job in the Agent, the tab bar shows where the job sits: "Unassigned", "Annotating", "Review", or "Dataset". Hover over a step to read what happens in that stage. If Review Mode is switched off for your workspace, the "Review" step is greyed out. If your plan does not include it, the step offers a way to upgrade.
Buttons beside the stepper move the job forward or back. Which ones you see depends on the stage and your permissions:
"Move N to Annotating" creates a labeling job from unassigned images and assigns it to you.
"Submit for Review" sends a job that is being annotated to review.
"Move Back to Annotating" sends every image in a job under review back for more labeling.
"Add N to Dataset" adds the job's ready images to the dataset. If you add only some of them, the rest stay in the job.
Each button re-checks the job's stage before it writes. If someone else moved the job in the meantime, nothing changes and the Agent tells you why.
Approve and Reject Images
When the job you opened is in the "Review" stage, the editor shows the same approve and reject controls as the annotation board. Click "Approve" or "Reject" on the current image, or press A to approve, R to reject, and U to undo your last decision. This needs the Review Image permission.
HTTP API
You can use the jobs endpoint to get info about your annotation jobs, their current status or assign images for labeling by creating a new Annotation Job with images from one of your batches.
Create a New Annotation Job
You need to set both reviewer and labeler to an email that belong to a user in your workspace.
If your workspace does not have Role Based Access Control enabled, you won't see the reviewer in the app interface, and should set the reviewer to the same user as the labeler.
You can make a POST request to the /jobs endpoint of your project to create a new Annotation Job in your project. To create a new job the you need to provide some JSON encoded data in the body of your POST request.
To create a new Annotation Job in a project, that assigns 10 images from one of the batches in the project to be annotated by lenny@roboflow.com you can make a POST request like this:
This request returns the following data:
Retrieve Annotation Job Data from the API
To retrieve annotation job data from the REST API, make a request to the following endpoint:
This will return data in the following format:
To retrieve information about a specific job, specify the job ID:
Here is the response format from this request:
Python SDK
Create a New Annotation Job
To create an annotation job, you will need an upload batch that you would like to assign to a labeler and a reviewer:
Last updated
Was this helpful?