> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/workflows/evaluate/workflow-evals.md).

# Workflow Evals

Evaluate and compare Workflows against labeled Cases through the Web App, HTTP API, Python SDK, CLI, or MCP.

Workflow Evals checks whether a Workflow meets your requirements on a set of examples. Use it to find failures, compare Workflow versions, and check for regressions before deployment.

Follow the [model comparison tutorial](/workflows/tutorials/compare-models-with-workflow-evals.md) to define a bolt-counting Eval and use an agent to build and compare candidate Workflows.

## About

An Eval combines a Dataset of Cases with a Spec that defines what to check. Each Case contains inputs and ground truth: the expected values or annotations. A Check selects an evaluator and its settings, such as how closely predicted boxes must match your annotations.

A Run evaluates one or more Workflows against those Cases and Checks. Each Run captures its own inputs, ground truth, Workflow definitions, and evaluation settings. Editing or removing a Case affects future Runs; it does not change the results of existing Runs.

Checks come in two kinds: an "Output check" compares a Workflow output with ground truth, and a "Latency budget" sets a time limit per Case. Thresholds for ratio metrics (ex: precision, recall, F1, mAP) are decimals from 0 to 1, so enter 0.8 for 80%. Classification checks compare the `top` or `predicted_classes` output with the expected classes, by exact match or F1.

You can import ground truth from a Dataset test set. The import uses the current annotations on the original images, not the preprocessed or augmented images of a Dataset Version. It supports boxes, classification, and segmentation. Keypoint datasets import images only. Imported labels stay unapproved until you review them.

### Latency

"Latency Avg" is the average recorded request time per Case, including failed requests with a recorded duration. It includes network time and any input preparation timed by the runner, and excludes evaluation queueing and scoring. It is not the sum of all Case durations or a measure of model inference time alone.

Runs with a "Latency budget" check run every Case again, so they do not reuse cached results. To add a latency check to an older Run, start a new Run; a Rescore cannot measure latency.

### Credits

Each accepted Workflow execution in an Eval costs 1/10,000 of a credit, whether you run it locally or on Serverless. Evaluating 3 Workflows over 100 Cases costs 3/10,000 of a credit for evaluation, plus any inference charges.

Serverless Workflow inference and Label Assist have separate charges. A new Replay or Rescore creates a new billable evaluation execution; retrying the same creation request with its original idempotency key does not. Embedding generation does not consume evaluation credits. Smart Select while you label ground truth uses AI Labeling credits, the same as in the annotation editor.

## Web App

Open "Workflows" in your Workspace, select "Evals", and click "Create Eval". Follow these steps:

1. "Select Workflow": choose the Workflow. Click the server chip in the step header to switch inference servers.
2. "Upload Ground Truth": add images from your device, the Asset Library, or a Dataset test set, then select the Workflow outputs to check.
3. "Label Ground Truth": enter expected values and annotate the images. Use the "Assist" card to get suggestions from a model or Workflow, and turn on "Auto-annotate" to label Cases as you open them. Click "Add to Ground Truth" to approve each Case.
4. "Define Eval Checks": click "Add check" and set the thresholds. Click "Run preview" to test your Checks on one Case.
5. "Run Evaluation": review the setup and credit estimate, then start the Run.

The Eval page lists its Runs next to the ground truth and Checks. Open the ground truth from there to edit it. To find Cases, type in "Search saved ground truth" to filter by ground truth (ex: class, count, completeness) or by the source image (ex: filename, tags, semantic search).

![Eval page with quality, latency, and failing Check trends, four Runs, ground truth images, and two Checks](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-c9b38e2d3b29b8a9419e5780fc673efa923b122e%2Fworkflow-evals-eval-page.png?alt=media)

Open a Run to see its results. When it has two or more Workflows, "Overview" plots each Workflow's quality against latency or credits, with a heatmap of Checks passed per Case. "Metrics" compares results per Check, and "Vector Analysis" groups similar Cases so you can find clusters of failures.

![Overview tab plotting six Workflows by Checks passed against latency, above a heatmap of Checks passed per Case](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-9a7612903ca0ae3e2043d0bc43c710a0ef9594da%2Fworkflow-evals-results-overview.png?alt=media)

### Deploy Gates

A deploy gate blocks a Workflow version from going live until it passes an Eval. To set one up, open the Workflow and click the dropdown next to "Publish". Under "Publishing Requirements", select "Require an Eval to pass". Choose an Eval, set the minimum pass rate for each Check (default 100%), and click "Save Checks".

Click "Run Checks" to run the Eval on the latest saved version. When every Check passes, click "Publish". In "Auto-Publish" mode, each save runs the checks and the version publishes by itself when it passes.

Each new version needs its own passing run. If you edit the Eval, versions that passed before must run the checks again. You need permission to manage Workflow deploy gates to set or remove a gate.

## HTTP API

Use the API to manage Evals, Specs, Datasets, Cases, Runs, results, exports, and embeddings. The namespace is `https://api.roboflow.com/workspaces/{workspace}/workflow-evals`, where `workspace` is your Workspace slug.

Authenticate with a private Workspace API key or OAuth access token in the `Authorization: Bearer` header. Folder-restricted credentials are not supported. OAuth access also follows the user's Workspace permissions and the permissions required to read source images, datasets, or Workflows.

Workflow Evals has four independent OAuth scopes:

* `workflow-evals:read`: read Evals and discover evaluators and schemas.
* `workflow-evals:write`: create and edit evaluation resources.
* `workflow-evals:run`: start and control Runs.
* `workflow-evals:export`: export results.

Request every scope your integration needs. One scope does not grant the others. See [OAuth authentication](https://docs.roboflow.com/reference/authentication/authentication/oauth) for connecting your application.

### Discover evaluators

Fetch the evaluator catalog before building Checks. It returns evaluator types and configuration schemas for the installed engine.

{% openapi src="/files/cs4Es5f2HGBR0uB8X6xi" path="/workspaces/{workspace}/workflow-evals/evaluators" method="get" %}
[workflow-evals-discovery.openapi.json](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-2928c95a7448a440bb1fffe80e25051dc965410d%2Fworkflow-evals-discovery.openapi.json?alt=media)
{% endopenapi %}

### Discover the API contract

Fetch the Workspace's OpenAPI document for the complete operation list, request schemas, required scopes, and headers. Use it with the evaluator catalog rather than guessing output types or evaluator settings.

{% openapi src="/files/cs4Es5f2HGBR0uB8X6xi" path="/workspaces/{workspace}/workflow-evals/openapi.json" method="get" %}
[workflow-evals-discovery.openapi.json](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-2928c95a7448a440bb1fffe80e25051dc965410d%2Fworkflow-evals-discovery.openapi.json?alt=media)
{% endopenapi %}

### Manage resources and Runs

Use Specs to define Checks, Datasets to hold Cases, and bindings to map Workflow outputs to the Spec. Validate the Spec and bindings before starting a Run. List responses use `items` and, when there are more results, `nextCursor`.

Where an operation requires `Idempotency-Key`, send a UUID v4 and keep that key when retrying the same request. Where it requires `If-Match`, send the resource's current revision to avoid overwriting another edit. Runs, imports, and exports are asynchronous: poll their status until they finish, and use the explicit cancellation operation to cancel a Run.

To search a Dataset's Cases, send `POST /workspaces/{workspace}/workflow-evals/eval-datasets/{evalDatasetId}/search`. Filters on the source image and semantic search also need the `image:read` scope. Keep sending `nextCursor` until it is absent, even after an empty page.

You cannot export a failed or partly completed Run that still has Cases to retry. To export it as it is, send `allowPendingRetries: true` in the export request. You cannot export an active Run.

{% hint style="info" %}
The cloud backend cannot connect to your computer's `localhost`. Local Runs use a client runner that executes the Workflow and submits its outputs for scoring. Creating a Run through the API does not make your local server reachable from the cloud.
{% endhint %}

### Delete an Eval

Deletion takes two steps: preview the impact, then confirm with the returned `deletionKey`. If the Eval backs deploy gates, `impact.deployGates` lists the affected Workflows. Confirming removes those gates and cancels their pending deploy checks. Deleting a gated Eval also needs `workflow:update` and permission to manage Workflow deploy gates; an ungated Eval does not need those extra permissions. Poll the returned deletion operation until it finishes.

## Python SDK

Workflow Evals requires version 1.7.0 or later of the [Roboflow Python SDK](https://docs.roboflow.com/reference/platform/python-sdk). Install or update it:

```bash
pip install --upgrade roboflow
```

Use the Workspace's Workflow Evals client to discover evaluators and list Evals:

```python
import os

from roboflow import Roboflow

rf = Roboflow(api_key=os.environ["ROBOFLOW_API_KEY"])
evals = rf.workspace("your-workspace").workflow_evals()

catalog = evals.list_evaluators()
page = evals.list(limit=20)

for evaluation in page["items"]:
    print(evaluation["id"], evaluation["name"])
```

The same client exposes `create`, `start_run`, `get_run`, `wait_for_run`, and `openapi`. Create the Spec, Dataset, and bindings before starting a Run, using the schemas from `openapi()` and `list_evaluators()`.

## CLI

The [Roboflow CLI](https://docs.roboflow.com/reference/platform/cli/install-and-set-up-the-cli) includes the `workflow-eval` command group. Authenticate with your private API key and select a Workspace:

```bash
export ROBOFLOW_API_KEY="your-private-api-key"
roboflow --workspace your-workspace --json workflow-eval list
roboflow --workspace your-workspace --json workflow-eval evaluator list
roboflow --workspace your-workspace --json workflow-eval run list --eval your-eval-id
```

Use `roboflow workflow-eval --help` to see resource and Run commands. This command group evaluates Workflows; it is separate from model evaluation commands.

## MCP

Connect the [Roboflow MCP Server](https://docs.roboflow.com/agents/mcp-server) to your agent. Ask it to load `workflow_evals_agent_manifest` and the Workflow Evals guidance before creating an Eval. The tools cover evaluator discovery, Spec validation, authoring, Runs, and results through the same public API. Use `workflow_evals_datasets_search` to find Cases by ground truth or source-image filters; source-image and semantic search also need `image:read`.

For example, ask: "Create a Workflow Eval for this Workflow using these test images. Help me define ground truth and Checks, validate the setup, and show me the credit estimate before starting a Run."