> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/workflows/evaluate/workflow-evals.md).

# Workflow Evals

Evaluate and compare Workflows against labeled Cases through the Web App, HTTP API, Python SDK, CLI, or MCP.

Workflow Evals checks whether a Workflow meets your requirements on a set of examples. Use it to find failures, compare Workflow versions, and check for regressions before deployment.

Follow the [model comparison tutorial](/workflows/tutorials/compare-models-with-workflow-evals.md) to define a bolt-counting Eval and use an agent to build and compare candidate Workflows.

## About

An Eval combines a Dataset of Cases with a Spec that defines what to check. Each Case contains inputs and ground truth: the expected values or annotations. A Check selects an evaluator and its settings, such as how closely predicted boxes must match your annotations.

A Run evaluates one or more Workflows against those Cases and Checks. Each Run captures its own inputs, ground truth, Workflow definitions, and evaluation settings. Editing or removing a Case affects future Runs; it does not change the results of existing Runs.

## Web App

Open "Workflows" in your Workspace, select "Evals", and click "Create Eval". Follow these steps:

1. "Select Workflow": choose the Workflow and inference server to use.
2. "Upload Ground Truth": add images with "Browse Device", "Pick from Asset Library", or "Use a Dataset test set". Select the Workflow outputs to check.
3. "Label Ground Truth": enter expected values and annotate the images. Review any Label Assist suggestions before adding them to ground truth. In the Assist card, pick an AI model or a Workflow to make suggestions. You can pick any Workflow used by this Eval or its Runs. Workflows that were deleted appear under "Discarded workflows" and run from their saved copy. By default, suggestions fill only empty fields. Turn on "Overwrite" under "Existing labels" to replace labels that are already there. Turn on "Auto-annotate" in the Assist card to label empty Cases as you open them. It saves the suggestions as a draft; you still click "Add to Ground Truth" to approve them.
4. "Define Eval Checks": choose the evaluators and thresholds that express your requirements. Click "Add check" and pick "Output check" to compare an output with ground truth, or "Latency budget" to set a time limit per Case. To pass a Case only when the output equals ground truth, pick "Matches ground truth exactly". Pick "Custom rule" to set your own metric and threshold, such as a tolerance. Classification checks compare the `top` or `predicted_classes` output with the expected classes, by exact match or F1. Click "Run preview" to test your Checks on one approved Case without starting a Run.
5. "Run Evaluation": review the setup and credit estimate, then start the Run.

To review saved ground truth, open the Eval and click "Browse Ground Truth". The gallery shows each Case image with its annotations and marks it "Labeled" or "Needs labels". Click a Case to edit its ground truth in a full-screen editor.

After a Run, use "Cases" to inspect individual failures and "Metrics" to compare results. When a Run has two or more Workflows, the first tab is "Overview". It plots each Workflow's quality against latency or credits, and its heatmap shows how many Checks each Workflow passed on each Case. "Vector Analysis" groups similar Cases using image embeddings so you can inspect clusters of failures. Embedding generation runs asynchronously; wait for it to finish or retry a failed generation from that tab.

### Latency

"Latency Avg" is the average recorded request time per Case, including failed requests with a recorded duration. It includes network time and any input preparation timed by the runner, and excludes evaluation queueing and scoring. It is not the sum of all Case durations or a measure of model inference time alone.

Runs with a "Latency budget" check run every Case again, so they do not reuse cached results. To add a latency check to an older Run, start a new Run; a Rescore cannot measure latency.

### Credits

Each accepted Workflow execution in an Eval costs 1/10,000 of a credit, whether you run it locally or on Serverless. Evaluating 3 Workflows over 100 Cases costs 3/10,000 of a credit for evaluation, plus any inference charges.

Serverless Workflow inference and Label Assist have separate charges. A new Replay or Rescore creates a new billable evaluation execution; retrying the same creation request with its original idempotency key does not. Embedding generation does not consume evaluation credits. Smart Select while you label ground truth uses AI Labeling credits, the same as in the annotation editor.

## HTTP API

Use the API to manage Evals, Specs, Datasets, Cases, Runs, results, exports, and embeddings. The namespace is `https://api.roboflow.com/workspaces/{workspace}/workflow-evals`, where `workspace` is your Workspace slug.

Authenticate with a private Workspace API key or OAuth access token in the `Authorization: Bearer` header. Folder-restricted credentials are not supported. OAuth access also follows the user's Workspace permissions and the permissions required to read source images, datasets, or Workflows.

Workflow Evals has four independent OAuth scopes:

* `workflow-evals:read`: read Evals and discover evaluators and schemas.
* `workflow-evals:write`: create and edit evaluation resources.
* `workflow-evals:run`: start and control Runs.
* `workflow-evals:export`: export results.

Request every scope your integration needs. One scope does not grant the others. See [OAuth authentication](https://docs.roboflow.com/reference/authentication/authentication/oauth) for connecting your application.

### Discover evaluators

Fetch the evaluator catalog before building Checks. It returns evaluator types and configuration schemas for the installed engine.

{% openapi src="/files/cs4Es5f2HGBR0uB8X6xi" path="/workspaces/{workspace}/workflow-evals/evaluators" method="get" %}
[workflow-evals-discovery.openapi.json](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-2928c95a7448a440bb1fffe80e25051dc965410d%2Fworkflow-evals-discovery.openapi.json?alt=media)
{% endopenapi %}

### Discover the API contract

Fetch the Workspace's OpenAPI document for the complete operation list, request schemas, required scopes, and headers. Use it with the evaluator catalog rather than guessing output types or evaluator settings.

{% openapi src="/files/cs4Es5f2HGBR0uB8X6xi" path="/workspaces/{workspace}/workflow-evals/openapi.json" method="get" %}
[workflow-evals-discovery.openapi.json](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-2928c95a7448a440bb1fffe80e25051dc965410d%2Fworkflow-evals-discovery.openapi.json?alt=media)
{% endopenapi %}

### Manage resources and Runs

Use Specs to define Checks, Datasets to hold Cases, and bindings to map Workflow outputs to the Spec. Validate the Spec and bindings before starting a Run. List responses use `items` and, when there are more results, `nextCursor`.

Where an operation requires `Idempotency-Key`, send a UUID v4 and keep that key when retrying the same request. Where it requires `If-Match`, send the resource's current revision to avoid overwriting another edit. Runs, imports, and exports are asynchronous: poll their status until they finish, and use the explicit cancellation operation to cancel a Run.

You cannot export a failed or partly completed Run that still has Cases to retry. To export it as it is, send `allowPendingRetries: true` in the export request. You cannot export an active Run.

{% hint style="info" %}
The cloud backend cannot connect to your computer's `localhost`. Local Runs use a client runner that executes the Workflow and submits its outputs for scoring. Creating a Run through the API does not make your local server reachable from the cloud.
{% endhint %}

## Python SDK

Workflow Evals requires version 1.7.0 or later of the [Roboflow Python SDK](https://docs.roboflow.com/reference/platform/python-sdk). Install or update it:

```bash
pip install --upgrade roboflow
```

Use the Workspace's Workflow Evals client to discover evaluators and list Evals:

```python
import os

from roboflow import Roboflow

rf = Roboflow(api_key=os.environ["ROBOFLOW_API_KEY"])
evals = rf.workspace("your-workspace").workflow_evals()

catalog = evals.list_evaluators()
page = evals.list(limit=20)

for evaluation in page["items"]:
    print(evaluation["id"], evaluation["name"])
```

The same client exposes `create`, `start_run`, `get_run`, `wait_for_run`, and `openapi`. Create the Spec, Dataset, and bindings before starting a Run, using the schemas from `openapi()` and `list_evaluators()`.

## CLI

The [Roboflow CLI](https://docs.roboflow.com/reference/platform/cli/install-and-set-up-the-cli) includes the `workflow-eval` command group. Authenticate with your private API key and select a Workspace:

```bash
export ROBOFLOW_API_KEY="your-private-api-key"
roboflow --workspace your-workspace --json workflow-eval list
roboflow --workspace your-workspace --json workflow-eval evaluator list
roboflow --workspace your-workspace --json workflow-eval run list --eval your-eval-id
```

Use `roboflow workflow-eval --help` to see resource and Run commands. This command group evaluates Workflows; it is separate from model evaluation commands.

## MCP

Connect the [Roboflow MCP Server](https://docs.roboflow.com/agents/mcp-server) to your agent. Ask it to load `workflow_evals_agent_manifest` and the Workflow Evals guidance before creating an Eval. The tools cover evaluator discovery, Spec validation, authoring, Runs, and results through the same public API.

For example, ask: "Create a Workflow Eval for this Workflow using these test images. Help me define ground truth and Checks, validate the setup, and show me the credit estimate before starting a Run."