> For the complete documentation index, see [llms.txt](https://docs.roboflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.roboflow.com/workflows/tutorials/compare-models-with-workflow-evals.md). # Compare Models with Workflow Evals Compare models on a bolt-counting task with Workflow Evals, reviewed ground truth, and an agent connected to the Roboflow MCP Server. In this tutorial, I use [Workflow Evals](/workflows/evaluate/workflow-evals.md) to find which off-the-shelf model counts bolts most accurately on my test images. You can follow the same process: define ground truth, set Checks, and ask an agent to build and compare candidate Workflows. Workflow Evals is like model evaluation, but you can check more than model predictions. You can evaluate counts, transformed predictions, or other Workflow outputs against expected values. ## Prerequisites To follow along, you need: * A [Roboflow account](https://app.roboflow.com) and a Workspace with access to Workflows and Evals. * Test images and a count you have checked for each image. * An agent connected to the [Roboflow MCP Server](https://docs.roboflow.com/agents/mcp-server). I use Claude in this example. * Access to the models you want to compare, including any credentials their Blocks require. Review [Workflow Evals credits](/workflows/evaluate/workflow-evals.md#credits) before starting Runs. Evaluation and inference charges are separate. ## Counting bolts Imagine I have a camera above my workbench and want to count how many bolts remain in a plastic container. I don't want to train my own fine-tuned model. I just want an off-the-shelf model inside a Workflow that returns a bolt count. I could use Roboflow Playground to test one or two images against different models, but Workflow Evals lets me run the same set of images through many Workflows and compare their results. ![Overhead camera view of loose bolts in a plastic container on a workbench](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-c1a8a068be3214d95bd013643945cf1aa4bfc718%2Fworkflow-evals-bolts-example.jpg?alt=media) {% hint style="info" %} The screenshots keep the original "screw" labels and `screw_count` field. These correspond to the bolts and `bolt_count` examples in this tutorial. The conversation is edited to use the same bolt terminology. {% endhint %} ## Baseline Workflow Start with a [Workflow](/workflows/build/create-a-workflow.md) that accepts an `image` input and exposes a numeric `bolt_count` output. My baseline uses SAM3 to find instances, then a Property Definition Block with the `SequenceLength` operation to count the predictions. Each candidate must accept the same image and return its count through `bolt_count` so the Eval can compare them. For the LLM candidates, I use this shared prompt, then a JSON Parser Block to extract `bolt_count` as the Workflow output: ``` Count every individual bolt visible in this image exactly once. Do not double-count any bolt. Respond with only JSON in exactly this form: {"bolt_count": } ``` Ask your agent to check the available Block schemas and validate each Workflow before running it. Keep the counting task and output contract the same across candidates. ## Ground truth To get started, I created a ground truth Dataset. For this tutorial, I labeled five images using Auto Label for instance segmentation, then counted and manually confirmed the bolts in each image. My ground truth showed between 9 and 11 bolts per image. Each image is a Case with an input and known correct answers. To create your Eval, open "Workflows" in your Workspace, select "Evals", and click "Create Eval". Choose the baseline in "Select Workflow", add your images in "Upload Ground Truth", and enter each image's expected `bolt_count` in "Label Ground Truth". You can keep mask annotations, but this comparison only scores the count. It's tempting to outsource this step to an agent entirely. Don't. This is where you define success and set the standard the agent will try to meet. Use tools like Auto Label, but carefully review their results. This is where I invest human attention so I can save time elsewhere. ![Ground truth editor with an image, an expected bolt count, and labeled masks](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-1f36722b46b9c4a38a4ef19ddbb94ff856d6f9aa%2Fworkflow-evals-bolts-ground-truth.gif?alt=media) ## Checks Next, I set the expectations through Checks. Workflows return JSON outputs, and a Check compares a selected output with its ground truth using an evaluator. For example, if a Workflow returns `pass_or_fail`, I can check that some Cases return `true` and others return `false`. In this example, I check that the bolt count matches the expected count. In "Define Eval Checks", select a numeric evaluator for `bolt_count` and pick "Matches ground truth exactly". A Case passes only when its count matches exactly. Then continue to "Run Evaluation", review the setup and credit estimate, and start the baseline Run. Other Checks can allow a numeric tolerance or compare detections with annotations. For this example, though, I stick to exact counts so LLMs and models that return detections can be compared without requiring them all to produce masks. ![Defining an exact-count Check with zero allowed difference from the expected bolt count](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-c88885e6851cf37ba8f32e5e6aaddd559535ea33%2Fworkflow-evals-bolts-define-checks.gif?alt=media) ## Agent conversation Now that you have a carefully defined set of success criteria, you can use agents to build dozens of Workflows and score them against your benchmark. Using the Roboflow MCP Server, I told Claude to find my Eval, list the available models that could complete the task, and create Workflows to find which scored best. Before asking your agent to do this, have it load `workflow_evals_agent_manifest` and the Workflow Evals guidance, as described in the [MCP instructions](/workflows/evaluate/workflow-evals.md#mcp). Here's a slightly edited version of my conversation, including the tool calls Claude chose to use. > Me: Create a plan for me: I want to find the best off-the-shelf models, implemented in a Workflow, for the Eval I've created. Create a plan for the Workflows to create and how to test them against the Eval. > Claude: I looked at the baseline Run first. SAM3 passes 0 of 5 images. It over-counts every time, by about 4 bolts per image, but its masks score 0.91 mIoU. Five images are also too few to separate models well. > > Tools called: `workflow_evals_specs_get`, `workflow_evals_datasets_get`, `workflow_evals_runs_list`, `workflow_evals_evaluators_list`, `workflow_evals_executions_list`, `workflow_evals_results_overview` > Me: OK, I don't have more images, but you can edit the existing Eval to only check for a count of bolts instead of bounding boxes or masks. > Claude: Done. The Eval now has one Check, "Bolt count matches annotation", which requires an exact match. > > SAM3's high mask score plus its over-count suggests duplicate instances of the same bolt. If that's the cause, a confidence threshold might help without a new model. Inspect the individual Cases to confirm it. > > Revised plan: > > * A. LLMs: Claude, OpenAI, Gemini, Qwen, Kimi, Grok. Prompted to return JSON with a numeric `bolt_count`. > * B. Grounding models: Florence-2, YOLO-World. Count the detections. > * C. SAM3 fixes: baseline plus higher confidence thresholds. Count the filtered predictions. > > Then: smoke-test each Workflow on one Case, run all of them on all five Cases, and repeat the LLM Runs three times in total. With five images, run-to-run noise can otherwise decide the ranking. Rank by average count error, then bias, then exact matches, then latency. Treat the result as a screen, not a verdict. > > Tools called: `workflow_evals_specs_validate`, `workflow_evals_specs_update`, `workflow_evals_get`, `workflow_evals_bindings_validate`, `workflow_evals_update` > Me: OK, execute the plan in Roboflow. I'm not adding more images to the Eval; I've confirmed the ground truth is correct. I want to find the best model for the task. Create the Workflows, run the Eval against all of them, and tell me which LLMs achieve the best score on the Eval. > Claude: I'll check SAM3's per-image misses and the Block schemas, then build the Workflows, validate and smoke-test each one, and run them all against the Eval. > > Tools called: `workflow_blocks_get_schema` x12, `workflow_specs_validate` x9, `workflows_create` x17, `workflow_evals_runs_start` x4, `workflow_evals_runs_get` x13, `workflow_evals_runs_retry_failed` x3, `workflows_update` x1, `workflow_evals_results_overview` x43, and a few more. Along the way, Kimi ran out of tokens on the smoke test. Claude raised its `max_tokens` to 8192 and ran it again. Claude generated 17 new Workflows: 12 LLM candidates, two grounding models, and three SAM3 confidence variants. With my existing SAM3 baseline, that made 18 Workflows. The first Run was a one-Case smoke test, followed by a full Run over five Cases, then two repeat Runs for the 12 LLM candidates. ![The 17 bolt-counting Workflows Claude created, one per model or setting](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-2e970f4c41e40495d869f3fd003c1f10aae552cd%2Fworkflow-evals-bolts-created-workflows.png?alt=media) ![Four Eval Runs showing the smoke test, the full comparison, and two LLM repeat Runs](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-b44dd181b564bc57e4f45093a5fa40e53be0c26a%2Fworkflow-evals-bolts-agent-runs.png?alt=media) ## Results The 12 LLM candidates each ran three times on the same five images, giving 15 trials per model. These are repeated trials, not 15 different test images. Exact matches count trials where the prediction equals the ground truth. Average count error is the mean absolute difference from the expected count, measured in bolts. These are the results reported for this example. They describe the models and settings tested here, rather than a general ranking for other tasks or a list of models currently available.
ModelExact matches out of 15Average count errorAverage latency
Gemini 3.1 Pro120.206.4 s
Claude Sonnet 5.590.603.6 s
Gemini 3.8 Flash90.7324.6 s
Claude Opus 5.580.804.1 s
Qwen 3.7 Plus81.003.3 s
Claude Haiku 5.570.673.1 s
Qwen 3.8 Max70.9312.8 s
Claude Fable 5.160.753.4 s
Grok 4.761.0727.8 s
GPT 6.1 Sol40.7313.0 s
GPT 5.541.4718.6 s
Kimi K2.631.4032.7 s
{% hint style="info" %} Claude Fable returned invalid outputs on 3 of its 15 trials. Its exact-match total includes all 15 trials, but its average count error uses only the 12 valid outputs. Review invalid outputs alongside accuracy when comparing candidates. {% endhint %} To my mild surprise, Claude recommended Gemini 3.1 Pro: it had the most exact matches and the lowest average count error in this comparison. However, its average latency was 6.4 seconds, so I won't be using that in production for this task anytime soon. Review [how Eval latency is measured](/workflows/evaluate/workflow-evals.md#latency) when choosing between speed and accuracy; it includes more than model inference time alone. After running the Eval, Claude also provided analysis: * The SAM3 baseline over-counts, possibly because of duplicate instances. This is a hypothesis to check in individual Cases. * SAM3 at confidence 0.7 has an average count error of 0.6 and latency of about 1.2 seconds per image, making it competitive with Claude Sonnet in this example. SAM3 ran once over five Cases; the LLM results pool three Runs. * YOLO-World and Florence-2 under-count in this example. Use "Overview" to compare Workflows and inspect individual failures, and "Metrics" to compare scores. Five images are a small test set. Before choosing a model for production, confirm the results on new images that cover your expected conditions. ![Eval results comparing each Workflow with ground truth and showing per-Case outcomes](https://1126624165-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoO0Ceoy9R0lHDZATcPyh%2Fuploads%2Fgit-blob-b48086b6663239ef71aaea5f23fe59f33ee39a91%2Fworkflow-evals-bolts-results-overview.gif?alt=media) ## Further tuning Now that I have an Eval set I've reviewed, I can compress weeks of Workflow building and experiment design into 30 minutes of prompting and reviewing. I can also try more focused improvements on the leading candidates. > Can you use the same Eval to test Gemini 3.1 Pro vs Claude Sonnet 5.5 at different model-level settings like temperature or thinking level? I'm not sure what options are available for each model. I want the best Eval outcome, and I want each model to perform at its best. Feel free to run more Evals, create more Workflows, or edit the Workflows already created. In this follow-up, Claude created seven new Workflows and ran the Eval five more times, then analyzed the results. Claude reported insights like "Gemini: The four settings are within noise of each other" and "Sonnet: Extended thinking made it worse, and temperature 0 and a larger image size did nothing." Based on this example, I'm happy with Gemini 3.1 Pro for further testing on this task. Those tuning observations come from the same five images; they still need confirmation on new images. ## Next steps Workflow Evals is available through the Web App, CLI, HTTP API, Python SDK, and MCP. Once you have ground truth and Checks, your agent can look for a better Workflow and check for regressions. Use these short prompts with your agent: > Plan Workflows to compare available models on my bolt-count Eval. > Test one Case first, then run all Cases three times per model. > Rank results by count error, exact matches, latency, and cost. > Tune the top candidates, then test on new images.