Compare Trained Models

Compare test-set accuracy and median latency for models on the same Dataset Version.

Compare trained models to find accuracy and speed tradeoffs. The comparison includes models trained on or uploaded to the same Dataset Version.

Comparison requests are read-only. They do not start evaluations.

Model Comparison requires Model Evaluation access in your Workspace. Without access, the "Compare" tab shows an upgrade prompt and comparison requests return an error.

About

Each model row contains its metrics, median latency, frontier status, and exclusion reason. Models without evaluations remain in the response.

The comparison uses the latest completed evaluation for each model. It does not fall back to an older eligible evaluation.

The default frontier metric depends on the Project type:

• Object Detection and Instance Segmentation: mAP@50:95 (mAP5095).

• Semantic Segmentation: mIoU (mIoU).

• Classification: F1 (f1).

Use another value from availableMetrics to compare a different metric. API accuracy values are ratios. The CLI displays percentages.

Latency is the median server-side inference time in milliseconds. It includes preprocessing and postprocessing. It excludes network time.

A model is on the Pareto frontier if no model is at least as accurate and as fast, with one strict improvement. Identical measurements can put multiple models on the frontier. Frontier membership is not a recommendation.

Web App

Open "Compare" under "Models" in the Project sidebar. Use the "Dataset Version" list to select a version.

The chart includes models that have all supported metrics and latency. Select a point to open its model card.

Use the metric selector to change the chart metric and frontier. Detection Projects can include mAP@50, mAP@50:95, mAP@75, Precision, Recall, and F1.

The "Some models need evaluation updates" panel lists models with missing comparison results and explains what each model needs.

• Click "Run Evaluation" for a model that has never been evaluated.

• Click "Update Evaluation" when an evaluation lacks latency or an accuracy metric.

• Click "Retry Evaluation" after a failed run.

Click "Update All" to evaluate all eligible models in the panel. Evaluations already queued or running are excluded.

Before any model has a comparison result, the tab shows the next step instead of the chart (ex: "Train a Model", "View Training", "Run Evaluations", "Update Evaluations"). "Run Evaluations" and "Update Evaluations" replace "Update All" there.

A NAS child without an evaluation links to its NAS training. A NAS child with an existing evaluation can use the evaluation retry flow.

HTTP API

Read the comparison for one Dataset Version. Requests do not start or update evaluations.

Compare model evaluations

get

Compare models on one Dataset Version. The operation includes models that have no evaluation. It is read-only and does not start evaluations.

Authorizations

AuthorizationstringRequired

Roboflow API key or OAuth access token.

Path parameters

workspacestringRequired

Workspace URL slug.

Query parameters

projectstring · min: 1Required

Project URL slug.

Example: chess-pieces

versionstringRequired

Positive Dataset Version number.

Example: 131Pattern: ^[1-9][0-9]*$

frontierMetricstring · enumOptional

Metric used to calculate the Pareto frontier. The response lists supported choices in availableMetrics.

Example: mAP5095Possible values:

Responses

200

Successful response.

application/json

projectstringRequired

Project URL slug.

versionstringRequired

Dataset Version number.

frontierMetricundefined · enumRequired

Metric used to calculate onFrontier. Null when the Project type has no supported default metric.

Possible values:

400

A required parameter is missing, malformed, or unsupported for this comparison.

application/json

401

Authentication is missing or invalid.

application/json

403

The Workspace does not have Model Evaluation access.

application/json

404

The Project or Version is missing or inaccessible. API-key scope failures can also return this status.

application/json

429

The request rate is too high.

application/json

get/{workspace}/model-evals/compare

GET /{workspace}/model-evals/compare?project=text&version=text HTTP/1.1
Host: api.roboflow.com
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "project": "chess",
  "version": "131",
  "frontierMetric": "mAP5095",
  "availableMetrics": [
    "mAP",
    "mAP5095",
    "mAP75",
    "precision",
    "recall",
    "f1"
  ],
  "models": [
    {
      "modelId": "acme/chess-fast",
      "evaluationId": "eval-fast",
      "metrics": {
        "mAP": 0.84,
        "mAP5095": 0.61,
        "mAP75": 0.68,
        "f1": 0.79,
        "precision": 0.82,
        "recall": 0.76
      },
      "medianLatencyMs": 3,
      "onFrontier": true,
      "exclusionReason": null
    },
    {
      "modelId": "acme/chess-new",
      "evaluationId": null,
      "metrics": {},
      "medianLatencyMs": null,
      "onFrontier": false,
      "exclusionReason": "evaluation_unavailable"
    }
  ]
}

Python SDK

Use Python SDK version 1.5.1 or later to get the same comparison response:

comparison = workspace.compare_model_evaluations(
    "chess-pieces",
    131,
    frontier_metric="mAP5095",
)

Use each row's modelId as the stable model reference. Use the server's onFrontier value instead of rebuilding the frontier.

CLI

Use roboflow version 1.5.1 or later to compare a Project's Dataset Version:

roboflow --workspace my-workspace eval compare \
  --project chess-pieces \
  --version 131 \
  --frontier-metric mAP5095

Add --json to return the complete response without display rounding. The key needs the model-eval:read scope and Project access.

MCP

Call the model_evals_compare tool with a Project slug, Version number, and optional frontier metric:

{
  "project_id": "chess-pieces",
  "version_number": 131,
  "frontier_metric": "mAP5095"
}

The tool uses the current Workspace. Use the returned onFrontier values to explain tradeoffs. The tool does not select a model.