Skip to main content
Metric Alignment uses your team’s human annotations to automatically improve a metric’s scoring prompt. When the LLM judge and your reviewers consistently disagree on a metric, run Alignment to distill those disagreements into a refined rubric — without writing the new prompt by hand.
Alignment builds on resolved annotations, so annotate and resolve at least one evaluation first. See Annotations for that workflow, and Metrics for how metrics are defined.
Alignment works on both built-in and custom metrics, and accepting a result behaves differently for each:
  • Built-in metric — creates a new custom metric (a fork) containing the distilled guidelines. The original built-in metric is unchanged.
  • Custom metric — bumps the metric to a new version with the refined prompt.

​
Start an alignment run

Open a metric’s detail page, find the Alignment section at the bottom, and click Start Alignment. Configure the run:
​
array
required
One or more completed evaluations that have resolved annotations for this metric. Only evaluations with resolved annotation data appear in the list.
​
string
The provider and model used to analyze disagreements and distill guidelines. Uses the platform default if left blank.
​
string
The model used to verify the distilled guidelines. Optional.
​
number
The minimum score gap between the judge and a human annotation for the pair to count as a disagreement.
Click Start — the run enters the queue immediately and moves through four phases: loading annotations, building disagreement traces, distilling guidelines, and finalizing. Most runs finish within a few minutes, depending on the number of conversations and the reflection model.
Cancel any in-flight run with Cancel. Cancellation is idempotent — cancelling an already-completed run does nothing.

​
Review and accept results

When a run reaches Aligned, expand it to see the distilled guidelines (in plain language), how many disagreements were found across the source evaluations, and how many agreeing rows were skipped. If the guidelines look good, click Accept:
  • On a built-in metric, a new custom metric is created with the distilled guidelines as its scoring prompt — the built-in stays untouched.
  • On a custom metric, the metric updates to a new version, and all future evaluations using it apply the new rubric.
If a run found zero disagreements or the output is too generic, discard it and try again with different source evaluations or a different reflection model.

​
Alignment history

The Alignment section lists all past runs for the metric, newest first, with each run’s status, creation time, and source evaluations. Expand any completed run to view its guidelines — even after accepting.

​
FAQ

Editing means writing the new rubric by hand. Alignment derives it automatically by analyzing disagreements between the judge’s scores and your team’s resolved annotations — most useful when you see a pattern of disagreement but aren’t sure how to express the correction in the prompt.
Annotation Calibration (on the evaluation detail page) shows you where the judge and your reviewers agree or disagree, per metric. Metric Alignment goes a step further and acts on those disagreements, rewriting the metric’s rubric to close the gap. Use Calibration to spot the problem, Alignment to fix it.
Yes. Accepting the result forks it into a new custom metric containing the distilled guidelines; the original built-in metric isn’t modified.
No hard minimum, but more resolved annotations give the reflection model more signal. Fewer than 5–10 resolved disagreements tends to produce generic, low-confidence guidelines — aim to annotate and resolve at least one full evaluation first.