Annotations

Annotations let team members score evaluation results with their own judgment alongside the LLM judge’s automated scores. When a score is in doubt, reviewers enter their own values, admins resolve disagreements, and the full history stays accessible for audit. Results export to CSV for stakeholders who aren’t on the platform.

---

​

How annotations work

Every completed evaluation has an Annotations tab — a spreadsheet with one row per conversation turn (or per conversation) and one column per metric. Each cell shows the LLM judge’s auto score alongside any human annotation entered for it. Reviewers annotate independently: each person sees the auto score and enters their own value without seeing other reviewers’ scores first, which prevents anchoring bias. A majority indicator previews the likely resolved score once enough annotations agree, and rows with unresolved disagreements are visually distinct from rows where everyone agrees. What you see on the tab depends on your role:

---

​

Choose an annotation scope

A toggle at the top of the tab switches between two scopes, which are independent — an evaluation can carry annotations at both levels at once:

---

​

Dispute a judge score

When a reviewer thinks the judge scored a turn wrong, they find the turn on the Annotations tab and enter their own score for that metric. The auto score stays visible as a chip (“Auto: X.X”) next to the input for direct comparison. The input adapts to the metric type:

• Quantitative, 1–5 scale — a row of numbered buttons; click a number to select, click again to clear.
• Quantitative, 0–1 decimal scale — a numeric input field.
• Qualitative — a dropdown of the metric’s label options.

Click Save Annotation to commit. Unsaved changes are tracked per turn and highlighted so nothing is lost by accident.

---

​

Resolve disagreements

Resolving a turn or conversation produces a single resolved score per metric by strict majority vote:

• The value with the most annotations wins.
• On a tie, the metric is flagged in the resolve dialog and the owner manually selects the resolved value before confirming. That choice is recorded with an override marker distinct from a majority-vote result.

Three resolution actions are available: Resolved rows lock — their cells become read-only for everyone. Owners can Unresolve any turn or conversation to reopen it. Resolved scores feed directly into the Annotation Calibration tab, which computes agreement rates between auto-evaluation and human judgment per metric.

---

​

Add comments

Comments are separate from scores and live in the Comments tab of the conversation modal, for free-form discussion. They’re visible to all team members with access to the evaluation.

---

​

Annotation history

Every annotation is stored with the reviewer’s identity and timestamp. In the conversation modal, expand the history panel on a turn to see all submitted annotations — including past values that were later updated — plus the auto-score history showing what the judge scored at each evaluation run.

---

​

Export to CSV

Click Export CSV at the top right of the Annotations tab to download all annotation data for the evaluation — auto scores, every reviewer’s scores, and the resolved scores for each turn and conversation. This is the recommended way to share results with stakeholders who aren’t invited to Arklex, such as subject-matter experts or compliance reviewers. They can review the full breakdown in any spreadsheet tool without an account.

---

​

FAQ