Skip to main content
An evaluation scores the conversations from one or more simulations using an LLM judge. It produces per-metric scores, detects behavioral failures, and supports human annotation for calibration.
New here? The Quickstart walks you through running your first evaluation end to end. This page is the full reference for every section and option.

​
See your evaluations

Navigate to Evaluations in the sidebar to see every evaluation in your organization.
Evaluations list
Each row shows the evaluation name, the simulations whose conversations were scored, who created it, the date, the number of behavioral errors detected, and its status (Completed, Running, Failed, or Cancelled). Use Filter to narrow by creator or status, Search to filter by evaluation or simulation name, and click any row to open the detail page.

​
Create an evaluation

Click New Evaluation on the Evaluations page — or, from the Simulations page, open the kebab menu on a completed simulation and select New Evaluation to start with that simulation pre-selected.
New Evaluation form — name the run, pick a simulation, and select metrics.
​
string
required
A name for this run — e.g. “Core Support — Identity & Policy Check.”
​
array
required
One or more completed simulations to evaluate. The selector lists all completed simulations; search or use the Created by me filter to narrow it.
​
array
required
The metrics to score. Goal Completion is always included and can’t be deselected. Add any built-in or custom metrics you care about — see Metrics for the full list and how to create your own.
Click Run Evaluation to start. Arklex scores each conversation and updates the status to Completed when done.

​
Understand the scores

Before diving into the detail page, it helps to know the two scales in play:
  • Most metrics are scored 1–5, where 5 is the best possible performance. On the detail page these are shown as a score out of 5 with a progress bar and a band label.
  • Goal Completion is scored 0–1, where 1.0 means the simulated user’s goal was fully achieved. Final conversation scores also use the 0–1 scale.
Band labels apply to the 1–5 metrics based on the percentage of the maximum score:

​
Read an evaluation’s results

The detail page opens with a stats strip — total Conversations scored and Avg Turns per conversation — and a left sidebar with six sections:
A card for each numeric metric, split into Turn-level (scored per assistant turn) and Conversation-level (scored once for the full conversation). Each card shows the metric name and score, a progress bar, and a band label from the table above.
Quantitative metrics cards with score bars and band labels
For metrics that produce categorical labels (e.g. sentiment, tone, response type), this shows the label distribution across all conversations. Expand a label row to see where it was assigned, then click through to open the conversation at that turn.
Qualitative metrics label distribution
Behavioral failures detected by the judge, grouped by severity: Critical (red), High (orange), Medium (amber), and Low (indigo). Each entry shows the severity and category, how many times it occurred, a description, a suggested fix where the judge provided one, and links to the exact conversations and turns where it appeared.
Unique errors grouped by severity
A table of every scored conversation:Click any row to open the conversation modal with the full transcript and per-turn scores. Expand the Reasoning section on an assistant turn to read the judge’s explanation — the fastest way to spot prompt improvements or calibration issues. Use Previous / Next to move between conversations without closing the modal.
Conversation modal with per-turn scores and judge reasoning
A per-turn view where reviewers add their own scores without overwriting the judge’s. See Annotations for the full workflow.
Evaluation Annotations
Agreement rates between the judge and human reviewers, per metric — the evidence for refining a metric’s prompt. See Annotations for details.

​
Rerun an evaluation

Open the kebab menu on any evaluation row and select Rerun. The sheet pre-fills the name and summarizes the original configuration (simulations, metrics, model); edit the name to distinguish the run, then click Run Evaluation. The rerun creates a new evaluation record using the same simulations and metrics as the original.

​
FAQ

Yes. The simulation selector supports multiple selections, which is useful for rolling several simulation runs up into a single scored view.
Most metrics use a 1–5 scale where 5 is best. Goal Completion and final conversation scores use a 0–1 scale, where 1.0 is a fully achieved goal. Exact criteria depend on each metric’s definition and the judge’s interpretation of its rubric.
Start with Unique Errors to find specific failures, then review the linked conversations to understand the pattern. Update your agent’s system prompt, knowledge base, or configuration to address the root cause, and rerun the simulation and evaluation to confirm the improvement.