Nobody skips evals because they disagree with them. They skip because the tooling looks like a quarter of work. It isn't - the minimum viable suite is a spreadsheet and an afternoon.
Thirty cases, four buckets
| Bucket | Count | What it proves |
|---|---|---|
| Happy path | 10 | The thing works on typical input |
| Edge cases | 10 | Empty fields, huge input, wrong language, ambiguity |
| Adversarial | 5 | Prompt injection, out-of-scope requests, data exfiltration attempts |
| Must-refuse | 5 | Requests it should decline, escalate, or hand to a human |
The last two buckets are the ones that catch the incidents. A suite of only happy paths tells you nothing you didn't already believe.
Four metrics that fit in a spreadsheet
- Task success - did it produce the right outcome, judged by a human or an exact-match check?
- Format validity - did the output parse against the schema, every time?
- Refusal correctness - did it decline exactly the cases in the must-refuse bucket, and none of the others?
- Cost and latency per run - p50 and p95, because the p95 is what your users complain about.
Score before you change anything, then re-score after every prompt edit. Prompt changes are not free: fixing one failure mode routinely reopens another, and without a baseline you will never notice.
The release rule
Write it down before you are under pressure to ship:
```
Release blocks if:
- any adversarial case leaks data or executes an out-of-scope action
- format validity < 99%
- task success drops more than 3 points below the last release
```
A rule agreed in advance survives a launch deadline. A judgement call made at 6pm on Friday does not.
Watch the same things in production
Evals are pre-launch. In production, log every run with input, retrieved context, tool calls, output, and cost, then sample 20 per week and score them with the same rubric. Two signals matter most: rising refusal rate (your knowledge base drifted from what users ask) and rising p95 latency (retries you are not seeing).
Human-in-the-loop is a design choice, not a fallback
Decide up front which actions require approval - anything irreversible, anything customer-visible, anything financial. An agent that drafts and waits is boring, useful, and shippable. An agent that acts unilaterally on day one is a postmortem waiting for a date.
Generate your own version with the [QA Agent](/qa-agent) tool.