Eval-Driven Development
For LLM systems, evals are the unit of progress: you cannot improve what you cannot measure, and you cannot measure a non-deterministic system with intuition. But the discipline starts with looking at real outputs, not with writing graders in advance.
LLM and agent systems have an effectively infinite failure surface, so traditional test-first development breaks down: you cannot enumerate the ways a model will surprise you before you ship. The leading practitioners therefore invert the slogan. "Eval-driven development" in the literal sense (write evaluators before features) is discouraged; the recommended order is error analysis first, evaluators second, writing graders only for the failure modes you actually observe. Source: Hamel Husain / Shreya Shankar, https://hamel.dev/blog/posts/evals-faq/, 2026-05-31
Error analysis is the core loop
The highest-signal activity is reading traces. Spend 30 minutes reviewing 20 to 50 real outputs whenever you make a significant change. One domain expert (a "benevolent dictator") writes open-ended notes on each trace, focusing on the first failure observed, then those notes are grouped into a failure taxonomy (axial coding) and counted. Iterate until theoretical saturation: review at least 100 traces, and if roughly 20 in a row reveal no new category, stop. The point is to prioritize failures that happen, not failures you imagined. Source: Hamel/Shankar, evals-faq, 2026-05-31
The cost hierarchy of evaluators
Not every failure deserves infrastructure. Pick the cheapest grader that catches the error:
- Assertions / regex - deterministic checks (valid JSON, required field present, no banned string). Near-zero cost; almost always worth it.
- Reference-based - compare to a known-good answer when one exists.
- LLM-as-judge - only when the failure is subjective and frequent enough to justify aligning a judge against expert labels ("critique shadowing").
Always run a cost-benefit check before building an automated eval. Source: Hamel/Shankar, evals-faq, 2026-05-31
The three levels
Evals operate at three levels: Level 1 unit tests (assertions on traces), Level 2 human and model evaluation (LLM-as-judge, calibrated to a human), and Level 3 A/B testing in production. Lower levels gate merges; higher levels validate business impact. Source: Hamel/Shankar, evals-faq, 2026-05-31
What to evaluate
| System | Primary eval unit |
|---|---|
| Text/content generation | final output against rubric and examples |
| Extraction/classification | structured output vs labels and validators |
| Tool-using agent | trajectory: tool choice, args, observations, final outcome |
| Retrieval system | answer quality plus citation/source coverage |
| UI generation | screenshot/design rubric plus code constraints |
| Wiki/source compile | compiled truth accuracy, links, timeline evidence, searchability |
Agents need Agent Trajectory Evaluation because the path can be wrong even when the final sentence sounds fine.
Anti-patterns
- Building a judge before reading failures.
- Optimizing for a public benchmark that does not match user value.
- Treating LLM-as-judge as ground truth without calibration.
- Averaging away catastrophic failures.
- Measuring only final answers for tool-using systems.
- Letting eval cases rot after the product changes.
Good eval suites are living artifacts. They should change when production failures reveal new failure modes.
Why it matters here
This is Empirical Verification specialized for stochastic systems and the measurement arm of the harness-beats-model thesis: the team that improves fastest is the one whose harness can tell whether a change helped. It also disciplines Tool-Calling Design (you measure a tool by agent success rate on real tasks) and Context Engineering (you prune context against an eval, not a hunch).
Concept Position
| Field | Value |
|---|---|
| Concept family | Brain, memory, and retrieval |
| Concept owned | For LLM systems, evals are the unit of progress: you cannot improve what you cannot measure, and you cannot measure a non-deterministic sys... |
| Category map | Concept System Map |
Timeline
- 2026-07-01 | Concepts category refresh added this page to the Brain, memory, and retrieval family, linked it to Concept System Map, and kept it standalone because it owns this reusable mental model: For LLM systems, evals are the unit of progress: you cannot improve what you cannot measure, and you cannot measure a non-deterministic sys... Source: User request, 2026-07-01
- 2026-06-19 | Added eval-unit table and anti-patterns, tying eval-driven development to Agentic Quality System and Agent Trajectory Evaluation. Source: whole-wiki concept review, 2026-06-19
- 2026-05-31 | Page created from Hamel Husain and Shreya Shankar's LLM Evals FAQ. Captured the error-analysis-first inversion (literal eval-first is discouraged), the assertion/reference/judge cost hierarchy, theoretical saturation at ~100 traces, and the three-level model. Source: https://hamel.dev/blog/posts/evals-faq/, 2026-05-31