← back
The maturity phases of running evals — Phil Hetzel, Braintrust
Takeaway
Begin with documented expert judgments on real failure modes, then scale evaluation sophistication with agent complexity.
Summary
- Evals establish confidence before deployment, while observability maintains confidence under real production usage.
- Start with meaningful failure modes supplied by builders or domain experts instead of attempting exhaustive coverage of every possible agent behavior.
- An evaluation combines a task, representative input data, and scoring functions; imperfect model judges can still provide useful directional evidence.
- The maturity path runs from initial checks through measurement, complexity management, and advanced techniques; early human annotations should include both ratings and reasons.
- Domain-specific annotation interfaces help capture expert reasoning that can later inform scalable model-based judging.
evalsbraintrusthuman-annotation
Original description
Most teams approach evals like unit tests and try to cover every possible failure. Phil Hetzel from Braintrust argues that is the wrong frame: enumerate your known failure modes, cover those specifically, and ship. The goal is a flywheel where production traces surface what is going wrong, feed back into offline experimentation, and guide the next improvement. The session walks four maturity stages: vibe checking with documented human justifications not just thumbs up or down, LLM as judge built from those justifications at scale, then the hard part, tool calls that touch external systems. Context gathering tools are manageable. CRUD tools are not, because you have to represent the state of external systems at the exact moment the original trace ran. Timestamp queries against a vector database and injecting captured system state directly into the trace are two approaches for getting there. Speaker info: / philliphetzel