📐 Evals
How to actually measure LLM and agent quality — golden sets, LLM-as-judge, regression gates, production tracing, observability.
The workflow
flowchart LR
A[Production traces] --> B[Sample & label<br/>golden set]
B --> C{Eval type}
C -->|Reference| D[Exact / BLEU /<br/>code-exec]
C -->|Reference-free| E[LLM-as-judge<br/>rubric scored]
C -->|Human| F[Pairwise<br/>preference]
D --> G[Aggregate metric]
E --> G
F --> G
G --> H[Regression gate<br/>in CI]
You cannot ship LLM products without evals. The most-watched talks all converge on: golden set + LLM-judge + CI gate.
Key takeaways
Videos (105)
Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil
Agents fail in the real world primarily because we evaluate them like LLMs; building cost-aware, environment-grounded benchmarks is the gating problem for production agents.
Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind
Evaluate both skill invocation and task performance, then retire obsolete capability skills while protecting organizational preferences.
Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Evaluate each stage of a multimodal pipeline against human-aligned criteria and log the full flow before building learning loops.
Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize
Evaluate agents at three layers — router decisions, individual skill correctness, and the convergence of the overall path — not just final-answer quality.
Evals 101 — Doug Guthrie, Braintrust
Treat evals as the central flywheel — connect offline test datasets, production traces and human review so every prompt or model change is measurably better.
How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh
Domain-specific eval systems are built bottom-up from unit tests, trace logging and frictionless human review — not from buying generic tools or jumping straight to LLM-as-judge.
The maturity phases of running evals — Phil Hetzel, Braintrust
Begin with documented expert judgments on real failure modes, then scale evaluation sophistication with agent complexity.
Five hard earned lessons about Evals — Ankur Goyal, Braintrust
Treat evals as a continuously engineered system, not synthetic data plus a judge, so a new model release can flip a feature from unviable to shippable.
[Evals Workshop] Mastering AI Evaluation: From Playground to Production
Evals are tasks + datasets + scores; cross-validate human judgment against scores to know whether to fix your evals or your app.
Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth
Assess agent reliability using success thresholds and checks for reward hacking, rather than headline benchmark scores alone.
Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind
Broader expert participation and transparent evaluation setups are essential to measuring useful real-world AI capabilities.
Evals Are Not Unit Tests — Ido Pesok, Vercel v0
Treat application evals as statistical measurements over real user traffic, not unit tests — prompt tweaks alone never close the demo-to-prod gap.
AI Agents, Meet Test Driven Development
Treat AI products like TDD systems with continuous eval datasets and LLM-as-judge metrics, not one-shot prompt engineering.
Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith
Treat benchmarking as a subset of evaluation, and pair performance tools (GuideLLM) with accuracy harnesses (lm-eval-harness, OpenAI Evals) to make production LLM rollouts measurable.
Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran
Stop benchmarking models and start instrumenting your app's component-level traces with task-specific LLM-as-judge or heuristic evals.
Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize
AI PMs should treat evals as the new product spec — non-deterministic, data-dependent, and the durable moat for any agentic product.
Why should anyone care about Evals? — Manu Goyal, Braintrust
Evals are the laboratory that lets you iterate offline and turn production traffic into the next training set — without them you ship blind.
Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain
Treat agent improvement as evidence-driven mining and experimentation over production traces.
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR
METR's measurement shows AI tools can slow experienced devs even when they feel faster, and capability growth is tightly coupled to compute scaling.
Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse
Improve the definition of success and its domain grounding before spending more tokens optimizing against it.
Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer
General LLMs collapse on noisy real-world finance queries and context; domain-specific models stay robust on the failure modes that matter in production.
Dark Factory: How OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc
Replace static benchmark evals with adaptive eval pipelines that evolve alongside your agents as you ship faster than you can review diffs.
What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench
Benchmarks all trend up while models still confidently bullshit on nonsense prompts; epistemic pushback is a major remaining gap.
The Future of Evals - Ankur Goyal, Braintrust
Eval work has been painfully manual; with Claude 4-class models, agents like Braintrust Loop can now autonomously improve prompts, datasets and scorers.
LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize
Combine multi-level observability with calibrated evaluation signals to improve agents without overlooking regressions.
Ending AI Slop — Thais Castello Branco, Taste Labs
Reduce AI slop by decomposing subjective goals into contextual, measurable components while preserving room for creative variation.
Why building eval platforms is hard — Phil Hetzel, Braintrust
Evals platforms are multi-persona, multi-stage systems problems that quickly outgrow spreadsheets and homegrown loops once teams take agent quality seriously.
Why Agent Hype can fall short of reality – Joel Becker, METR
Trust the exponential time-horizon trend for raw capability, but expect a gap between benchmark hype and real-world developer productivity gains.
Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar
Select and validate coding models using maintainability and security evidence alongside functional accuracy.
Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI
LLM judges only work when calibrated against human annotations per specific error type using prompt optimization like GEPA.
Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft
Useful evals combine representative simulations, explicit launch gates, and an operational feedback loop.
7 Habits of Highly Effective Generative AI Evaluations - Justin Muller
Evals exist to discover problems, not to compute a vanity score — building one is what turns a stuck GenAI prototype into a scalable production workload.
AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash
An eval platform becomes more effective when domain experts can directly shape datasets, rubrics, and workflows.
Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear
Pick a single business-aligned reliability metric for your AI app and iterate the prompt, model and data against it — generic NLP metrics are noise.
Evals Are Broken, Use Them Anyway — Ara Khan, Cline
Use evaluations as imperfect engineering evidence, combining task relevance with real-world observation.
2025 is the Year of Evals! Just like 2024, and 2023, and … — John Dickerson, CEO Mozilla AI
Agentic systems taking real actions are finally forcing evaluation tooling out of CIO-only sales and into board-level enterprise budgets in 2025.
Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop
Move from offline eval sets to production monitoring with implicit/explicit signals and live experiments — agents fail in ways unit tests can't anticipate.
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize
Production eval pipelines mean traces feeding many calibrated LLM judges per trace, with drift monitoring — modeled on what Duolingo and Reddit actually do.
New York Times' Connections: A Case Study on NLP in Word Games — Shafik Quoraishee, NYT Games
Connections is a reproducible NLP benchmark for abstract reasoning where graph-coloring formulations outperform raw semantic similarity.
Shipping complex AI applications — Braintrust & Trainline
Shipping production AI agents requires the same eval-and-observability discipline Trainline applies via Braintrust to keep agentic ticketing reliable.
Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor
Coding evals are moving from clean snippets to dynamic, real-world, multi-hour codebase tasks with auto-generated tests to combat contamination and brittleness.
Fuzzing in the GenAI Era — Leonard Tang, Haize Labs
Treating GenAI eval as adversarial fuzzing exposes brittleness that static golden-set tests miss, and the judge itself must be evaluated to be trusted.
Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic
Treat prompts as managed variables and use GEPA-style genetic optimization with golden-set evals to systematically improve agent reliability.
Iterating on LLM apps at scale Learnings from Discord: Ian Webster
At Discord scale, simple deterministic evals run on every PR like unit tests beat fancy LLM-graded eval pipelines for shipping safely.
Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind
Safety evaluation for mental-health AI must measure clinically appropriate responses in context, including the harm of unnecessary refusal.
Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier
Treat probabilistic agents like a data flywheel: instrument traces so any run becomes a replayable eval, then mine implicit feedback to drive continuous improvement.
From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Treat production-derived simulations as a continuously refreshed benchmark for the entire agent stack.
20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna
Define state of the art against your application and users, using representative evidence rather than a generic leaderboard winner.
How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado
Mature AI products require evals owned jointly by PMs and engineers, run in CI, with tracing across multi-tool agent flows — Zapier's 300% accuracy gain proves it.
The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI
Build benchmarks as validated measurement systems that both represent real tasks and direct useful research progress.
SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius
Reliable coding-agent benchmarks depend as much on fresh, validated tasks and controlled infrastructure as on the models being ranked.
The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Combine fixed evaluators with adaptive agent-based analysis to find failures across complex execution trajectories.
Mission-Critical Evals at Scale (Learnings from 100k medical decisions)
Real-time reference-free evals (LLM-as-judge + confidence) prioritize human review where it matters and let mission-critical AI scale beyond what clinicians could ever cover.
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
Evaluate coding agents with original, broadly distributed tasks and leakage-resistant environments, then inspect behaviors behind aggregate scores.
SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)
Generate realistic evaluation trajectories on demand so agent development does not wait for manual datasets or slow production feedback.
Fighting AI with AI — Lawrence Jones, Incident
Make evaluation and debugging data accessible through focused tools and files so AI can help maintain complex AI systems.
How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)
World-class AI products spend the majority of engineering time on evals and observability, not prompts — that's how Notion ships fast at consumer scale.
Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence
Specify rules, roles, domain meaning, and robustness requirements explicitly before generating tests for an agent.
Your Evals Are Meaningless (And Here's How to Fix Them)
Build evals as a dynamic system with SME-authored datasets and domain-specific LLM-judge criteria — don't rely on framework defaults that drift away from your users' definition of good.
Agent Evals: Finally, With The Map
A complete agent eval program covers both semantic and behavioral dimensions and treats the LLM-judge layer (EvalOps) as a first-class optimization target.
Evaluating AI Search: A Practical Framework for Augmented AI Systems — Quotient AI + Tavily
Evaluating production AI search needs live LLM-judge monitoring on real traffic, not just static benchmarks like SimpleQA, because both the web and user intent keep moving.
How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
Agent observability must measure behavioral quality and handle large text-rich traces alongside conventional reliability metrics.
Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK — Cedric Vidal, Microsoft
Use Azure AI Toolkit + Evaluation SDK to spot-check models, then scale to dataset evals, treating evaluation as a layered application-level concern.
Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai
Aesthetic evals must center human perception — current metrics like FID and CLIP miss what people actually find broken in generative imagery.
The Build-Operate Divide: Bridging Product Vision and AI Operational Reality
Crossing the V1-to-V2 quality chasm in AI products comes from a fast eval-iteration loop plus disciplined human-in-the-loop review, not better base models.
Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd
Design security-learning tasks with graduated difficulty and objective evidence of the specific capability being trained.
Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software
Evaluate long-running agents using multiple measures of task difficulty rather than a single hours-or-tokens headline.
How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads
Develop evaluation criteria from concrete failures and human agreement before scaling the rating process.
Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents - Aparna Dhinakaran
Agent evals must span tool calls, trajectories, and full conversations — and the evals themselves need to evolve alongside the agent.
Judging LLMs: Alex Volkov
Production LLM apps must log/trace everything from day one, and use a layered eval stack (programmatic + human + LLM-judge) rather than skipping straight to fine-tuning.
Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize
Eval is just testing for non-deterministic systems — capture traces, look at them, then write code + LLM-judge + meta-evals before tuning prompts or swapping models.
Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft
Record agent state transitions so production failures can be debugged even when fresh model calls differ.
Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs
Treat evaluation as continuously running reliability infrastructure for the whole agent workflow.
The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen
Evaluate historical personas against time-specific documentary evidence, separately from fluency and personality consistency.
Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley
Measure learning across related task sequences instead of assuming independent task scores reveal adaptation ability.
Verifiable Environments for AI in Biology — Kenny Workman, LatchBio
Biology agents need domain-specific, verifiable analysis environments that test scientific reasoning on real data.
When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI
Trustworthy benchmarks must resist contamination and reward hacking while measuring outcomes users actually value.
[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)
Eval is a methodology discipline (calibrate metrics with humans, expand to many signals) not a one-shot benchmark — learn from Google Search's 300-metric setup.
open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara
Open-rag-eval scores RAG quality without golden datasets using nugget-based generation evaluation and Umbrela retrieval scoring.
Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog
Treat recurring agent disagreement as a signal to clarify ambiguous policies and gather targeted human feedback.
Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face
Improving AI cyber defense requires realistic reasoning benchmarks that capture system-level access-control failures.
Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop
Improve agents through evidence from real failures and adaptable validation rather than evaluations tightly coupled to one implementation.
Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo
Clinical AI evaluation must capture the significance of omissions and changed decisions, not merely surface-level faithfulness.
Evaling Video Slop — Maor Bril, Character.ai
Video evaluation should judge temporal quality and explain defects quickly enough to guide generation while it is happening.
Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Evaluate and improve agents through repeatable environment rollouts with explicit verification and preserved trajectories.
Benchmarks Are Memes: How What We Measure Shapes AI—and Us - Alex Duffy, Every.to
Whoever designs the next benchmark shapes what frontier models become—build benchmarks that are multifaceted, generative, evolutionary, and experiential, not just easy to score.
Taming Rogue AI Agents with Observability-Driven Evaluation — Jim Bennett, Galileo
Reliable agents need step-level LLM-as-judge evaluations baked into observability pipelines from day one, not just final-answer scoring.
SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI
Long-running coding agents need independent behavioral checks and hardened verifiers because project completion and reward hacking are both difficult to assess.
Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs
Long-horizon evaluation must test domain transfer and emergent behavior, with real-world deployments complementing simulations.
Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft
Treat observability as a 3-phase loop (evaluate → monitor → optimize) built on OpenTelemetry tracing and agent-specific evaluators — non-determinism makes this a continuous practice, not a one-time eval.
Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust
Update evaluation coverage whenever model capabilities change the architecture and its possible failure modes.
Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers
Calibrating individual benchmark items can expose differences and data-quality problems hidden by aggregate accuracy.
Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs
Computer-use evaluations should test adaptation across verified environment variations rather than reward memorized action sequences.
Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i
Trustworthy coding benchmarks should measure realistic engineering outcomes with valid tests and contamination-resistant environments.
Don’t be data poor — Anuj Iravane, Anterior
Generate healthcare evaluation data from diverse policy paths and structured patient journeys rather than asking an LLM for complete records in one shot.
CI in the Era of AI: From Unit Tests to Stochastic Evals — Nathan Sobo, Zed
Shipping AI features inside Zed forced the team from fully-deterministic CI to programmatic stochastic evals where assertions interrogate specific agent steps.
How to run Evals at Scale: Thinking beyond Accuracy or Similarity — Muktesh Mishra, Adobe
Pick the eval metric to match the application type (RAG vs code-gen vs agent) and treat evals as a continuous, data-centric, parallelizable engineering practice.
The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani
Public benchmarks are rigged by Goodhart's Law and selective comparison — build small domain-specific evals from real production queries instead.
E-Values Evaluating the Values of AI: Sheila Gulati and Nischal Nadhamuni
Evals must become multifaceted, dynamic, and values-aware before agentic systems become fully self-sufficient, or we'll lose the ability to course-correct.
BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data
Pairing batch-based LLM evaluations with synthetic data generated from real support tickets is a fast path from POC chatbot to production-ready RAG.
Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia
Clinical conversational AI needs hazard-driven simulation and validated patient behavior before deployment because harmful conversations cannot be rolled back.
Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory
Software-maintenance agents are still poor at bug detection — high false positives, narrow reasoning, and lack of holistic code reading limit real-world reliability today.
Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen
llm-eval gives teams a local, Hydra-configurable CLI to test prompt and model changes across providers with stable multi-sample reporting.
Will Agent evaluation via MCP Stabilize Agent Networks? - Ari Heljakka
Exposing evaluators via MCP lets any agent get scored, explained feedback inline — turning evals from a one-off harness into a continuous stabilization loop for agent networks.
How to evaluate a model for your use case: Emmanuel Turlay
Use LLM-as-judge with task-specific rubrics and visualize score distributions — generic NLP benchmarks won't tell you which model fits your application.