← back
The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI
Takeaway
Build benchmarks as validated measurement systems that both represent real tasks and direct useful research progress.
Summary
- Agent capabilities are advancing faster than reliable measurement, leaving enterprises hesitant to deploy in demanding settings.
- Open benchmarks can define future capability goals alongside field deployments, red teaming, and private evaluations; Snorkel committed $3 million to benchmark grants.
- Benchmark science includes rigorous task validation, distributional diversity, and robust evaluation, while benchmark design also needs a thesis and accessible adoption.
- GPQA illustrates adversarial expert quality control through authors, reviewers, adjudicators, revisions, and agreement-linked incentives.
benchmarkstask-qualityopen-evaluation
Original description
ARC AGI 3 launched a few weeks before this talk with every task human solvable and frontier models under 1%. That gap is the argument: our ability to measure AI has fallen behind our ability to build it, and benchmarks that actually shape the field are bets on where capabilities are going, not snapshots of where they are. Vincent Chen draws a framework from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants. The science is task quality, distributional diversity, model headroom, and robust eval methodology. The art is having a thesis (Terminal Bench bet on the CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first class citizen. He closes on three axes he thinks the next generation of benchmarks needs to cover: environment complexity, autonomy horizon, and output complexity beyond plain text. Speaker info: https://x.com/vincentsunnchen / vincentsunnchen https://github.com/vincentschen