← back
The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Takeaway
Combine fixed evaluators with adaptive agent-based analysis to find failures across complex execution trajectories.
Summary
- Arize reports over 100 million monthly evaluations, but longer-running agents introduce failures that fixed-rubric judges may miss.
- Its Alex agent exposed issues including forgotten context, repeated tool calls, unclear completion, and inefficient trajectories.
- Agent-as-judge evaluation performs adaptive trace analysis alongside deterministic checks and conventional LLM judges.
- Arize's Signal is presented as a long-running evaluator that discovers recurring issues and can propose fixes through pull requests.
agent-as-judgeobservabilityarize
Original description
Across a dozen eval jobs Arize watches the top teams run, one pattern holds: the eval has to change as fast as the agent it grades. In 2023 an agent was barely more than a prompt; since then reasoning, tool calls, and long multi step loops piled on, and every jump in capability quietly broke the eval that came before. So the evals evolved with them. Deterministic checks catch what you can define up front, LLM as a judge adds the analysis a fixed rule cannot, and the newest step, agent as a judge, hunts for failure modes you would never think to write a check for and can open a pull request to fix what it finds. Aparna Dhinakaran's argument is that this arc, from static checks to an agent grading another agent, is where evals go next. Speaker info: https://x.com/aparnadhinak / aparnadhinakaran Timestamps: 0:00 - Opening: the future of the Evals track 2:06 - Why evals got harder as agents evolved 3:45 - From deterministic checks to LLM as a judge 4:36 - Agent as a judge, and where evals go next