← back
Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs
Takeaway
Treat evaluation as continuously running reliability infrastructure for the whole agent workflow.
Summary
- Production agent evaluations must measure planning, tool use, workflow execution, recovery, and coordination rather than final-answer accuracy alone.
- The proposed evaluation hierarchy combines repeatable benchmarks, realistic scenario simulations, and production telemetry.
- Detailed traces and targeted human review feed a continuous loop that turns observed failures into offline regression scenarios.
- Operational metrics include completion, tool success, escalations, safety, latency, cost, and recovery, governed through an evaluation control plane.
production-evalsobservabilityreliability
Original description
As AI systems evolve from chat interfaces into autonomous agents capable of reasoning, planning, and tool usage, traditional evaluation approaches are breaking down. Offline benchmarks and static datasets fail to capture the complexity, non-determinism, and operational risks of real-world AI systems operating in production environments. In this talk, I’ll share practical lessons and architectural patterns for building evaluation systems for agentic AI workflows at scale. We’ll explore how modern AI platforms are shifting from one-time benchmark testing toward continuous evaluation pipelines integrated directly into production infrastructure. Topics include: Why offline evals fail for autonomous AI systems Evaluating tool use, planning, reasoning, and multi-step workflows Online vs offline eval architectures Human-in-the-loop evaluation systems Detecting drift, hallucinations, and unsafe behaviors Building feedback loops for continuous improvement Observability and telemetry for agentic workflows Reliability metrics beyond model accuracy Attendees will leave with practical frameworks for designing scalable evaluation systems capable of measuring real-world AI behavior, reliability, and operational impact. Speakers: Nishant Gupta (Meta Superintelligence Labs): Nishant Gupta is a Software Engineering Tech Lead at Meta Superintelligence Labs focused on building the training and inference AI Infrastructure. LinkedIn: / nishantgupta-ai GitHub: https://github.com/nishantgpt-lab