← back
Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft
Takeaway
Useful evals combine representative simulations, explicit launch gates, and an operational feedback loop.
Summary
- Lyft evaluates customer-support agents offline before launch and online through production traces, graders, and human error analysis.
- Evaluation scores must gate releases and trigger clearly owned responses to regressions.
- A TauBench-inspired simulator pairs LangGraph agents with LLM users defined by intent, world state, and persona to test complete conversations.
- LLM judges assess interaction quality while deterministic assertions verify concrete actions such as granting concessions.
customer-supportllm-judgesimulation
Original description
Your agent passes offline evals at 90%. You ship. Production immediately finds failure modes your eval never saw. Sound familiar? The culprit is almost always the same: the "customer" in your offline eval is an off-the-shelf LLM that sounds nothing like your real users, and your synthetic test set doesn't capture how messy, angry, or off-topic real conversations get. Your eval was too easy. At Lyft, our customer-care agents resolve roughly a third of all customer issues — millions of conversations a month. To trust them at that scale, we built an adversarial user simulator: a fine-tuned LLM trained on real Lyft rider and driver transcripts that can role-play frustrated, confused, and adversarial users with the same distribution as production. It found regressions our synthetic dataset missed for months. This talk walks through the full eval lifecycle that surrounds it: the harness primitives that let any engineer write a benchmark in 20 lines, how we calibrate LLM-judge rubrics against human labels until they match inter-rater agreement, how we route failed production traces back into the offline test set, and the continual-learning loop that feeds improvements into prompts, harness, and the model. Slides: https://docs.google.com/presentation/... Speakers: Nick Ung (Lyft): Nick Ung leads Data Science for Safety & Customer Care at Lyft, where his team built and operates the multi-agent platform that powers AI agents resolving roughly a third of all Lyft customer issues. LinkedIn: / unglikteng Akshay Sharma (Lyft): Akshay Sharma is a Senior ML Engineer at Lyft and tech lead of the Lyft’s Customer Care Agent Architecture and the Builder Platform, which he co-founded with Nick. LinkedIn: / akshay-sharma-1995