← back
SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)
Takeaway
Generate realistic evaluation trajectories on demand so agent development does not wait for manual datasets or slow production feedback.
Summary
- Nubank and Snowglobe report that generating evaluation trajectories in simulation shortens agent iteration from weeks to hours, supporting a claimed twentyfold speedup.
- Manual multi-turn data creation is costly, while production experiments depend on sparse, noisy customer feedback and expose real users to variants.
- Snowglobe wraps the real agent, mocks selected tools, and uses personas and use cases to generate thousands of stateful conversations.
- Per-turn judges feed these trajectories into evaluation pipelines; Nubank connects the approach to improved customer satisfaction across production support agents.
agent-simulationsnowglobecustomer-support
Original description
Nubank serves 135 million customers, so an AI agent that mishandles a support conversation fails at scale. The talk opens with the result: five agents in production, higher customer satisfaction, and roughly 20 times faster shipping. Shreya Rajpal, CEO of Snowglobe, argues the thing that unlocked that pace was evals, and specifically simulated data standing in for real conversations. Good agent evals are hard because the data is multi turn and stateful, not single turn question and answer, and hand curating it and waiting on production to confirm can take forever. Snowglobe points at the agent and generates grounded simulations, a customer like Maria trying to order a credit card, complete with account context, tone, and intent, so you can ship, observe, simulate, and feed the results back in a tight loop. Human review found the simulated conversations comparable to real ones about 80% of the time, enough to bring up new agents, derisk changes, and protect the self service rate. With aligned metrics and cheap simulation, the team now tests open models and variant agents freely, because the eval bottleneck is gone. Speaker info: https://x.com/aman2304 https://x.com/ShreyaR / shreya-rajpal http://shreya-rajpal.com Timestamps: 0:00 - Opening with the results 0:39 - Nubank at 135 million customers 2:34 - Why evals matter most 2:47 - Why simulated data works 3:49 - What makes agent eval data hard 4:49 - How teams get eval data today 6:44 - Simulations in minutes, not weeks 7:35 - Pointing Snowglobe at your agent 8:48 - A grounded simulation: Maria orders a card 10:31 - Ship, observe, simulate, repeat 11:34 - How close simulations are to real 13:43 - Testing models and variants