← back
Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Takeaway
Evaluate each stage of a multimodal pipeline against human-aligned criteria and log the full flow before building learning loops.
Summary
- Uber Eats aims to improve merchant photography selectively while preserving food authenticity, brand identity, and marketplace diversity.
- An understanding agent produces structured observations; a router selects enhancement or preservation, followed by editing, iterative QA, and final publication checks.
- Router evaluation uses confusion matrices, precision, and recall, with representative human-labeled data as the reference.
- Comprehensive shared logs support diagnosis and continuous improvement; failed edits are withheld, and routing guardrails prioritize catching poor images.
multimodal-evalsimage-editinguber-eats
Original description
This talk covers how Uber designed evals for its food enhancement agent, which edits food photography to better present dishes for smaller, independent Uber Eats merchants, along with the pitfalls and lessons learned along the way. The problem is uniquely hard: the system must stay faithful to the original dish, preserve each merchant's brand and packaging, and avoid homogenizing the marketplace, all without an existing playbook for multimodal evals in a narrow domain. Soumya Gupta and Jai Chopra explain how they navigated reward hacking, built a closed feedback loop combining offline and online signals, and balanced creativity against rigid safety guardrails at scale. ML and applied AI practitioners working on multimodal systems, agentic pipelines, or eval design will take away practical strategies for narrow-domain multimodal evaluations, countering reward hacking, and production feedback loops. Speakers: Soumya Gupta — ML Engineer, Uber Soumya is a Tech Lead and Applied AI Engineer who architects and scales production-grade generative AI and computer vision systems at Uber. X/Twitter: https://x.com/guptasoumya12 LinkedIn: / guptasoumya12 Jai Chopra — Product Manager, Uber Jai is a Product Lead on Uber's Applied AI team and previously worked at Cruise and several startups. X/Twitter: https://x.com/jai_chopra LinkedIn: / jaichopra