← back
Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind
Takeaway
Collaborative commerce agents should progressively clarify intent and adapt both their research and presentation to the user.
Summary
- Shopping agents must help users articulate fuzzy preferences instead of assuming they already know precise search terms.
- A discovery, research, and adaptive-response loop combines preference elicitation, background comparison, and suitable visual or tabular presentation.
- The working state combines conversation history, personal context, hard constraints, inferred visual preferences with confidence, and live variables such as inventory.
- State evaluation checks fact retention, confidence calibration, and whether changing a query changes only the relevant inferred constraints.
multimodal-agentscommercepreference-elicitation
Original description
Asked to redo a living room on a budget, the agent Nidhi Kaushik Vyas walks through does not start recommending. It works out what it does not know, then picks the single question worth asking: how wide is the room. Everything downstream is moot if the furniture will not fit, so that one answer carries the most information per turn. Choosing the next question by expected information gain, rather than working down a checklist, is the heart of her argument about agents that handle fuzzy intent. Most shopping agents behave like a wrapper around the search bar, she says, assuming the user already holds a well formed goal and the vocabulary to express it. Real users arrive with a vibe, and closing that articulation gap is the agent's job rather than theirs. Her loop runs discovery, research, response. Discovery assembles a working state from past conversation, personal context and any reference images, separating the hard constraints in a query from the soft ones a picture only implies, attaching a confidence score to each, and flagging the variables that must refresh in real time because stale inventory makes an answer worthless. Research decides how to ask: for a subjective preference a visual board beats a text question, and hovers and clicks then feed the confidence model. Response picks a shape, a summary for a policy question, a comparison table for two products, imagery for style. Holding it together is an autorater at every stage, including one that flips part of a query to check the extracted constraints move when they should and stay put when they should not. Speaker info: / nidhivyas Timestamps: 0:00 - Agents that behave like a wrapper around the search bar 1:08 - The articulation gap: users arrive with a vibe 2:03 - The loop: discovery, research, response 4:45 - Building a working state from images and context 6:33 - Variables that have to refresh in real time 7:28 - Grading the state: facts, calibration, counterfactuals 8:21 - The intent gap, and picking the highest value question 10:10 - Bridging a constraint to the product ontology 11:04 - When a visual board beats a text question 12:57 - Choosing the response format, and grading it 16:38 - Four takeaways 17:34 - Questions: merchant ontologies, UCP, and agent buyers