← back
Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind
Takeaway
A reasoning model can orchestrate specialized generators to turn mixed source material into multiple useful output formats.
Summary
- The Gemini API combines multimodal understanding with specialized models for image, speech, and other generation tasks.
- A NotebookLM-style agent first synthesizes PDFs, videos, images, and audio, then uses function calls to decide which assets to create.
- The example produces study summaries, infographics, and two-speaker podcast audio about the Attention Is All You Need paper.
- File uploads, inline data, selected timestamps, and context caching support larger inputs and repeated queries.
geminimultimodal-agentsfunction-calling
Original description
Draw arrows on a map and ask Gemini to generate a picture of what you see. It produces the Golden Gate Bridge. Not because it matched pixels, but because the image generation model is built on top of Gemini's world understanding and knows what those arrows are pointing at. Patrick Löber walks through the full any-to-any stack: multimodal understanding where Gemini ingests PDFs, video, and audio up to nine-plus hours at once, native image and speech generation called as tools from an agentic loop, and a live audio model where audio goes in and audio comes out through a single architecture with no cascaded pipeline. The session ends with the building blocks for a Notebook LM clone where a reasoning agent decides what to generate rather than a hardcoded workflow. Speaker info: https://x.com/patloeber / patrick-l%c3%b6ber-403022137 https://github.com/patrickloeber Timestamps: 0:00 Introduction to the session 0:58 Defining "Any-to-Any" and the Gemini ecosystem 2:56 Building a NotebookLM clone using an agentic approach 3:51 The agentic architecture for multimodal applications 4:50 Implementation details for multimodal understanding 6:10 Tips for audio/video processing and context caching 7:56 Multimodal generation phase 8:37 Native image and infographic generation 9:04 Native speech generation and podcast style audio 9:57 Implementing function/tool calling 11:28 The power of native generation models 12:37 Multi-language and accent capabilities in audio models 13:46 Live API and real-time interaction 15:06 Final summary and additional model shout-outs