🎨 Multimodal
Vision-language models, video understanding, image generation, multimodal agents. GPT-4V, Claude vision, Gemini, open-source VLMs.
The workflow
flowchart LR
A[Image / Video<br/>input] --> B[Vision encoder<br/>ViT / CLIP]
B --> C[Cross-attention<br/>to LLM tokens]
C --> D[LLM generates<br/>text or actions]
D --> E{Output mode}
E -->|Text| F[Description /<br/>Q&A]
E -->|Image| G[Diffusion<br/>decoder]
E -->|Action| H[Robot / GUI<br/>agent]
The interesting frontier — UIs, robots, video — once the vision encoder can reliably ground language.
Key takeaways
Videos (31)
Moondream: how does a tiny vision model slap so hard? — Vikhyat Korrapati
A tiny VLM can punch above its weight when scope is narrow and synthetic data is meticulously engineered rather than scraped from larger models.
HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori
Use HTML as an editable source format for agent-generated visuals, letting browsers handle layout and rendering.
Vision AI in 2025 — Peter Robicheaux, Roboflow
Vision needs DinoV2-style self-supervised pre-training feeding into transformer detectors to unlock language-style scaling — and harder benchmarks like RF100-VL to measure it.
HTML Is All Agents Need — James Russo, HeyGen
Use familiar web languages for agent creativity and deterministic browser rendering to turn compositions into reliable video output.
From Text to Vision to Voice Exploring Multimodality with Open AI: Romain Huet
GPT-4o collapses audio+vision+text into one omni-model, enabling natural real-time human-computer interaction at half the price of GPT-4 Turbo.
SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind
Lower latency and multimodal editing can make generated media part of everyday creative, educational, and localization workflows.
Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face
Affordable, hackable robots let a wider community shape voice and embodied AI interfaces.
Waymo's EMMA: Teaching Cars to Think - Jyh Jing Hwang, Waymo
Multimodal LLMs like Gemini turn driving into a generalist reasoning task, and Waymo's EMMA shows camera-only end-to-end planning can beat heavy specialized stacks.
Veo 3 for Developers — Paige Bailey, Google DeepMind
Veo 3 + Imagen 4 + Lyria 2 collapse video, image and music generation into one developer-accessible stack with native multimodal control and SynthID safeguards.
Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind
A reasoning model can orchestrate specialized generators to turn mixed source material into multiple useful output formats.
Real-time Experiments with an AI Co-Scientist - Stefania Druga, fmr. Google Deepmind
AI co-scientists work best when they observe real-world experiments live via sensor multimodality, not just retrospective paper analysis.
FLUX, Open Research, and the Future of Visual AI — Stephen Batifol, Black Forest Labs
Black Forest Labs is pushing open-source visual AI toward 'visual intelligence' — combined generation, editing, and multi-reference composition — with Flux 2 as the latest base.
Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind
Collaborative commerce agents should progressively clarify intent and adapt both their research and presentation to the user.
Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind
Training frontier image/video diffusion models like VEO is mostly a data-curation, sampling, and distillation problem — not a modeling problem.
ComfyUI Full Workshop — first workshop from ComfyAnonymous himself!
ComfyUI's node-based architecture plus embedded-metadata sharability turned it into the de-facto open visual-gen platform with millions of users and instant cross-model interoperability.
Prompt to Pipeline: Building with Google's Gen Media Stack — Paige & Guillaume, Google DeepMind
Build media workflows around customer needs while accounting for rapid changes in general model capabilities.
The Multimodal Future of Education: Stefania Druga
Multimodal AI tutors are inevitable; building child-facing tools like Cognimates teaches AI literacy and critical thinking before mental models calcify.
AI Music Generation, From Prompt to Production: Phlo Young
AI music tools like Suno and Udio reward unintuitive prompt craft and pen-name experimentation more than musical training.
See, Hear, Speak, Draw: Logan Kilpatrick & Simón Fishman
Today you can chain GPT-4V and DALL-E 3 through text to mimic unified multimodal reasoning; tomorrow that bridging code disappears.
Robots as professional Chefs - Nikhil Abraham, CloudChef
Generic dual-arm robots plus cooking-specific multimodal embeddings can replicate professional chef labor at sub-human cost in real kitchens.
Google Photos Magic Editor: GenAI Under the Hood of a Billion-User App - Kelvin Ma, Google Photos
A billion-user GenAI app is really a system of on-device models with bespoke benchmarks, hardware co-design, and model management — generative quality alone isn't enough.
Voice agents with Realtime Video — Sidney Primas, LemonSlice
Interactive video avatars require expressive audio conditioning and causal, efficient video generation beyond visual realism alone.
Let's go Bananas with GenMedia — Guillaume Vernade, Google DeepMind
Generative-media APIs support both iterative asset creation and live multimodal experiences, with model choice shaped by the desired interaction.
The State of Generative Media - Gorkem Yurtseven, FAL
Generative media is a brand-new market — ads, e-commerce try-on, and (especially) video are the early product-market fits driving its explosion.
The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor
Real-time control could turn generated video into a programmable, interactive medium.
120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson
CLIP's text/image embedding similarity enables open-ended multimodal games and apps with a fraction of the code traditional CV required.
Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful
Agentic consumer video editing combines media understanding and code-based composition with approval and accessible correction tools.
Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs
Video memory requires persistent, source-grounded relationships across time and modalities, beyond clip retrieval.
Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD
Multi-modal apps win on perceived responsiveness via parallelism, structured streaming, and early playback while later assets still generate.
When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain
Agents working with large multimodal files need structured, queryable metadata and a data execution harness to make analysis practical at scale.
Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal
Azure AI Studio centralizes a vast multi-modal multi-vendor model catalog with end-to-end multimodal demos like menu reasoning and voice-preserving video translation.