← back
When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain
Takeaway
Agents working with large multimodal files need structured, queryable metadata and a data execution harness to make analysis practical at scale.
Summary
- Physical data such as video and robot telemetry needs a dedicated agent data harness: a small collection of files can expand into millions of nested frames, detections, and relationships.
- DataChain represents metadata with Pydantic schemas and translates Python operations into database queries, avoiding both millions of JSON sidecar files and a separate SQL workflow for researchers.
- In a dashcam demo, a coding agent selects a small YOLO model and per-frame analysis; processing roughly 90 videos takes 24 minutes and produces about 100,000 records.
- The resulting database quickly answers that 82 of 91 clips contain detected people, without downloading and parsing each metadata file.
- Generated schemas capture frame identifiers, timestamps, classes, confidence, bounding boxes, and file provenance such as paths, checksums, ETags, and sizes.
datachainphysical-datadata-harnesses
Original description
Ask an agent to find every night-time pedestrian frame across terabytes of dashcam video in S3. The first pass can cost thousands of dollars and run for hours or days. Once you’ve paid that cost, the agent’s favorite move - loop, inspect, re-derive - becomes the worst thing it can do. Most agent intuition comes from a different physics: coding and business automation, where recompute is cheap, verification means re-running code, and state fits in context. That physics is real, useful, and everywhere. But like mechanics at cosmic or particle scale, those intuitions stop working when scale changes. Large-scale data work is where the cracks start showing. OpenAI’s in-house data agent needed layers of context engineering to operate over 600 petabytes in a structured warehouse - and warehouses are still the forgiving version, with schemas, indexes, query engines, and cheaper ways to check the work. Now move to the data behind physical AI: video, logs, and sensor data from robots, vehicles, labs, and factories. Petabytes will never fit in a context window. "Just verify it" may mean another expensive inference job. A follow-up question should recall what the system already paid to discover, not trigger another perception pass. This talk is a tour of that other physics: what breaks, what inverts, and what replaces today’s harness assumptions when recompute is the enemy. Materialization becomes the default, recall becomes first-class, and the dataset - not the context window - becomes the unit of state. I’ll demonstrate the pattern live with Claude Code over raw data in S3, where follow-ups return in seconds and cents instead of reprocessing everything, with measured gaps on the order of millions between recompute and recall. Same word: harness. Different physics. Speakers: Dmitry Petrov (DataChain): Dmitry Petrov is the co-founder and CEO of DataChain, where he builds open-source infrastructure for AI agents to work with unstructured and physical-world data, and previously created DVC (Data Version Control). X/Twitter: https://x.com/FullStackML GitHub: https://github.com/dmpetrov