← back
Computer-use models will agentify the web, not APIs — Dhruv Batra, Yutori
Takeaway
Agents must work with the web’s existing visual interfaces rather than assume comprehensive API coverage.
Summary
- Batra argues that computer-use models will automate the long tail of the web because many organizations will never expose useful APIs.
- Restaurant menus and school procurement pages illustrate information scattered across text, scanned PDFs, image galleries, and manual request workflows.
- Even modern sites may load answers asynchronously or encode important state through visual interface elements.
- Existing APIs remain sensible where available, but browser interaction is necessary to handle heterogeneous interfaces and slowly changing institutions.
computer-useyutoriweb-agents
Original description
To learn what a US school district is buying, you file a Freedom of Information Act request. Someone scans the email you sent, puts the scan on Google Drive, and attaches the relevant PDFs. Dhruv Batra's question is whether anyone seriously expects that office to publish an MCP server. He grants the popular claim that agents will drive most of the action on the web, then rejects its usual next step, that the web will meet them with APIs. The head of the distribution might. The long tail, some 200 million active sites where infrastructure changes over decades, will not. Reading the HTML instead does not save you, because much of what you see was never written down anywhere. A basketball score is missing from the page that first loads and arrives later as JSON. A product page contains no text reading sold out, only a quantity of zero and a script that grays the option out. State is calculated and rendered rather than stored, which makes the browser closer to a game engine than a document and makes pixels the source of truth. He calls this the bitter lesson for web agents: scaffolding built per site does not generalize, and the general solution is the one the web was actually built for. Their Navigator model runs screenshot in and clicks out, now writes JavaScript when that is quicker, and checks the result on screen. It misses 8 of 300 trajectories on a benchmark he thinks should be retired. Speaker info: https://x.com/DhruvBatra_ / dhruv-batra-dbatra https://dhruvbatra.com Timestamps: 0:00 - The argument, and the part of it that is wrong 3:23 - Restaurant menus on easy, medium, and hard mode 5:31 - School district procurement, up to a FOIA request 7:38 - 200 million active sites that change slowly 8:42 - Why reading the HTML does not rescue it 10:14 - Sold out is not text, it is a rendered zero 11:33 - The browser is a rendering engine, pixels are the truth 12:53 - Navigator, and writing JavaScript when that is faster 15:51 - Are computer use models actually good enough 17:20 - Latency and cost per task 18:28 - Another layer of mess that we will call an API