← back
From MCP to Scale: Pipelines That Build Themselves — Rafael Levi, Bright Data
Takeaway
Use models for pipeline construction and repair, and deterministic scripts for repeated extraction at scale.
Summary
- Levi recommends using an LLM to build and maintain a scraper instead of paying it to interpret every page in a large collection job.
- Bright Data skills supply scraper-building instructions while its MCP tools retrieve HTML or markdown so the agent can inspect structure, identify selectors, and generate a reusable script.
- The live demo begins with Walmart, then switches to an audience-suggested Very marketplace to avoid reusing an existing scraper, reporting 90 collected products before a currency-encoding fix.
- Periodic agent checks can validate collected fields and repair broken scrapers; the same pattern supports large datasets or personal monitoring such as housing alerts.
web-scrapingmcpdata-pipelines
Original description
Scraping is not the hard part anymore. Maintaining scrapers is. This session shows what it looks like when an agent uses MCP to inspect a site, understand its structure, generate a production scraper, and keep that pipeline working when the site changes. Using Bright Data's MCP, APIs, and browser infrastructure, the flow moves from one-off extraction to something much more useful: agents that build parsers, save tokens by switching from page parsing to reusable scripts, and repair broken collection jobs without a human getting dragged in at 2am. If you're thinking about web data, automation, or agents that operate beyond a single prompt, this is a practical look at pipeline building at scale. Speaker info: / rafael-levi https://github.com/ScrapeAlchemist