The Web Scraper Didn't Die — It Got a Protocol

I spend a lot of time from Kathmandu watching the plumbing of the web get re-poured, not the flashy model releases sitting on top of it. Over the past twenty months, the unit of "AI reads a web page" quietly changed. It used to be a CSS selector or an XPath expression, hand-written against one site's HTML. Increasingly, it's a protocol call — Model Context Protocol (MCP) — paired with an API that hands back clean markdown instead of raw HTML. That's not a story about a clever new scraping library. It's an infrastructure story about who controls the pipes between AI systems and the open web, and it has moved faster than most infrastructure stories do.

The protocol layer: how MCP became the pipe

Anthropic open-sourced the Model Context Protocol on November 25, 2024, built by David Soria Parra and Justin Spahr-Summers and described in Anthropic's own launch post as "a new standard for connecting AI assistants to the systems where data lives." At launch, early adopters included Zed, Replit, Codeium, Sourcegraph, Block, and Apollo — a reasonable but unremarkable list for a new open standard.

What happened next is the part that actually earns the word "standard." OpenAI adopted MCP on March 26, 2025, across its Agents SDK, Responses API, and ChatGPT desktop app — Sam Altman posted at the time that "people love MCP and we are excited to add support across our products." Google DeepMind followed on April 9, 2025, with Demis Hassabis calling it a "rapidly emerging open standard for agentic AI." Microsoft built MCP support into Windows 11 and showcased it at Build 2025. Three direct competitors converged on the same protocol inside twelve months of its release — that's not typical behavior for infrastructure standards, which usually take years to consolidate, if they ever do.

On December 9, 2025, Anthropic donated MCP to a newly formed Agentic AI Foundation under the Linux Foundation, co-founded with Block and OpenAI — a governance move that matters because it takes the protocol out of any single vendor's hands. In that same donation post, Anthropic cited more than 97 million monthly SDK downloads across Python and TypeScript, and more than 10,000 active public MCP servers. That server count is worth a caveat: independent directories disagree with each other — the official registry counts fewer once you deduplicate, community trackers like PulseMCP report more — so treat 10,000+ as Anthropic's own figure, not a settled census. GitHub's own description of the project by mid-2026 calls it a protocol that "evolved from an experiment to a widely adopted industry standard," which is a strong claim, but by that point a defensible one.

MCP isn't just convenient, though — it's also a new attack surface. Invariant Labs disclosed in April 2025 that a maliciously crafted MCP tool description could get an AI agent to exfiltrate a user's WhatsApp message history without ever tripping an obvious warning, because the poisoning lived in text the model reads as instructions, not in code a security scanner would flag. Standardizing the pipe didn't standardize trust in what flows through it.

The extraction layer: Firecrawl and Brave turn pages into APIs

If MCP is the protocol, Firecrawl and Brave's Search API are two of the extraction tools most commonly wired up behind it. Firecrawl — YC-backed, founded by Caleb Peffer, Eric Ciarla, and Nicolas Silberstein Camara, grown out of an earlier product called Mendable — raised a $14.5M Series A led by Nexus Venture Partners on August 19, 2025, bringing its total funding to $16.2M. Its API exposes /scrape, /crawl, /map, /extract, and /search endpoints, returning either clean markdown or schema-validated JSON pulled through its own rendering engine. Pricing as published at firecrawl.dev/pricing runs Free (1,000 credits/month), Hobby ($16), Standard ($83, 100,000 credits), Growth ($333), and Scale ($749). At its Series A, Firecrawl's own blog and TechCrunch cited 43,000–48,000 GitHub stars; the repository has since moved from the mendableai to the firecrawl GitHub organization and grown substantially further — treat any current count as a live snapshot worth checking at the source rather than a fixed figure, since the only value with a solid dated citation is the August 2025 one.

Brave Search API runs an independent index Brave describes as covering "over 30 billion pages," refreshed by more than 100 million page updates daily, and markets itself directly as "the leading search tool for applications that use Claude MCP." It removed its free tier in February 2026, moving to metered billing at roughly $5 per 1,000 queries. Its relevance grew by circumstance as much as by design: after Microsoft retired the Bing Search API in mid-2025, Brave became one of the only independent, Western search indexes still available to developers at API scale — a genuine gap it happened to be positioned to fill.

Why the old scraper broke — and why it isn't dead

The honest version of this story isn't "traditional scrapers are dead." It's that the default access pattern inverted, and the old approach got pushed underneath a new layer rather than eliminated.

The economic driver is Cloudflare's July 1, 2025 "Content Independence Day" change, which made blocking AI crawlers the default for every new domain on its network — a network that proxies roughly 20% of all web traffic — and introduced an HTTP 402 pay-per-crawl mechanism. The Associated Press, Time, Quora, and Stack Overflow publicly backed the move. Cloudflare's own Radar data explains why publishers wanted it: in the window of June 19–26, 2025, Anthropic's Claude crawler made close to 71,000 HTML page requests for every referral click it sent back to a publisher (roughly 70,900:1), against about 1,091:1 for OpenAI's GPTBot, 195:1 for Perplexity, and 41:1 for Bingbot. That ratio moves fast — Cloudflare measured ClaudeBot at 38,065:1 later in July 2025, down from 286,930:1 in January — but the direction is consistent: AI crawlers take vastly more than they send back, by a widening margin relative to traditional search. Reader behavior compounds the squeeze. Pew Research Center, analyzing 68,879 real searches from 900 U.S. adults (published July 22, 2025), found people clicked a traditional search result in only 8% of visits where an AI-generated summary was present, versus 15% without one, and just 1% clicked a link inside the summary itself. A brittle fetch()-based scraper hitting a Cloudflare-protected page today is increasingly likely to silently ingest a "Just a moment…" interstitial instead of content — which is the practical reason teams moved to protocol- and API-based access in the first place.

None of that means Playwright and Puppeteer are obsolete. DOM-driven browser automation remains the reliability gold standard for known, previously mapped pages, and the AI browser agents marketed as the scraper's replacement — Anthropic's Computer Use, OpenAI's CUA, Browserbase's Stagehand — run a real Chromium instance underneath, typically via Playwright's CDP layer. What changed is where the intelligence sits: instead of a human writing a CSS selector for one page's layout, an LLM interprets the page (or markdown extracted from it) and decides what to click or extract, which is more resilient to layout changes but costs real money per page — Firecrawl's JSON extraction mode runs about 5 credits per page — and adds latency a hand-written selector never had. It's also not immune to the same anti-bot systems: Cloudflare Turnstile, DataDome, and PerimeterX defend against AI browser agents the same way they defend against curl scripts. And the legal exposure is live, not hypothetical — News Corp sued Perplexity in the Southern District of New York in October 2024 (No. 24-cv-7984) over republished, licensed content, seeking $150,000 per infringement, while News Corp separately signed a five-year, roughly $250M licensing deal with OpenAI the same year. Getting the data is easier now. Being allowed to have gotten it is a separate, unresolved question.

The orchestration layer: scout, reader, synthesizer

Once you have a protocol and an extraction API, the natural next layer is coordinating multiple calls into a single task — a scout that discovers URLs, a reader that extracts them, and a synthesizer that merges results into structured output. This is a legitimate, buildable pattern, not vendor marketing fiction. CrewAI, which raised $18M in October 2024 led by boldstart and Insight Partners and has roughly 45,000 GitHub stars, models this directly as role-based agent "crews." LangGraph reached a 1.0 general-availability release in October 2025, modeling the same idea as a stateful graph instead of fixed roles. Microsoft's older AutoGen framework is now in maintenance mode, explicitly superseded by the newer Microsoft Agent Framework. Meanwhile the model-provider SDKs closed the gap from the other direction: OpenAI's Agents SDK and Responses API (March 2025) and Anthropic's Claude Agent SDK (launched September 29, 2025 alongside Sonnet 4.5, renamed from the earlier Claude Code SDK) both ship native MCP client support, so "orchestration" increasingly means three chained function calls inside one route handler rather than a separate framework at all.

Put together, the shape of 2026's AI-to-web stack is three layers: a protocol for connecting agents to tools and data (MCP), an extraction layer that turns pages into markdown or typed JSON (Firecrawl, Brave), and a thin orchestration layer coordinating calls across both. The CSS selector didn't disappear from this picture — it moved one layer down, running quietly inside Playwright, inside the extraction API, doing exactly the job it always did. What's new is everything stacked on top of it.

Sources

• Anthropic, "Introducing the Model Context Protocol," anthropic.com, November 25, 2024
• Anthropic, "Donating the Model Context Protocol to the Agentic AI Foundation," anthropic.com, December 9, 2025
• TechCrunch, "OpenAI adopts rival Anthropic's standard for connecting AI models to data," March 26, 2025
• TechCrunch, "Google DeepMind adopts Anthropic's MCP standard," April 9, 2025
• TechCrunch, "AI crawler Firecrawl raises $14.5M Series A," August 19, 2025
• Firecrawl, company blog and pricing page, firecrawl.dev
• Brave, "Brave Search API," brave.com/search/api
• Cloudflare, "Content Independence Day," blog.cloudflare.com, July 1, 2025
• Cloudflare Radar, "The crawl before the fall of referrals," blog.cloudflare.com, 2025
• Pew Research Center, "Google users are less likely to click on links when an AI summary appears," July 22, 2025
• Invariant Labs, "MCP Security Notification: Tool Poisoning Attacks," April 2025
• TechCrunch, "News Corp outlets sue Perplexity for copyright infringement," October 21, 2024
• CrewAI funding announcement, October 22, 2024; LangGraph 1.0 release notes, October 2025

Written by Abhishek Kushwaha, founder and writer at Global Tech Search, based in Kathmandu, Nepal.