ZERO REFERRALS · PART 3 OF 4
In Part 1, I described hard paywalls as vaults: the server checks your subscription and never sends the text if you don’t have one. In Part 2, I explained how language models work with meaning rather than exact wording. Put those two ideas together and you get the problem this post is about. An AI system doesn’t need to break into the vault if it can rebuild most of what’s inside from pieces lying around outside.
The referral line is falling
First, the scale. According to Similarweb data reported by Pelcro, traffic to the world’s 500 most-visited publishers fell 20% to 27% year over year between 2024 and 2025, an average loss of roughly 64 million visits a month. The New York Times reportedly saw referral traffic drop 27.4% in the second quarter of 2025 alone.
A caveat I’ll repeat throughout this post: Pelcro sells subscription software, so it has an interest in this story. Treat those numbers as industry estimates. But the direction is consistent across sources, and it matches what publishers themselves are saying.
The open pipeline loses traffic to zero-click AI summaries. The paywalled pipeline keeps it, at the cost of reach. (Figure 3 from the white paper.)
Who the bots are working for
Automated collection is growing quickly. Coronium.io, a proxy-services provider, estimates that major infrastructure providers handle up to 50 billion AI crawler requests a day, and that AI crawler traffic grew 757% in a single year.
Most AI bot traffic collects data. Very little of it serves a live reader. (Vendor estimate.)
The most telling number is the split. In early 2026, “training crawlers” (bots collecting content for model training) made up 49.9% of AI bot traffic. Only 2.2% of AI bot requests appeared to serve an actual user’s question. If those estimates are even roughly right, most AI crawling is extraction rather than anything that might send a human reader back to the source.
Two ways around the gate
AI systems get past paywalls in two broad ways.
The direct route works on soft paywalls. As Part 1 explained, many soft paywalls send the full article to the browser and hide it with an overlay. An agentic AI browser or a crawler reading raw HTML can simply skip the curtain.
The reconstruction route works even on hard paywalls, and it’s the more interesting one. The model never contacts the publisher’s server. Instead, it gathers the fragments a story leaves across the open web:
- quotes posted to X and other social platforms
- screenshots shared on professional networks like LinkedIn
- paraphrases and discussions on forums like Reddit
- open RSS feeds, cached previews, and search snippets
Using the pattern-matching and meaning-based processing described in Part 2, the model cross-references those pieces, fills in gaps from what it already knows, and writes a clean summary. No login is bypassed, and nothing shows up in the publisher’s server logs.
Fragments from across the open web go in, a synthesized version of the paywalled article comes out. (Figure 9 from the white paper.)
How much can it rebuild?
This is where the honest answer is “it depends, and the estimates vary.” Pelcro estimates that AI summaries capture 70% to 80% of a paywalled article’s value. Independent testing by Henk van Ess found AI tools reconstructed roughly half of the paywalled articles he sampled.
Two estimates of AI reconstruction. They measure different things, and one comes from a vendor.
Those aren’t directly comparable. One measures the share of an article’s value, the other the share of articles. But even the more conservative figure means a reader has a coin-flip chance of getting a usable version of a paywalled story for free.
Pelcro also describes four common ways users drive this behavior:
- Direct summarization prompts, which simply ask an assistant to summarize or recreate a paywalled article.
- Social fragment mining, where assistants with real-time access to platforms like X or Reddit compile discussion of a locked story.
- Granular metric extraction, which asks for specific numbers or takeaways, such as financial indicators from a Wall Street Journal piece, and captures the useful part without a visit.
- Academic reconstruction, which pieces together locked research from abstracts, citation networks, and open commentary.
Ghost citations
Even when an AI system tries to give credit, it doesn’t always get it right. SEO practitioner Andres Castro describes “ghost attributions”: citations to links that are hallucinated or out of date. Complex paywalls, aggressive lazy loading, and JavaScript-heavy pages can keep crawlers from reading the live page, so the model falls back on what it remembers and may cite a URL from memory rather than one it actually checked.
Castro argues this can hurt publishers twice. They lose accurate referral credit, and if readers click a broken or wrong link and immediately bounce, the system may start treating the publisher’s domain as low quality. That second claim comes from practitioner observation rather than published research, but it’s a plausible failure mode and worth watching.
Publishers push back
Publishers aren’t standing still.
Publishers’ defensive measures are growing fast.
By the end of 2023, 79% of the most widely used U.S. news sites were blocking OpenAI’s crawlers. TollBit, which monitors AI scraping for publishers, reported blocking 26 million scraping attempts in March 2025 alone, and Cloudflare saw bot traffic rise from 3% to 13% of traffic in a single quarter. Some publishers also deploy “honeypots,” decoy pages and paths designed to catch and flag automated collectors.
Those measures work against crawlers that come to the front door. They can’t do much about reconstruction, because reconstruction never touches the publisher’s servers. Perimeter security protects the perimeter. The fragments are already outside it.
Blocking keeps a publisher’s verified reporting out of the AI’s direct view. It doesn’t keep the story out of the answer.
There’s also an uncomfortable side effect, which I covered in the first series. When authoritative outlets block crawlers while free, low-quality sites stay wide open, AI systems end up drawing on whatever is left. Researchers call those gaps “data voids.” In the final post, I’ll look at what all of this means for the reader at the end of the chain, especially the one who can’t pay.
Up next: The Free-Tier Problem (Part 4 of 4)
Read the full paper: This companion series is adapted from Konderla, J. (2026). The Toll for Truth: Paywalls, Information Deserts, and the Asymmetric Web (Version 1.0) [White paper]. Zenodo. https://doi.org/10.5281/zenodo.23131192 The paper includes the complete argument and full bibliography.
Sources
Castro, A. (2026, May 31). Securing AI citation integrity and attribution accuracy to prevent misleading generative reports. https://andresseo.expert/geo/securing-ai-citation-integrity-attribution-accuracy/
Coronium.io. (2026, April 7). The AI crawler war: 50 billion daily bot requests reshaping web scraping (2026). https://www.coronium.io/blog/ai-web-scraping-crawler-war-2026
Fletcher, R. (2024, February 22). How many news websites block AI crawlers? Reuters Institute. https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
Pelcro. (2025, August 5). How AI chatbots are reconstructing paywalled content. https://www.pelcro.com/blog/how-ai-chatbots-are-reconstructing-paywalled-content
van Ess, H. (2025, February 10). How AI bots quietly dismantle paywalls via web search. Digital Digging. https://www.digitaldigging.org/p/how-ai-bots-quietly-dismantle-paywalls
Vasipenok, V. (2025, January 28). AI browsers are sneaking past paywalls: How do they do it? Quasa News. https://quasa.io/news/ai-browsers-are-sneaking-past-paywalls-how-do-they-do-it