On a website with 1.5 million pages, 99% of visits over the past year came from bots—not human readers, but crawlers deployed by AI companies. Real human visitors accounted for just 1%. These are the numbers recently shared by site owner Nick in his post titled 99% of My Website Traffic Is Bots, which sparked over 300 comments on Hacker News.
Consider an even more striking statistic: in a single week, Anthropic’s search crawler scraped 420,000 pages from his site. During that same week, Claude referred exactly 12 human visitors back to him. Scraping 420,000 pages in exchange for 12 visitors captures the essence of running a website in the AI era.
Nick makes his living on the web by compiling public records. He acknowledged the irony in his post: his own data was originally gathered by scraping public documents, so a crawler complaining about crawlers sounds inherently ironic. Yet that transparency lends weight to his grievance—when even scrapers say things have gone too far, a boundary has clearly been crossed.
Why AI Crawlers Scrape Your Site in Secret
Large language models are not born knowledgeable. Models like ChatGPT and Claude can answer questions because they ingested nearly the entire readable web during their training phase. Where did that text come from? It was fetched page by page by web crawlers.
Crawlers act like digital couriers for AI companies. The difference is that borrowing library books requires checkouts, whereas crawlers take content without asking. According to Cloudflare, by June 2026, 57.5% of web requests across its network originated from AI crawlers. In other words, for any random webpage open on your screen, more than half of its traffic isn’t human.
Cloudflare also published a “crawl-to-click ratio”—measuring how many pages a platform scrapes before referring a single human visitor. Anthropic led the pack: in early 2025, it took roughly 280,000 scraped pages to yield one visitor, dropping to 38,000 pages per visitor by mid-year. OpenAI averaged around 1,400 pages per visitor, while Perplexity rose from 55 pages to 195 pages per visitor. The vast majority of these scraped pages feed training pipelines—80% of AI crawl traffic is dedicated to model training, while only 2% stems from active user queries in chat interfaces.
Figure: Comparison of crawl-to-click ratios across major AI platforms. Source: Cloudflare Official Blog
A Gentlemen’s Agreement Can’t Stop Bad Actors
Websites rely on a file called robots.txt to opt out of scraping—a digital “Do Not Enter” sign posted at the door that search engines and AI companies are expected to respect. The key phrase here is “expected to.” Born in the early days of the web, robots.txt is an honor system with no police and no penalties. Respecting it is a courtesy; ignoring it carries no immediate consequence.
Nick detailed precise instructions in his robots.txt: specifying which content was open for search indexing and which was off-limits for AI training. The outcome? Cloudflare reported in 2025 that Perplexity used unannounced, stealth crawlers to bypass block directives. Other crawlers spoofed standard browser headers—commenters joked that bots running the “latest macOS” browsed seamlessly, often more smoothly than real users. Nick tested Cloudflare’s non-interactive turnstile verification, only to watch the crawlers bypass it effortlessly.
Figure: AI crawler traffic categorized by purpose. Source: Cloudflare Official Blog
The True Cost of Defense: Host Bills, Collateral Damage, and Cat-and-Mouse Games
When rules fail, defenses get upgraded. Nick’s site normally incurs around $90 per month in hosting fees, but during peak crawl months, his bill surged by nearly 500%. Crawlers devoured server bandwidth and repeatedly crashed his databases, rendering visitor analytics useless: “I wanted to know what real humans were reading so I could decide what to do next, but the analytics were completely overwhelmed by bots.”
He also documented crawler attack patterns. In November 2025, 4,000 “visitors” flooded the site over a few days. Each hit a single page and left immediately—a 99% bounce rate with no referrer traffic—targeting foundation data pages that only 10% of human visitors ever visit. As Nick dryly noted: those 4,000 hits were just a warmup.
In the comments section, small site operators shared increasingly surreal stories. One owner of a Dutch boat rental site described AI crawlers furiously clicking “next month” on calendar widgets thousands of times a day until exhausting the date picker. Another site was hit with hundreds of bot account registrations before launch, forcing the owner to manually delete 100 fake accounts daily. When admins block IP ranges, crawlers instantly pivot to residential IP proxies, returning like couriers with fresh identification. You cannot block them all—it is a cat-and-mouse game, and the mice have far deeper pockets.
Collateral damage poses an even bigger issue. Nick implemented CAPTCHA challenges to verify human users, but real people despise them. VPN users face endless verification loops, while visitors on older browsers get locked out completely. One reader commented: “Whenever I see a ‘Verifying you are human’ prompt, I just close the tab.” Another reader recounted trying to buy a $2,000 home appliance while monitoring prices with a custom script; the merchant’s aggressive anti-bot shield blocked even his manual browser requests—ensuring he would never buy from that store. Over-defense turns away legitimate customers, representing a massive hidden tax on anti-bot measures.
The author experienced this firsthand while researching this article: attempting to access Nick’s original post resulted in a blunt error message—“Sorry, you have been blocked.” A year of anti-bot warfare left the webmaster blocking the very humans trying to read his work. This irony fueled the heated debate on Hacker News.
A Bigger Question: Who Decides Who Accesses the Web?
The top-voted comment on Hacker News put it bluntly: outsourcing the decision of “who gets to view your site” to Cloudflare effectively lets a single corporation define the boundary of the open web. If Cloudflare flags you as a bot, you cease to be a visitor, with zero recourse or appeal.
The opposing perspective is equally firm. Some argue that publicly accessible web content is meant to be read, and authors cannot dictate what tools readers use—just as authors cannot control how readers turn the pages of a printed book. Others counter: if you run a physical shop, are you not allowed to ask shoplifters to leave? Another commenter offered dark humor: even printed books get scanned into training datasets—paper cannot stop AI companies either.
Beyond Scraping: AI Overviews Are Cutting Off Organic Search Traffic
Small websites face a second, quieter loss: search engines are stopping referral traffic. Cloudflare tracked traffic trends across news publishers and found Google referral traffic declined sharply starting in February 2025—down 9% in March compared to January, and down 15% by April. This timeline coincided with Google’s rollout of AI Overviews, which answer user queries directly on search results pages without sending clicks onward. Previously, users searched a query and clicked through to read the source; today, answers appear inline, leaving websites without even an opportunity for a click.
Nick expressed his dilemma candidly: he wants search engines like Google and Bing to crawl his site because they bring new readers, but he does not want everyone else treating his server as an open quarry. The technical challenge is that servers cannot distinguish between crawlers that bring traffic and crawlers that extract data—they are all bots, differing only in intent. Inbound traffic is getting cut off by AI, while hosted content is getting extracted by AI, bleeding small publishers from both ends.
Why This Online War Matters to Everyday Users
This struggle affects online users more than it might seem. The quality of answers you get from AI depends on the material it ingests. If web creators stop publishing because their work is expropriated without compensation or credit, AI models will eventually train on recirculated, degraded content. While major publishers can negotiate licensing deals with AI giants, small site owners receive nothing while funding spiked hosting costs out of pocket. Some platforms have filed lawsuits against AI companies, while others are moving content behind login paywalls.
Companies like Cloudflare are proposing pay-per-crawl mechanisms—requiring AI bots to pay whenever they fetch content. Whether this model succeeds remains uncertain, but it forces a fundamental question into the open: should content in the AI era remain free for machine extraction?
There is no simple side to take. From a webmaster’s perspective, watching a site get scraped for a year while hosting costs quintuple is infuriating. From a user’s perspective, locking the web behind CAPTCHAs and login walls ruins the open internet. The only certainty is that the foundational assumption—that websites exist for human readers—is being rewritten. The next time you look up a piece of trivia via AI, consider where that answer came from, and whether the person who wrote it received anything at all.
Reference Links:
- PatronView: 99% of My Website Traffic Is Bots
- HN Discussion (item?id=49211386)
- Cloudflare: The Crawl-to-Click Gap (Data Report on AI Crawlers & Referral Traffic)
- Cloudflare Blog: From Googlebot to GPTBot (AI Crawler Market Share Shift)
