Around 6 million requests are flooding the official Linux kernel source code repository (git.kernel.org) every day. Under the most optimistic statistical models, genuine human requests account for only 2% of these visits. The remaining 98% are AI crawlers trying to devour every line of code to train large language models.
The Tireless Consumers of Computing Power
This official website cluster, consisting of 5 geographically distributed nodes and 90 CPU cores, is enduring an indescribable asymmetric load. Between 14 and 16 cores are perpetually operating at full capacity, and their sole task is to render code commit pages for crawlers. This directly consumes 20% of the entire system’s computing capacity, and crawlers always launch attacks in waves, making the actual instantaneous peaks even sharper.
The greed of the crawlers exceeds conventional engineering understanding. The Linux kernel has approximately 1.48 million code commit records, and there are about 922 developer forks on the website, intertwining to create billions of valid accessible links. AI crawlers mechanically traverse every single link, scraping the exact same 922 copies of duplicate content one by one. This indicates that current training data acquisition has abandoned engineering optimizations like deduplication, evolving into a computing power crush that uses the most brute-force exhaustion to overwhelm target servers.
A Paper Defense of Computing Power
To block this unrestrained brute-force scraping, the Linux kernel website deployed an anti-crawler system named Anubis. This is a cryptographic math problem based on Proof of Work (PoW): it requires the visitor’s device to combine a given string with its own IP address and the server’s dynamic key, and then continuously calculate its SHA-256 hash value until the first N bits of the result are all zeros.
Currently, Anubis has successfully blocked 66% of low-level scanning programs, but 33% of high-level bots have solved the math problem and successfully entered the main site.
Image: git.kernel.org request funnel: 66% blocked, 33% passed, only 2% legitimate. Source: people.kernel.org
Faced with the relentless 33% of high-level crawlers, the defenders’ only countermeasure is to increase the hash calculation difficulty from level 4 to level 5. However, the core mechanism of this defense has a fatal weakness—it requires consuming machine computing power using JavaScript scripts in the browser. Using a high-level interpreted language to execute intensive hash calculations is inherently an extremely inefficient and highly asymmetric method of confrontation.
A Dimensional Strike by Hardware Instructions
How much of an obstacle does this math problem actually pose to real AI crawlers? In the technical discussion section of Hacker News (HN), a developer named semiquaver provided a devastating technical proof.
He spent merely 10 minutes writing a customized Safari browser extension. By calling the SHA256H* native hardware acceleration instructions unique to the ARM architecture and rewriting the core problem-solving logic using the C language, he squeezed the cracking speed of a level 6 difficulty math problem from the 180 seconds originally required on an iPhone down to the millisecond level. This demonstrates that pure software-level defense mechanisms are vulnerable against dedicated hardware instruction sets. When attackers can bypass inefficient browser sandboxes and directly call underlying hardware computing power, the defenders’ carefully designed delay mechanism vanishes instantly.
Image: The painful curve of Anubis difficulty stepping up and bots following suit. Source: people.kernel.org
Systemic Loss of Control in the Arms Race
Meanwhile, the tactics of the attackers are also undergoing a qualitative change. From initially reporting their true identities honestly in request headers (UA), they have evolved to fully disguising themselves as normal consumer-grade browsers. Today, this has further evolved into advanced tactics that hijack millions of home broadband and mobile network IPs for rotation. Through various stealthy proxy SDK monetization tools, many ordinary users’ smart TVs have unknowingly become botnet nodes helping others scrape data.
Some developers countered in discussions that as long as extreme difficulty challenges are not used, a normal device’s JavaScript engine can solve the problem in 8 seconds. They argue that the vast gulf from 180 seconds to milliseconds is simply caused by the three orders of magnitude execution efficiency difference between JavaScript engines and native C language.
But this is precisely the defenders’ predicament. Crawlers can recklessly employ native languages, server clusters, and even underlying hardware, while real users are trapped in the slow execution environment of a browser. On what basis do defenders assume the long-term advantage lies with them?
Anti-crawler designs that force machines to pay a computational cost have completely failed in the face of dedicated hardware and native code. The speed at which machines execute computational tasks will always be several orders of magnitude faster than the normal terminals used by humans. Linux kernel maintainer Konstantin Ryabitsev ultimately admitted helplessly that there is currently no simple technical solution, and they can only reduce scrabable links by disabling some query features. The collapse of the Anubis anti-crawler scheme means that: in the computing power arms race between AI crawlers and website defenses, the defenders are facing a systemic loss of control. When computing power thresholds suffer a dimensional strike from hardware, the only thing complex CAPTCHAs can block are the real humans trying to quietly read code. The author believes that in the face of this escalating arms race, any defensive strategy could be dismantled by a breakthrough in computing power the very next second, and we must remain humble in our judgments of future trends.
Reference Links:
- Original report from people.kernel.org
- Technical discussion from HN community
