Across 40,000 game sessions and 409,000 decision prompts asking whether to “Allow or Deny”, human reviewers missed one-third of the hidden malicious commands. That is the real-world data released last month by ScaleX from a browser mini-game. Right at the opening, the game explicitly warned players: there are malicious commands disguised in the stream, so pay close attention. The warning proved largely ineffective.
AI assistants are rapidly integrating into daily life—writing code, organizing files, booking tickets, and responding to messages. At every step of execution, a dialog might pop up: “Allow execution?” Click “Allow”, and it proceeds; click “Deny”, and it tries an alternative path. In the industry, this mechanism is known as Human-in-the-Loop (HITL) approval—positioning a human as the final gatekeeper. It is akin to handing an intern an access badge: every time he moves, he turns back to ask “Can I do this?”, while you nod absently amidst your busy workload.
The experiment gamified this approval workflow. Players stepped into the role of reviewers, tasked with quickly classifying each command under countdown pressure as either legitimate or malicious. Incorrect decisions led to score deductions. The dataset contained blatant threats like database deletion alongside covert adversaries secretly sending data to unfamiliar remote servers.
A 40,000-Game “Gatekeeper Exam”
The report card was far from impressive. On average, players missed 1/3 of malicious commands, yielding an overall accuracy of just 66.3%. Fully 32.9% of game sessions ended with a negative score—penalties from letting threats slip through and blocking safe commands outweighed points earned from correct decisions. While 35.2% of players caught all malicious commands, only 20.8% achieved this through normal auditing; the rest relied on the brute-force strategy of “blocking everything”. Furthermore, 7% of players approved every single prompt without blocking anything.
Figure: Overall statistics across 40,000 games. Source: scalex.dev
An engineering reality check: this test evaluated individuals who knew they were taking an exam, were explicitly warned, and were penalized for every mistake. In real-world work environments, there are no upfront warnings and no point deductions. Under actual working conditions, performance is bound to be even worse.
The Easiest to Approve Are Exactly What Should Be Blocked
Breaking down malicious commands by category reveals substantial variance in omission rates:
- Destructive attacks (database deletion, system formatting): 11.7% missed
- Persistence & backdoors (system backdoors, startup items modification): 23.8% missed
- Data exfiltration (silently sending data to unfamiliar servers): 33.4% missed
- Sensitive file access (reading passwords, secret keys): 35.0% missed
The pattern is unmistakable: users excel at stopping blatant vandalism, yet routinely approve data theft. The missed rate for data exfiltration commands was triple that of database deletion. What attackers covet most coincides precisely with what humans fail to guard against.
”Run Analytics”: Innocent Name, Swapped Payload
The single worst performer was a command labeled “Run Analytics”, which 64.7% of players approved—making it the most frequently missed among 37 threat types. While its name sounded completely benign, the historical execution log displayed right above the popup in black and white showed the true payload: packaging local data and transmitting it to an unknown remote server. Two-thirds of players still clicked “Allow”.
The underlying mechanism warrants a closer look. Commands like this function like desktop shortcuts: titled “Run Analytics”, but the actual execution script resides inside project files. Crucially, AI assistants do not need user authorization to edit project files. The agent first places the payload inside the file, then turns around to ask: “May I execute this?” As players noted in discussion forums: file edits bypass approval, while script execution requires approval—and attackers exploit precisely this gap.
Together, three similar stealth commands registered a 52.5% missed rate, compared to 28.4% for other data exfiltration attacks. Hiding malicious payloads behind familiar names doubled the attack’s success rate. Camouflage proved vastly more effective than overt destruction: reviewers were not failing to look; they simply skimmed over familiar labels.
The Further in the Game, the Looser the Guard
Data further revealed that missed rates climbed noticeably toward the end of each session. During early commands, players were alert and warming up; as the game progressed, attention steadily degraded. Under timer pressure, players began relying on instinct and reflexively approving prompts to maximize their scores. The author cited a statement from Anthropic: the more approvals a user encounters, the less attention they allocate to each one.
Figure: Missed threat rate increases as the game progresses. Source: scalex.dev
Friction wears down both sides. Three entirely safe commands—configuring internal package repositories, clearing build caches, and restarting stalled local services—were blocked by 59%, 45%, and 43% of players respectively. Excessive false positives induce fatigue; once numb, users fail to intercept actual threats. This represents a modern technical variant of “crying wolf”: what this dynamic genuinely depletes is human cognitive endurance.
The Experiment Sparked Fierce Debate
The methodology sparked intense debate on Hacker News. Critics argued that certain prompts were inherently ambiguous—for instance, reading a personal configuration file might be harmless for some workflows but a privacy leak for others; timed pressure failed to reflect reality where no timer forces immediate choices; and most fundamentally, players knew they were being tested, whereas real-world users do not maintain an adversary-auditing mindset toward their own AI assistants.
The author—a former senior Uber engineer known on HN as Wirbelwind—responded point by point. He conceded that two prompts contained genuine ambiguity and promptly corrected them. However, post-fix data showed no reduction in omission rates; comparing game datasets across different time windows, overall missed rates remained consistent, with non-HN peak traffic periods showing even worse accuracy.
Clarifying the prompts did not lower the error rate. The vulnerability lies not in the questions, but in human cognitive limitations.
Fairly speaking, defenders also raised valid points: 34% of commands in the game were malicious, whereas real-world threat density is dramatically lower. Players in the game operated under heightened vigilance due to threat concentration, making real-world behavior likely even more permissive. Yet both camps agree on one core fact: humans are fundamentally unsuited for continuously auditing streams of execution commands.
Reality Is Worse Than the Game
The top-voted comment on HN came from programmer pllbnk: in real life, users practically never read what their AI assistants are executing. He shared a command his AI assistant was currently running—a script pointing to a temporary directory with the word hidden in its filename. He admitted having no idea what it actually did: “It worked fine every time before, why should this time be different? Approved!” If every command demanded full comprehension, the productivity gains promised by AI would instantly evaporate.
Another commenter put it even more bluntly: approval popups were never designed as a security mechanism; they function as legal disclaimers for vendors—so when incidents occur, corporate lawyers can claim, “You approved it, so the responsibility is yours.” While sharp, when paired with a 33% failure rate, the assessment does not seem off base.
Three Takeaways for Everyday Users
Although participants in this experiment were software developers, these approval dynamics are swiftly migrating to everyday smartphones and desktop environments. Mobile and OS-level AI assistants already navigate apps, read files, dispatch messages, and process payments—triggering approval dialogs before major actions. As AI capabilities expand, human attention budgets remain strictly finite. The higher the popup frequency, the more casual user approvals become. AI phishing and social engineering leverage the exact same psychological trap: inducing an apparently harmless yet costly confirmation during a momentary lapse in focus.
Practical advice for general users requires no deep technical expertise: refrain from granting master-key permissions to AI agents, granting access strictly on a need-to-use basis; enforce secondary manual verification for critical operations like passwords, verification codes, and payments; and cultivate a healthy skepticism toward the “Allow” button when it appears incessantly.
Ultimately, system architecture must evolve. The author advocates for robust sandboxing: isolating AI agents within confined environments so that even if compromised, they cannot reach sensitive user files; alongside segregating credential storage so agents cannot read sensitive keys on demand. Shifting security boundaries from “human attention” to “structural isolation” is essential—data has convincingly demonstrated that human attention alone cannot hold the line.
40,000 game sessions proved one fundamental truth: approval dialogs outsource security responsibility to humanity’s scarcest cognitive resource—sustained attention—which inevitably degrades under time pressure, repetition, and familiarity. The next time an AI assistant asks for approval, pausing for just one extra second might be what keeps your system out of the missed one-third statistic.
Reference Links:
- ScaleX: Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays
- HN Discussion (item?id=49195468)
- Developers Digest: Approval Fatigue Is an Agent Security Bug
- TechMedia: From Deleting Databases to Deleting Emails: The “Fatal Flaw” in AI Assistants
