409,000 Clicks Prove Humans Are the Weakest Firewall

Would you approve npm run analyze? Of course you would. It's a script name you've typed a thousand times. That's exactly the problem.

ScaleX, the outfit that previously found users rubber-stamp 93% of permission prompts without thinking, built a browser game to test whether humans can actually spot malicious AI-agent commands hiding in plain sight. The results, drawn from 40,000+ game sessions and roughly 409,000 individual approve/deny decisions, are not flattering. Players got it right 66.3% of the time. Flip that number around and you get the headline: people missed one in three threats.

The worst offender wasn't some obviously sketchy shell command. It was npm run analyze — approved 64.7% of the time — a string so mundane it barely registers as a decision point at all. Compare that to cat ~/.ssh/id_rsa, which 82% of players correctly blocked. Turns out humans are pretty good at spotting the word "ssh" next to "id_rsa." We're much worse at questioning a build script that looks like every other build script we've ever run.

That's the real finding buried under the clickbait fraction. This isn't really a story about AI agents being sneaky. It's a story about vigilance decay. The longer players sat there clicking "approve," the sloppier they got — and, in a nice bit of irony, they also started over-blocking harmless stuff like rm -rf dist/, a completely standard cleanup command. So you get the worst of both worlds: real threats sail through while legitimate work gets needlessly flagged. Security theater and productivity loss, served in the same interface.

What this means for anyone shipping agentic coding tools:

1. Stop treating human approval as your primary safety net. It's a speed bump, not a wall.

2. Sandbox everything. Least-privilege credentials aren't optional anymore.

3. Make high-risk prompts rare and visually distinct — if every prompt looks the same, humans stop reading them.

4. Log and audit every tool call, because the approval itself is not the control — the paper trail is.

5. Batch the boring stuff. Reserve human attention for decisions that actually deserve it.

Hot Take

Every vendor currently marketing "human-in-the-loop" as a security feature owes their customers an asterisk. Human-in-the-loop isn't security — it's a liability shield dressed up as a feature. This data, game or not, confirms what anyone who's watched a developer alt-tab through fifteen Slack notifications already knew: humans are terrible at sustained pattern-matching against disguised threats, especially when the disguise is boring. The npm run analyze result should be printed out and taped above every product manager's desk who thinks a confirmation dialog counts as a threat model.

Yes, it's a browser game, not a peer-reviewed lab study measuring trained SOC analysts under enterprise conditions — that caveat matters, and ScaleX and outlets like The Register are right to flag it. Game participants know they're being tested, which can cut both ways. But even generously discounting the numbers, a 33% miss rate on adversarial commands should terrify anyone who's told a client "don't worry, a human approves every agent action." That sentence is now a punchline, not a security posture.