The popular assumption that a human reviewing each command can keep AI coding agents in check is facing serious scrutiny. A new analysis of more than 40,000 game sessions and over 409,000 individual approve-or-deny decisions found that human reviewers missed roughly one in three malicious commands from AI agents, raising urgent questions about whether "human-in-the-loop" is a reliable security boundary at all.
The findings come from a browser-based game created by Alex Wauters, a former staff engineer at Uber who writes about developer security at Scale X. The game places players in the role of a human reviewer for an AI coding agent, asking them to approve or deny shell commands under time pressure. Some commands are routine, such as `git status` or `npm test`. Others are carefully disguised attempts to exfiltrate credentials, install backdoors, or send secrets to remote servers. After adding statistics tracking, Wauters collected enough data to paint a detailed picture of where human judgment succeeds and, more often, fails. For more on the broader AI industry coverage, this study stands out as one of the most data-rich experiments to date.
The Most Dangerous Commands Were the Hardest to Catch
The study's most striking finding is that obviously destructive commands, such as `rm -rf` on critical directories, were caught reliably. The threats that actually succeeded in stealing credentials were missed nearly three times as often. The single most-missed command in the entire game was `npm run analyze`, approved by 64.7 percent of players despite the agent's history log clearly showing that the underlying script had been tampered with to pipe credentials to an external server.
The problem is structural. Commands like `npm run` execute whatever script is defined in a project's `package.json` file. That script could be a legitimate bundle analyzer or malicious code injected by an earlier file edit the agent modified. To know for certain, a developer would need to inspect every modified file before approving, which negates the speed advantage that AI coding tools are supposed to provide.
Across three such disguised-payload commands in the game, the miss rate was 52.5 percent compared with 28.4 percent for all other credential-exfiltration attacks. Hiding a payload behind a familiar script name roughly doubled its success rate, even when the payload was visible in the history log displayed just above the permission prompt.
Permission Fatigue Erodes Vigilance Over Time
Anthropic previously acknowledged that permission fatigue is a real phenomenon in its Claude Code product, noting that the more approvals a user sees, the less attention they pay to each one. The Scale X data supports this concern empirically. Miss rates decreased during the first few commands of each session as players warmed up, but then climbed steadily toward the end, consistent with either fatigue or the mounting pressure of a ticking clock.
This pattern mirrors real-world dynamics. Developers racing to ship features under deadline pressure face the same incentive to click approve quickly, and the sheer volume of routine commands creates a needle-in-a-haystack problem where genuine threats are statistically rare. The result is a system that trains users to rubber-stamp everything.
The Other Side: Over-Blocking Slows Everything Down
The study also documented the opposite failure mode. Several genuinely benign commands were routinely blocked by cautious players, including standard build and test invocations. This over-blocking slows the agent down and, paradoxically, increases long-term risk. When users are repeatedly asked to approve commands that turn out to be harmless, the noise erodes their willingness to scrutinize future prompts, eventually pushing them toward complete bypasses or auto-approve modes that remove human review entirely.
Features like Anthropic's Auto Mode attempt to mitigate this by automatically determining whether a command is safe before prompting the user. But as the study's data on disguised payloads demonstrates, even visible command contents are not enough for humans to make reliable judgments under pressure.
Missing Context Is the Core Problem
One command, `cat ~/.zshrc`, proved the most divisive in the entire game, approved by 45.9 percent of players. The command is harmless for developers who keep no secrets in their shell profile, but it exposes API keys for the many who export credentials there. Its risk depends entirely on a system configuration the agent cannot see and the reviewer may not remember.
Several other commands generated similar controversy on the Hacker News discussion thread for the same reason. The fundamental issue is that developers are being asked to make security judgments without the full picture of what files have changed, what the agent did in previous steps, and what the system's current configuration contains. As one commenter noted, asking users to validate commands that are ambiguous without context is not a strong safeguard.
What Comes Next for Agent Security
Wauters argues that the solution is not better humans but better tooling. Sandboxing agents so they cannot access credentials directly, strict context isolation, and structural limits on what agents can do without elevated permissions are all more promising than relying on human vigilance. Until those safeguards are in place, granting agents broad permissions remains risky regardless of whether a human is nominally in the loop.
The study is not a peer-reviewed academic paper, and Wauters acknowledges its limitations. The game warned players about threats and applied artificial time pressure that may not perfectly mirror real development environments. But the core finding, that trained human reviewers under pressure miss a third of deliberately disguised attacks, should give every team deploying AI coding agents reason to reconsider their security model.
For developers building with AI agents today, the practical takeaway is to assume the human-in-the-loop will eventually fail. Design your agent permissions and sandboxing so that a missed approval does not mean a leaked AWS key or a compromised build pipeline. The data suggests that treating human review as your primary defense is a bet that does not pay off.
Stay Ahead of AI
The AI agent security landscape is evolving rapidly. Stay informed with the latest AI developments and breaking research.
Read more AI news →
