Claude Code's 1,000 False Positives Found One Real 23-Year Linux Bug
Everyone keeps telling us AI will revolutionize security auditing. Finally, they say, we can automatically scan codebases and find the bugs humans missed. The reality? Nicholas Carlini from Anthropic just spent three months wading through 1,000 false positives to find a handful of real vulnerabilities.
Sure, one of those bugs had been hiding in Linux's NFS driver for 23 years. That's genuinely impressive. But let's do the math here.
<> "Claude Code's results were astonishing - it found non-obvious bugs with little guidance" - Nicholas Carlini/>
Astonishing, maybe. Efficient? That's a different story.
The Signal Buried in Noise
Carlini's approach was elegantly simple: a bash script that fed every Linux kernel source file to Claude Code with CTF-style prompts asking it to "find a vulnerability." The AI dutifully complied, generating reports on suspected flaws across the entire codebase.
The NFS vulnerability it discovered enables remote attackers to read sensitive kernel memory over the network. This isn't some surface-level pattern matching - it requires deep protocol understanding. Credit where it's due: that's legitimately sophisticated analysis.
But here's what the headlines won't tell you:
- 1,000+ false positives requiring manual review
- Three months of developer time for triage
- A process that's barely more efficient than traditional auditing
The Elephant in the Room
Nobody wants to talk about the economics here. If it takes three months to separate real bugs from AI hallucinations, what exactly are we automating?
The cybersecurity industry loves to point out there are 723,000+ open-source projects with crypto flaws alone. At Carlini's pace, we'd need armies of triagers just to keep up with the false positive deluge.
And this isn't Claude's first rodeo. Earlier in 2026, Claude Opus 4.6 found 22 vulnerabilities in Firefox and generated exploits for CVE-2026-2796. Impressive technical work, but again - how much human oversight did that require?
When AI Security Tools Break Themselves
Here's the kicker: Claude Code itself has a high-severity vulnerability. After 50+ subcommands, it silently disables its own deny rules - the very protections meant to prevent malicious behavior. The bug allows SSH key theft via cloned repos with weaponized CLAUDE.md files.
So we're using buggy AI tools to find bugs in other software. The irony is thick enough to cut with a knife.
The Reality Check We Need
Don't get me wrong - finding a 23-year-old Linux kernel vulnerability is genuinely impressive. The technical capability is real. But the efficiency promises? Those are still marketing fiction.
Anthropic will undoubtedly tout this as validation of Claude Code's enterprise value. They'll probably win some contracts from companies desperate to audit their sprawling codebases. But until someone solves the false positive problem, we're just shifting the bottleneck from finding bugs to triaging AI output.
The future of AI security auditing isn't about replacing human experts. It's about finding the sweet spot where AI pattern recognition amplifies human insight without drowning analysts in algorithmic noise.
We're not there yet. The 1,000 false positives prove it.
