Anthropic has touted its Project Glasswing initiative as evidence that AI can discover dangerous software vulnerabilities faster than most human researchers. Seven months later, security researchers are debating whether those discoveries are delivering the promised protection or simply creating more work for developers.
The dispute sharpened Wednesday when independent researcher Gautam Parab challenged a Monday assessment by VulnCheck criticizing Glasswing’s results. Parab’s review of 45 AI-generated reports delivered to the widely used curl software project found that 37 were rejected as meaningful security vulnerabilities, yet at least 15 still produced code fixes.
The disagreement raises a larger question about AI-powered vulnerability hunting: Should success be measured by security threats uncovered, software bugs fixed or the human effort required to sort through the findings?
Glasswing’s Promise Meets a Reality Check
Anthropic launched Glasswing in April, saying its Claude Mythos model could uncover thousands of serious vulnerabilities and give defenders a head start against attackers.
In October, the company expanded that effort with a free, opt-in OSS Scanner that delivers AI-generated vulnerability reports directly to open-source maintainers, without human review. Anthropic acknowledged that faster reporting could produce inaccurate findings.
That trade-off drew scrutiny Monday from VulnCheck researcher Patrick Garrity, who examined the first 34 published reports from a batch of 45 sent to curl, software embedded in countless applications and internet services.
Garrity found that curl maintainers had rejected 19 as security vulnerabilities, confirmed six as low severity and left nine awaiting assessment. He characterized the results as a 55.9% false-positive rate and questioned why Glasswing’s growing number of discoveries wasn’t producing more visible security fixes.
Excessive false positives consume developers’ limited time investigating harmless findings, potentially delaying fixes for vulnerabilities that pose genuine threats.
Rejected Vulnerabilities, Real Bugs
But independent researcher Gautam Parab, revisiting the underlying records Wednesday, found that the distinction between a false security alarm and a legitimate software bug was getting lost.
Of the 45 reports, curl had classified 37 as posing no meaningful security risk. Yet at least 15 had already prompted code fixes.
One involved a use-after-free bug in cookie handling. Another concerned credentials and HTTP/2 server push. Both received fixes despite failing curl’s threshold for a security vulnerability.
Curl founder Daniel Stenberg’s experience supports that distinction. In an Oct. 22 blog post, he described AI-generated reports as increasingly accurate and useful, while warning that their growing volume was creating new challenges for maintainers.
“The AIs are also still bad at figuring out exactly where the line is between a bug and a vulnerability,” Stenberg wrote.
His assessment, which predates the researchers’ November disagreement, supports Parab’s argument that rejected security findings aren’t necessarily worthless. But Stenberg also acknowledges the workload imposed by increasingly numerous and detailed reports, a central concern raised by VulnCheck.
A Dispute Over the Numbers
Parab also challenged Garrity’s accounting of Glasswing’s remediation progress.
VulnCheck reported that Anthropic’s public disclosure ledger showed just 15 additional fixes between late August and November. But Parab found that Anthropic’s broader upstream patch count increased by 208, from 421 to 629, over the same period.
The discrepancy matters because the public ledger reveals fixes only after their disclosure windows close. It does not reflect every patch already made.
Similarly, Anthropic’s reported 94.2% true-positive rate applies to findings reviewed by outside security firms, not the unreviewed scanner reports delivered directly to projects such as curl.
None of this settles whether Glasswing delivers enough security benefit to justify the burden on maintainers. But the new analysis suggests that treating every rejected security warning as a failed AI discovery misses another measure of value: legitimate software bugs that developers actually fix.
AI can find bugs. Whether it can reliably distinguish the dangerous ones — without overwhelming the people tasked with fixing them — remains a different question.
