The security scanners that companies rely on to vet a fast-growing category of AI infrastructure are getting it wrong most of the time, according to a Fudan University study this week.
The findings underscore the challenges cybersecurity defenders face when trying to secure one of the most important pieces of the fast-growing internet plumbing connecting AI assistants to real emails, files, databases and company systems.
Fudan researchers focused on the Model Context Protocol ecosystem and the security scanners practitioners rely on to determine cybersecurity risks, flaws and bugs.
An unpatched flaw or misconfiguration in an MCP server can hand an AI agent’s trusted access straight to an attacker, giving them the ability to trigger a quiet data leak, a hijacked database query, or a command executed on a company’s behalf that no one authorized. Scanners exist to catch that kind of exposure before it ships.
The Fudan team’s finding is that, right now, those scanners can’t be trusted to do it.
If you’ve used an AI assistant like ChatGPT, Claude or Gemini for anything beyond writing text, you’ve likely relied on MCP without knowing its name.
On its own, an AI model can only do one thing: predict the next chunk of text. It can’t check a calendar, read a file, query a database or send a message. To act in the world, it needs a bridge to outside tools and data.
The Model Context Protocol, released by Anthropic in late 2024, is that bridge. It’s a shared standard so any AI system that “speaks” MCP can connect to any tool built to the same spec, the way any laptop with a USB-C port can plug into any USB-C charger regardless of who made either one.
An MCP server is the piece on the other end of that connection. It’s a small program that hands the AI a specific capability, like reading files, querying a database or controlling cloud infrastructure. Developers have built and published tens of thousands of them, largely on open, reusable code repositories, which is exactly what let the technology spread so fast, and exactly what makes it hard to police.
The protocol itself doesn’t require an MCP server to prove it’s safe or limit what it can touch. An AI agent generally trusts whatever a connected server tells it. If a server that’s supposed to search a user’s email was written — or quietly modified — to also forward what it reads elsewhere, the AI has no built-in way to notice.
The findings
The researchers tested eight scanners against the 37,288 MCP servers in their dataset that could actually be run and interacted with live, out of more than 64,000 collected overall. The scanners flagged 96.89% of those interactable servers as risky. When the team manually checked a sample of those flags by hand, fewer than half held up as genuine problems. Different scanners, pointed at the same server, frequently disagreed with each other entirely.
The finding, they wrote, “shift[s] the conclusion from MCP servers are unsafe to a more critical insight: current MCP security scanners are not yet reliable enough to support ecosystem-level security claims.”
What’s on the line?
Companies and developers use these scanners right now to decide which MCP servers are safe to plug into their AI systems. If the scanners are wrong most of the time, that vetting process is closer to a coin flip than a safeguard.
Security Point Break reached out to all eight vendors named in the study. Four responded before deadline — including one with a pointed rebuttal about how its tool was tested in the first place. Their full replies are below.
A note on the science here
The study is an arXiv preprint. ArXiv is a repository where researchers post papers before, or sometimes instead of, formal peer review.
It’s important for SPB to emphasize, the study “Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability” has not been peer-reviewed or accepted by a journal or conference yet. That means it hasn’t been formally checked by outside referees the way published academic work eventually is. That’s a real caveat, and it’s why this report treats vendor pushback as seriously as it treats the paper’s own numbers.
The scanners are getting scrutiny here, and so is the study.
However, it’s also not a reason to dismiss the study results.
The authors are computer-science researchers at Fudan University, and their methodology — sample sizes, validation process, two independent human reviewers checking the automated pipeline’s work, a public query interface anyone can use to check servers themselves — is fully disclosed, which is sometimes more transparency than a peer-reviewed paper.
We’re covering it as a serious, citable preprint with real numbers behind it, however not as a settled, peer-reviewed verdict.
Vendor specific results
The paper names eight existing scanners that are publicly available, and grades each one: Snyk’s Agent-Scan, Tencent Zhuque Lab’s AI-Infra-Guard (tested in both static and dynamic modes), Cisco AI Defense’s MCP-Scanner, Ant Group’s MCPScan, the academic tool MCPSafetyScanner, Lasso Security’s mcp-gateway, Nova Hunting’s nova-proximity, and Aira Security’s mcp-armor.
These are the study’s numbers below, not our editorial judgment about any product. Full vendor context follows further down, not as an afterthought.
Reported-risk rates swung wildly. MCPScan flagged 80.04% of the servers it analyzed as risky, the highest of the eight. mcp-gateway flagged 2.60%. MCPSafetyScanner flagged 0.54%, the lowest of the eight.
Precision told a different story. mcp-gateway’s alerts were the most trustworthy when it did flag something — 96.88% precision. MCPSafetyScanner and nova-proximity also posted strong precision, at 74.03% and 52.78%.
But none of those three caught a single one of the ten confirmed, real-world CVEs the researchers used as a ground-truth benchmark.
MCPScan — the noisiest scanner by volume — had the opposite problem in reverse. Middling precision, 45.53%. But 74.29% recall, easily the best of the eight, and the only scanner to catch a majority of the confirmed vulnerabilities.
Cisco’s MCP-Scanner and Snyk’s Agent-Scan landed in the weaker middle on both axes (precision and recall, the two dimensions in the chart below): 24.64% and 28.21% precision, with recall around 21% and 50%.
Chart: precision vs. recall for all eight scanners tested.
Source: Fudan University’s arXiv preprint (Tables 5 & 7) — an unreviewed preprint, not yet peer-reviewed; see “A note on the science here,” above, for context. Bubble size = share of servers each scanner flagged as risky. Vendor responses and disputes are in “What the vendors say,” below. Hover or tap any bubble for exact figures.
The researchers traced why
In one sampled server, “ecommerce-store-mcp,” a source-code-aware scanner caught a real, unsafe input flow reaching the model that metadata-only scanners missed entirely — the tool’s description read as benign, so they never checked what the code actually did. Elsewhere, several scanners flagged “credential leakage” in servers where the only evidence was a schema field literally named “token,” an unused placeholder with no sensitive data behind it.
Scanners also barely agree with each other. Pointed at the same server, the average overlap between any two scanners’ flagged-server lists was just 15.66%. The highest agreement in the whole study, between Ant Group’s MCPScan and Tencent’s A.I.G dynamic mode, was still under half — 47.80%.
Some of the vendors named above dispute aspects of how their tools were categorized in this benchmark.
The second data point in nine days
This is not an isolated finding. It’s the second time this month Security Point Break has covered a study showing that automated, AI-adjacent security tooling can’t reliably separate real threats from noise.
Last week, SPB reported on a set of independent benchmarks showing that large language models used as automated security analysts — GPT-5.4, Claude Opus 4.7 and Gemini 2.5 Pro, tested by AI-security firm DeepTempo’s SOCBench benchmark — misclassified as much as 86% of legitimate network traffic as malicious.
It’s worth noting here that SOCBench is an industry benchmark run by a commercial AI-security vendor, not an academic peer-reviewed study like the Fudan paper. That’s a different kind of source, with a different set of incentives worth keeping in mind.
Separate benchmarks from Simbian and Nextron Systems, cited in that piece, found the same pattern in different forms. Models that catch most real threats, but only by burying analysts in false alarms along the way.
The MCP scanner study, according to study, follows the same pattern, one layer over. Different research team, different target. MCP servers instead of network traffic. Same underlying problem. A detection tool that flags almost everything isn’t being cautious, it’s being useless, because the humans on the other end can no longer tell which alert is the one that matters.
MCP threat landscape
In September 2025, researchers at Koi Security discovered an npm package called postmark-mcp, built to let AI assistants send transactional email through the Postmark platform. The code also quietly BCC’d a copy of every outgoing email to an attacker-controlled address, phan[@]giftshop[.]club.
The package worked exactly as advertised, and cleanly, for its first 15 published versions. Version 1.0.16, uploaded September 17, added a single line of code that did the damage. Postmark itself confirmed it had no involvement — the package was an unauthorized copy of its legitimate open-source code, republished under Postmark’s name to borrow its credibility.
By the time it was pulled from npm roughly a week later, Koi Security estimated it had been downloaded around 1,500 times, with as many as 300 organizations actively routing email traffic — password resets, invoices, internal memos, two-factor codes — straight through the attacker’s inbox.
Security researchers have since called it the first publicly documented case of a malicious MCP server used in a live attack. Koi Security co-founder Idan Dardikman, describing the incident to Dark Reading, put the scale of the damage in three short sentences: “One developer. One line of code. Thousands upon thousands of stolen emails.”
The exposure numbers this complicates
MCP has had a year of scary headlines, and they’ve never quite added up to one story. The Fudan study is a reason to ask which of those numbers were measuring real risk — and which were measuring scanner noise.
Trend Micro found 492 exposed, unauthenticated MCP servers in mid-2025; by this spring, that had “tripled” to 1,467. GBHackers reported 5,832 flagged issues, 2,259 confirmed, out of 9,695 servers scanned. In April, OX Security said a flaw in Anthropic’s MCP SDK put up to 200,000 instances at risk of takeover; Anthropic called it a design choice, not a defect. And attackers are watching the same scoreboard. The Internet Storm Center caught automated scans hunting for exposed MCP servers and stray AI credentials on July 13.
None of that is in dispute. The exposed servers are real. The OX takeover was demonstrated live, and postmark-mcp genuinely stole email.
What the Fudan study complicates is treating a scanner’s raw “risky” percentage like a body count. If scanners average 45.53% precision, some of this year’s greatest hits are probably inflated by noise. And with recall stuck at 24.17%, the opposite is just as likely: real, exploitable servers are slipping through uncounted.
What the vendors say
Security Point Break contacted all eight scanner vendors named in the study ahead of publication, asking whether they’d reviewed the findings, disputed the methodology, or wanted to add context. Our deadline for responses was Monday, July 20 at 11am ET. Four responded.
Ant Group (MCPScan): The company said it was not involved in the study and, to its knowledge, has had no direct contact with the Fudan researchers. Without access to the authors’ underlying data or evaluation settings, Ant Group said it’s not in a position to validate or dispute the paper’s methodology or figures, and referred detailed questions to the research team.
On design intent, spokesperson Vick Li Wei described MCPScan as “a screening and investigation aid, rather than a final vulnerability determination.” Broad coverage and high sensitivity are deliberate choices, the company said, reflecting a conservative posture meant to reduce missed risks — paired with contextual review and ongoing precision improvements. Ant Group also noted that reported risk rates can shift materially depending on dataset, scanner version, rules and thresholds, so direct comparisons require those conditions to line up.
Lasso Security (mcp-gateway): CPO Ophir Dror said Lasso doesn’t dispute the numbers or methodology reported for mcp-gateway, and welcomed the study’s central question: “can today’s MCP security scanners be treated as ground truth at ecosystem scale?” Lasso agrees the answer is no.
Dror’s main point was about categorization, not accuracy. mcp-gateway, he said, is an inline runtime plugin framework that sits between an agent and its MCP servers, scoring server reputation from marketplace and GitHub signals and scanning tool descriptions for prompt injection and other tool-poisoning indicators before a connection is allowed. It does not match server versions against a CVE database, which is precisely what the study’s recall benchmark measures.
“A server can be popular, well maintained, and completely benign in its tooling, and still run an unpatched version,” he said. Reputation and patch status are different signals, and the paper’s own data shows the CVE-recall benchmark favored static source-code scanners built specifically for that job. Dror pointed instead to precision — 96.88%, the highest of the eight — as the more relevant number for a tool operating on live production traffic.
Lasso also flagged one figure for the record. The study’s Table 5 lists MCPSafetyScanner at a 0.54% flag rate, below mcp-gateway’s 2.60%, making mcp-gateway the second-most conservative tool in the set rather than the lowest. mcp-gateway is open source under MIT, at github.com/lasso-security/mcp-gateway.
Tencent Zhuque Lab (A.I.G): The team said it has begun reviewing the preprint. Its central point: A.I.G in dynamic mode is an agentic, multi-stage workflow, not a single-pass model call — it plans an assessment, selects tools, generates and adapts probes, and interprets results before deciding whether evidence supports a finding.
“The underlying model is therefore part of the system being evaluated,” the team said, and can materially shape both precision and recall.
The study ran all LLM-based scanners on a fixed local Qwen3-235B-A22B-Instruct backend for experimental control. That’s useful for comparison, Tencent said, but the resulting numbers describe A.I.G under that specific configuration rather than model-independent performance or necessarily its intended deployment setup, a distinction the team said matters more for the dynamic-mode workflow than for static analysis.
Tencent asked that any reporting distinguish scanner-reported potential risks from confirmed vulnerabilities, and frame results as specific to this preprint’s evaluation configuration.
Nova Hunting (nova-proximity): Founder Thomas Roccia raised the most detailed objection of the four responses — not to the reported numbers, but to how nova-proximity was categorized.
The tool, he said, is an MCP discovery and reconnaissance utility, not a vulnerability scanner. It inventories a server’s tools, prompts and resources, and performs security analysis only if a user supplies their own NOVA detection rules. It ships with no default rule set beyond a handful of examples.
The paper’s “NOVA rules + LLM” label is technically accurate, Roccia said, but the LLM is only an optional evaluator inside a user-defined rule, not a standalone analysis engine — and the paper doesn’t specify which rules, if any, were configured for testing, which he said materially affects results in a rule-driven tool.
On the study’s central benchmark, Roccia didn’t dispute the outcome so much as its applicability. “There is no security scanning that can be meaningfully evaluated” without user-supplied rules, he said. And nova-proximity “was never intended to detect CVEs, implementation vulnerabilities, or perform source-code vulnerability analysis.”
The 0% recall result is drawn from just four applicable cases, he noted, in a category outside the tool’s intended scope.
Roccia said Nova Hunting hasn’t been in contact with the Fudan team, has not disputed the reported figures themselves, and welcomes independent evaluation of the project. He suggested a more representative test would measure how well the tool inventories servers, or how effectively specific supplied rules catch what they’re written to catch.
He also noted the open-source nova-proximity repository — used alongside the NOVA framework — is a proof of concept distinct from SecurityBreak’s commercial products, NovaHunting and Proximity, which use the same underlying NOVA engine for different, production-oriented purposes.
No on-the-record statement by deadline
Snyk (Agent-Scan), Cisco AI Defense (MCP-Scanner), the MCPSafetyScanner team, and Aira Security (mcp-armor) did not provide an on-the-record statement by publication time. This story will be updated with any responses received afterward.
Pushback is expect and not unwarranted
A study putting hard numbers on eight named products threatens the “servers at risk” pitch several of these vendors sell on. The scanners with strong precision — mcp-gateway, MCPSafetyScanner, nova-proximity — have an easy story to tell about their own results, though the study’s recall numbers complicate it. The ones lower on both axes have more context to add — several already have, above.
Security teams that spent the past year restricting MCP adoption on scanner output now have reason to ask how much of that caution was chasing false alarms — a case engineering teams have likely been making informally for months.
None of this makes MCP safe to deploy carelessly. The confirmed vulnerabilities above are real, and recall stuck at 24.17% means current scanners are still missing most of them. But the next MCP policy conversation inside most organizations might want to start with “which scanner, and how do we know it’s right” instead of a flagged percentage taken at face value.
Now what?
Real, independently confirmed MCP vulnerabilities exist this year, and some are serious. The Fudan study doesn’t argue otherwise. Its sharpens the point, echoed almost exactly by the SOC-triage benchmarks nine days earlier, and that is an automated tool’s “risky” verdict is a hypothesis, not a conclusion, and that a scanner can look precise and still miss vulnerabilities.
As AI systems get wired into more of the software the next generation of security professionals will be defending, and as more of the tools defending them are themselves built on AI, checking the checker will become a core part of the job.
