A new DeepTempo benchmark is the latest in a string of independent tests to reach the same conclusion: frontier LLMs deployed as security-operations triage agents catch most real threats, but only by flagging a large share of legitimate activity along with them. That is a tradeoff that keeps showing up regardless of who runs the test or which models are involved.
Over-flagging at these rates can easily result in alert fatigue. Analysts working through a queue that’s mostly noise lean toward moving fast versus carefully resulting in a real red flag getting the same rushed glance as everything else.
The team at DeepTempo say that general-purpose models can catch most real threats, but produce a false-positive rate that ranges from 36 percent to as high as 86 percent depending on the test conditions. The benchmark, called SOCBench, tested GPT-5.4, Claude Opus 4.7, and Gemini 2.5 Pro as SOC agents sorting malicious network flows from benign ones — not a test of commercial traffic-filtering products, but of whether general-purpose LLMs, given tools and a security-analyst persona, can do a specialized detection job.
“Almost none of them publish how they evaluated the system, or how their AI does on the telemetry a defender actually sees,” wrote DeepTempo researcher Mayank Kumar. “AI models and systems are black boxes, and the only way of finding their true capabilities is to run domain-specific evaluations on them.”
It should be noted that DeepTempo offers its own AI screening model, LogLM, built for this exact task. So the results that DeepTempo is grading its competitors against is itself. The company has made the benchmark and its underlying test data available for its report.
The same tradeoff has turned up elsewhere. Security vendor Simbian ran its own SOC benchmark testing models from Anthropic, OpenAI, Google, and DeepSeek in mid-2025 and found that top-performing models completed between 61 and 67 percent of full investigation tasks. That’s respectable on paper, but Simbian’s own framing was that raw completion rates don’t capture how those investigations actually resolve.
Florian Roth, CTO of Nextron Systems, built a separate benchmark this spring specifically to isolate the false-positive and false-negative tradeoff in LLM-based finding triage, after concluding that gene,ral-purpose benchmarks miss it. The benchmark tests triage of findings from THOR, Nextron’s own compromise-assessment scanner — a disclosure worth noting given the same DeepTempo-grading-itself dynamic applies here. Roth’s results distinguish between over-escalated benign findings, and the more dangerous failure mode, real incidents misclassified as benign.
A separate academic benchmark, SIABENCH, tested Claude-4.5-Sonnet and GPT-5 on alert triage across two established intrusion-detection datasets and found both models performing strongly but inconsistently. Accuracy for the same model swung from the high 90s on one dataset to the high 80s on the other, underscoring how sensitive these results are to the specific traffic being evaluated.
Taken together, the pattern across DeepTempo, Simbian, Roth’s benchmark, and SIABENCH is consistent: the accuracy figure that gets quoted looks strong, but it obscures how much of that performance comes from over-flagging rather than precise detection.
The findings also cut against a common assumption in the current AI-adoption push that using a frontier model in a specialized, high-stakes task is close to plug-and-play. These benchmarks suggest the opposite. General capability doesn’t transfer cleanly to domain-specific judgment, and the gap shows up in the exact metric that determines whether a tool gets trusted or ignored.
Part of what’s driving those false-positive rates may be that the underlying traffic itself has gotten harder to read, independent of which model is doing the reading.
Encryption now covers roughly 85 percent of web traffic, up from 55 percent in 2017, according to FortiGuard Labs. That explains why metadata-only detection has become the default. There’s simply less payload left to inspect. Automation has grown just as fast. The 2026 Thales Bad Bot Report found bots made up 53 percent of global web traffic in 2025, with bad-bot traffic alone climbing to 40 percent, a seventh straight year of growth.
That growth means legitimate and malicious bots are converging behaviorally. HUMAN Security’s 2026 State of AI Traffic report found only half a percentage point separates benign automation from malicious automation across its platform. These models aren’t just sorting malicious from benign, they’re doing it on a network that’s increasingly non-human, where the honest and dishonest versions of that traffic look nearly the same.
“If not done right, the analyst has to triage all ten flows to figure out which of these are bad, on real network where the unit might be a thousand flows instead of ten, this is the entire workload,” DeepTempo’s Kumar said.
