Illustration of small attacking figures overwhelming a padlock shield in front of server racks, representing the OpenSSL HollowByte vulnerability

OpenSSL Patches 11-byte DoS Flaw it Decided Didn’t Need a CVE

Researchers have uncovered a denial-of-service flaw that might be the most dangerous Eleven since Millie Bobby Brown.

The team at Okta said that a flaw dubbed HollowByte could allow an attacker to take down an OpenSSL server with just 11 bytes worth of text. This would potentially trivialize the process of conducting denial of service (DoS) attacks.

“Every so often, a vulnerability reminds us how deeply our digital infrastructure relies on foundational libraries,” Okta’s Red Team writes in its summary.

“By sending a malicious payload of just 11 bytes, a remote, unauthenticated attacker can force a server to allocate disproportionate chunks of memory before any security handshake even begins.”

The flaw, the researchers say, lies in how OpenSSL handles the header that opens every TLS handshake. That header carries a 4-byte field declaring how large the incoming message will be — and older OpenSSL versions take that number on faith, allocating a receive buffer to match before a single byte of real data has arrived.

In a HollowByte attack, the threat actor sends a legitimate-looking handshake header but lies about the size in that 4-byte field. Because OpenSSL pre-allocates based on the claim rather than the actual payload, an 11-byte packet can trigger an allocation of up to 131 KB. The connection then just sits there, waiting on data that’s never coming.

In practice, the attack would look like this.

“Read Header⟶grow_init_buf()⟶OPENSSL_clear_realloc()⟶malloc(attacker_size)”

The Okta team said that the issue boils down to the way memory is allocated for OpenSSL connections. Ideally, terminated connections would result in their associated memory contents being cleared, freeing up space.

Due to the bug, however, those memory contents remain populated. This would allow an attacker to send multiple requests of headers packed with garbage data, overload the target server, and cause a crash.

“Holding connections open to exhaust threads is a classic trick (like Slowloris). HollowByte introduces a nastier compounding effect due to how the GNU C Library (glibc) handles memory,” the researchers explained.

“When an attacking connection drops, OpenSSL frees the buffer. However, glibc does not immediately return small-to-medium allocations to the operating system; it keeps them for potential reuse.”

That’s what makes repeat attacks so effective. Waves of connections with randomized claimed sizes fragment the heap faster than glibc can reuse it, so a server’s memory footprint climbs and stays climbed. In Okta’s own testing, an unpatched server running on 1 GB of RAM was OOM-killed once fragmented memory hit 547 MB; on a 16 GB box, the attack locked up a quarter of total system memory while staying under standard connection-limit thresholds — meaning the usual defenses against connection-exhaustion attacks don’t help here.

While DoS vulnerabilities are not usually considered serious flaws, the issue could pose a major threat to enterprises. OpenSSL sits underneath a wide swath of the internet’s plumbing — web servers like Apache and NGINX, language runtimes including Node.js, Python, Ruby, and PHP, and databases such as MySQL and PostgreSQL all inherit the exposure.

OpenSSL’s own pull requests confirm the account. The issue was reported directly to the project’s security team by Okta’s Red Team, and the security team opted to handle it as a “bug or hardening” fix rather than a tracked vulnerability. Notably, the fix only covers TLS — developers flagged DTLS as vulnerable to the same pattern but left it unaddressed, calling a fix “much more complicated and invasive” for now.

OpenSSL resolved the issue by moving to incremental buffer growth, merged via three pull requests – 30792, 30793, and 30794 – and folded into the v4.0.1 release, with backports to the 3.6.3, 3.5.7, 3.4.6, and 3.0.21 branches. OpenSSL classified this as a hardening fix rather than a CVE-tracked security advisory. Administrators watching only for CVE alerts may want to check their changelog directly. Okta recommends updating to the latest supported release regardless of the CVE question.

Notably, HollowByte wasn’t alone in the release cycle.

The same June 9 patch batch also fixed CVE-2026-34183, a separate memory-exhaustion DoS in OpenSSL’s QUIC stack triggered by flooding a server with PATH_CHALLENGE frames — and that one did get a CVE, rated Moderate severity.

The split raises an obvious question for anyone triaging this release by CVE list alone: two memory-exhaustion bugs, patched the same day, and only one of them will show up in a scan.

Jan Schaumann, chief information security architect at Akamai, raised the same concern on the oss-security mailing list on Saturday. He wrote teams that track only OpenSSL’s security-flagged commits, a common practice for keeping local forks current, would have had no signal that this change mattered.

Author

  • Shaun Nichols

    Shaun Nichols, Senior Editor at Security Point Break, is a veteran cybersecurity journalist who has spent nearly two decades covering the enterprise tech market—from the era of hard-drive iPods to the rise of Agentic AI. Formerly of The Register and TechTarget, Shaun is known for his sharp wit and deep technical dives into malware, ransomware, and the intersection of government policy and security. Follow him on X: @shaundnichols

Total
0
Shares

Leave a Reply

Previous Article
Four MCP servers rated differently by security scanners — one marked safe, one flagged risky, one marked critical, and one unmarked despite a hidden threat icon

The Tools Meant to Secure AI's New Plumbing Are Wrong More Than Half the Time, Study Claims

Related Posts

Discover more from Security Point Break

Subscribe now to keep reading and get access to the full archive.

Continue reading