Late at night on September 8, 2026, security research firm Knostic submitted a vulnerability report to Cloudflare. The report disclosed that when invoking long-context models on the Workers AI platform, the service returned input data belonging to other users—without generating any errors. Within 25 minutes of receiving the report, Cloudflare’s engineering team successfully reproduced the cross-tenant crosstalk using two independent accounts. The API gateway returned a standard HTTP 200 status code. While the generated text was syntactically fluent and natural, it visibly contained private prompt data from preceding requests.
The affected workload was isolated to the @cf/zai-org/glm-5.3 model. Launched on Workers AI on August 27, it was the only hosted deployment on the platform impacted by the flaw. GLM-5.3 itself is merely a set of open-weight parameters; the root cause stemmed from container resource provisioning and a critical state machine defect within the underlying SGLang inference engine. In multi-tenant LLM serving, the physical boundary of isolation resides in GPU memory pages. The gateway layer cannot guard this boundary: whenever an inference engine records a request abort on one side of its state machine while the scheduler reads from the other, multi-tenant isolation silently collapses.
Figure: Incident timeline and remediation milestones published by Cloudflare. The company confirmed that the impact was strictly limited to a single long-context model deployment, triggered by a configuration disconnect in a disaggregated architecture. Source: Cloudflare (blog.cloudflare.com).
Memory Pool Mismatch Meets a Broken Scheduler Flag
A severe capacity discrepancy existed in the production rollout. The hosted endpoint advertised support for a 1,048,576-token context window, but the KV cache pool allocated to prefill worker nodes could physically accommodate only 681,280 tokens. The provisioned pool was nearly a third smaller than the externally advertised specification. While massive context windows are routinely championed as primary marketing differentiators, physical VRAM allocation remains strictly constrained by per-host hardware budgets.
KV cache pool sizes are not static configuration parameters. Upon initialization, SGLang dynamically computes its memory pool based on available VRAM. The platform infrastructure failed to enforce validation between this dynamic allocation and the advertised context window. Because the two values were maintained across separate operational workflows, an inverted capacity gap developed unnoticed. The unconstrained static limit laid the groundwork for abnormal out-of-bounds handling.
When a user prompt exceeded 681,280 tokens, the prefill scheduler detected insufficient memory and triggered its rejection pathway. The scheduler invoked the internal method Req.set_finish_with_abort. This helper replaced the oversized prompt with a single-token dummy placeholder—a mechanism originally engineered to gracefully release request resources without crashing the worker process.
At this exact juncture, SGLang’s internal state machine diverged. The abort marker was registered in an internal dictionary (to_finish), but the primary status method finished_reason was never updated. Consequently, req.finished() continued returning False. A request that the engine had deliberately slated for termination retained an active status in subsequent lifecycle queries.
Eight Pages Reserved, One Written: Dirty VRAM Passes into Attention
In a conventional monolithic inference deployment, this state discrepancy would not have led to a data breach. The single-node scheduling loop would have subsequently caught the abnormal termination and propagated an abort directly to the client. On Workers AI production clusters, however, Cloudflare employed a Disaggregated Prefill and Decode architecture. Prefill and decode nodes were decoupled, and intermediate KV caches were transferred at high bandwidth over RDMA networks via the Mooncake framework.
Figure: The KV cache data leak path under disaggregated inference. When an oversized prompt was aborted during prefill, state propagation failed. The decode node subsequently bound unzeroed residual memory pages into the attention computation, leaking prior tenant data. Source: Cloudflare (blog.cloudflare.com).
To maximize concurrent throughput, physical GPU memory pages are not zeroed out prior to reallocation. Scrubbing every page on deallocation severely degrades memory bandwidth and request latency. The system inherently trusts scheduler metadata to govern memory page lifecycles. As a result, multi-tenant security rests entirely upon the integrity of state flags.
During cross-node handoff, the admission queue checked finished() rather than the internal abort dictionary. The scheduler erroneously treated the request as valid and dispatched it downstream. The prefill node allocated a single physical page for the 1-token placeholder, populated it with data, and notified the decode cluster that transmission was complete.
The receiving node lacked a crucial length-reconciliation check. Based on the original request’s massive advertised size, the decode node had pre-allocated 8 physical pages, yet it received only 1 freshly written page. The compute pipeline fed all 8 pre-allocated pages directly into the Attention kernel. The remaining 7 pages retained un-overwritten residual data belonging to prior tenants. That dirty, unscrubbed VRAM content was directly incorporated into subsequent matrix multiplications.
155 Responses with HTTP 200: Green Monitoring Hides the Breach
The data leak occurred entirely in silence across the production environment. Subsequent exhaustive log audits revealed that 155 responses delivered cross-tenant context to end users. Every single affected request had an input length strictly exceeding the 681,280-token threshold. The capacity boundary mismatch turned oversized requests into prime victims of the underlying memory hazard.
Deliberate malicious probing was ruled out. Cloudflare filtered internal debugging records and penetration tests conducted by security researchers, confirming that the timestamps and volume of affected invocations matched legitimate long-context usage patterns. The incident was triggered by standard production workloads rather than adversarial exploitation.
Monitoring probes along the entire inference pipeline failed to raise alarms. The API gateway recorded clean HTTP 200 status codes. Request latencies remained within expected baselines, and token accounting matched billing specifications. Infrastructure dashboards remained entirely green, leaving the vulnerability obscured beneath healthy operational metrics.
The natural text-completion capabilities of neural networks further masked the anomaly. When processing spliced Attention vectors, large language models still generated grammatically coherent prose. The model itself cannot verify the semantic ownership of its inputs. Traditional error logs and exception traps were bypassed, resulting in a textbook case of silent data leakage.
Inspecting Status Codes but Not Outputs: Outsourcing Security to the Engine
Similar warning signs had surfaced across the open-source ecosystem. As early as July 6, 2026, SGLang issue #30233 documented anomalous abort handling in the prefill scheduler. Cloudflare had also observed occasional anomalies internally since deploying v0.5.10, but sporadic incidents were initially dismissed as transient network jitter. Critical precursors were lost in operational background noise.
In its post-mortem, Cloudflare highlighted a pervasive blind spot across the AI infrastructure industry. Conventional pre-release quality assurance concentrates on API status codes, overall throughput, and long-context stress testing. Automated validation pipelines rarely check for semantic consistency between generated outputs and incoming prompts. Cloud providers have routinely assumed that as long as an inference engine does not throw unhandled exceptions, its internal state machine remains trustworthy.
The security perimeter for multi-tenant isolation lies squarely in GPU page ownership tracking. Because API gateways merely route traffic and enforce authentication, they cannot inspect low-level VRAM allocation tables. When cloud platforms lack independent verification of engine state, their defense-in-depth is effectively outsourced to a single line of inference framework code.
Doubling Headroom and Enforcing Cross-Node Reconciliation
Cloudflare’s incident response demonstrated commendable operational agility. The incident was formally prioritized on September 9 at 13:22, and the failure was reproduced within 25 minutes. The model was pulled offline by 14:00, all prefill nodes were rolling-restarted to scrub memory by 14:52, and a hotfix patch restored service at 19:40. In close collaboration with upstream SGLang maintainers, comprehensive hardening across production clusters was completed between September 11 and September 19.
On the configuration front, robust safety margins were instituted. Cloudflare expanded the prefill KV cache pool to approximately 2.04 million tokens—twice the advertised context window—aligning with the stable baseline previously established for GLM 5.2. The deployment pipeline added an automated admission gate: before any container starts, it directly inspects the engine’s physical KV pool allocation and validates it against the advertised context window, permanently eliminating capacity inversion.
Engine-level patches were promptly upstreamed. SGLang maintainers merged PR #38704, PR #36988, and PR #38935. Requests marked for abortion are now immediately dropped at the admission queue stage, strictly prohibiting memory allocation or cross-node transmission. Simultaneously, decode nodes now enforce strict page-count verification, raising immediate exceptions if received pages do not match reservations. These fixes are generally available in SGLang v0.5.21 and v0.5.22.
Cloudflare’s post-mortem exposes the engineering trade-offs inherent in disaggregated inference. In pursuit of throughput and memory efficiency, modern architectures decouple prefill and decode stages, streaming KV caches across high-speed fabrics while reusing dirty memory pages across requests. Yet this optimization placed the burden of multi-tenant isolation on a single, fragile boolean check. As long as cloud infrastructure verification stops at HTTP status codes, silent data leaks deep within GPU memory remain an ever-present risk.
References:
- Cloudflare Incident Report: KV Cache Exposure Remediation
- SGLang PR #38704: Verification of transmitted vs. reserved token count in disaggregated inference
- SGLang PR #38935: Drop aborted requests early in admission queue
- SGLang Issue #30233: Prefill scheduler abort handling notes
