On September 17, 2026, a development challenged the prevailing orthodoxies of the global compute crunch. z.ai publicly disclosed how it built a full production-grade inference service from scratch across a massive cluster of more than 100,000 Chinese-made AI accelerators. Crucially, all production inference traffic for its flagship GLM-5.3-Flash model is now running entirely on this non-NVIDIA infrastructure. What is most striking is the deployment velocity: from the model’s first successful run on the unfamiliar silicon to reliably handling the full torrent of live production workloads, the entire engineering cycle was compressed into just two weeks.
62 Trillion Real-World Tokens in 14 Days
Before its formal public reveal, GLM-5.3-Flash underwent an intensive full-load stress test. Running under the pseudonym Ox-Alpha, it was deployed anonymously across two popular developer platforms: OpenCode and OpenRouter. Within six days, the model processed more than 62 trillion tokens, quickly claiming the top spot on usage leaderboards across both services. This massive concurrent demand subjected the domestic infrastructure to a trial by fire under authentic production workloads.
Rather than buckling under the volume, the sustained concurrency exposed significant latent headroom in the underlying silicon. Over a two-week tuning period, the cluster’s end-to-end throughput surged roughly 3× relative to the initial baseline. This leap fundamentally transformed hardware utilization efficiency, bringing per-token inference costs into direct parity with mainstream NVIDIA GPU architectures. Achieving cost parity demonstrated that, by bridging ecosystem gaps through rigorous software engineering, non-NVIDIA accelerators can establish a viable commercial cycle even under unforgiving market realities.
The central catalyst of this rapid turnaround was an Infra Agent driven by GLM-5.3 itself. The model took the field directly, actively participating in the construction, debugging, and continuous optimization of the physical environment that serves it. Using a frontier reasoning model to systematically eliminate low-level compute bottlenecks became a repeatable, scalable engineering practice across a 100,000-chip network.
Figure: The end-to-end throughput evolution of GLM-5.3-Flash. Source: z.ai official blog
Algorithmic Engineering Overcomes Inherent Hardware Constraints
The engineering team did not shy away from the harsh initial conditions. The cluster’s accelerators suffered from severe intrinsic constraints in memory capacity and inter-chip bandwidth. To make matters worse, the accompanying software ecosystem was nearly barren, with kernel-level driver support riddled with edge-case bugs. Across many critical architectural junctures, essential low-level documentation was entirely absent, leaving engineers to rely on technical intuition, educated guesses, and tedious reverse-engineering.
To circumvent these physical limitations, the infrastructure team stacked an aggressive software optimization pipeline. Architecturally, they implemented intra-node tensor parallelism for linear attention and the LM Head, while introducing ReplaySSM to control internal state. On the memory front, they rolled out W8A8 quantization coupled with mixed-precision KV Cache quantization across INT8, FP8, and BF16 formats. Finally, combined with Layer Split techniques and an Encode-Prefill-Decode (EPD) disaggregated serving architecture, fine-grained compute scheduling squeezed efficiency out of every clock cycle.
These layered optimizations aimed directly at breaking through memory and bandwidth bottlenecks. When on-chip VRAM could not hold the full parameter footprint, and physical inter-chip interconnect limits capped data transfer rates, engineers compensated with microsecond-level execution slicing and relentless scheduling precision. Rather than derailing the project, the immaturity of the hardware stack compelled the team to develop a problem-solving methodology completely free from CUDA-centric assumptions.
Figure: The Infra Agent optimization loop built around dense feedback. Source: z.ai official blog
Deconstructing the Black Box with Dense Feedback Loops
Completing such a radical low-level codebase overhaul within 14 days relied on a foundational methodology: “dense feedback.” When debugging across a 100,000-node heterogeneous cluster, the worst nightmare is unaccounted performance degradation. Rapidly transforming an opaque system error into actionable attribution—pinpointing which layer failed, why throughput dropped, and what experiment to run next—became the cornerstone of their tuning speed.
To pierce the black box, the optimization team established three non-negotiable feedback principles. First, feedback had to be strictly localized. Observability tools needed to pierce the hardware layer to isolate specific kernel executions, environmental triggers, and microsecond-level runtime intervals. The staggering unpredictability of a massive compute fabric was thus dismantled into localized, observable, and controllable state slices.
Second, the cost of obtaining this feedback had to be minimal, and turnaround times had to be instantaneous. Any performance hypothesis that could be tested via kernel-level unit tests or local micro-benchmarks was strictly resolved during development, never pushed blindly into full cluster deployment. Third, every optimization had to be objectively verifiable through rigorous comparative experiments, forbidding speculative conclusions based merely on loose runtime correlations.
Figure: Inference system optimization workflow. Source: z.ai official blog
Impressive Benchmarks Collide with Real-World User Scrutiny
The dramatic throughput trajectory showcased in the official report did not convince everyone in the engineering community. A lively debate on Hacker News, spanning hundreds of comments, revealed a notable divergence in user experiences. While many practitioners marveled at the rapid overhaul of such a massive domestic cluster, others pointed to their own API latency logs to mount a sharp counterpoint.
Several developers reported that in complex, long-context production scenarios, GLM’s API response times felt sluggish, accompanied by aggressive concurrency rate limits. Developers attempting overnight autonomous inference jobs noted that sessions often faltered within a few hours due to unexpected network errors or restrictive quota caps. This friction highlighted an inevitable gap between pristine synthetic benchmarks in isolated lab environments and the chaotic realities of live network conditions and unpredictable client workloads.
The conversation quickly spilled beyond engineering metrics into broader AI safety and capabilities narratives. Critics noted that the paper’s portrayal of AI exhibiting self-improving potency that bordered on dangerous felt disconnected from the brittle service reliability they encountered when querying endpoints. Meanwhile, overseas engineers posed a provocative question: if 100,000 constrained accelerators can reach baseline parity purely through algorithmic ingenuity and agent-driven automation, why haven’t well-capitalized Western tech giants with top-tier GPUs executed comparable co-designed optimizations?
Iteration Velocity Redefines the Hardware Competition Baseline
The engineering team explicitly framed this agent-led deployment as an early form of recursive self-improvement (RSI). The model demonstrated the capacity to inspect, comprehend, analyze, and actively rewrite the infrastructure codebases that sustain its own execution. Yet the team maintained measured restraint, emphasizing that humans must remain firmly in control when setting optimization targets, defining code alteration boundaries, and mitigating safety risks.
The system forged over those two intense weeks delivers an undeniable signal: the long-held industry consensus that non-NVIDIA domestic compute is unviable in production is already half obsolete. On a constrained hardware platform plagued by limited memory capacity and zero interconnect advantages, the team drove cloud inference costs down to parity with mainstream GPU vendors by maximizing feedback density and delegating grueling trial-and-error tasks to LLMs.
The hardware ecosystem chasm will not vanish through a single two-week sprint. The true variable is how tightly the loop from error detection to kernel repair can be compressed once feedback achieves fine granularity. This architecture demonstrates an upper bound: on chips where memory and bandwidth are constrained, iteration velocity can partially substitute for raw hardware specifications. Exactly how far that substitution can go remains an open question, with only two weeks of data from a single company to guide us.
References:
