155 Trillion Daily Tokens Expose Production-Scale Realities
On October 20, 2026, DeepSeek officially open-sourced its production inference framework, DeepInfer, alongside the protocol parser library deepseek-recipe and an unprecedented set of operational metrics. Processing an average of 155 trillion tokens per day per cluster while generating an accounting gross run-rate of $38 per GPU-hour, this release offers an unvarnished look at the compute throughput required in hyper-scale production environments.
The $38 per GPU-hour figure reflects gross cluster turnover based on off-peak list pricing rather than net profit. Under a 1:1 hardware ratio between prefill and decode phases, the theoretical gross revenue per accelerator hovers around $36.70 per hour—deviating by a mere 3% from the published numbers. Sustaining such minimal variance at hyper-scale demonstrates that DeepSeek’s production traffic orchestration has achieved industrial-grade load leveling.
The supported hardware matrix establishes clear technical boundaries. DeepInfer’s initial release supports only two high-end accelerators: NVIDIA SM100 and Huawei Ascend 950. Consumer-grade GPUs are entirely excluded. The build environment strictly requires Linux, GCC 13.0, and Python 3.12, interfacing directly with CUDA 13 and CANN 9.2 at the lower level. The architecture is engineered exclusively for high-density concurrency across massive data centers, explicitly abandoning lightweight portability.
96% Cache Hit Rate Drives a 25x Performance Chasm
Prefix cache hit rates represent the primary technical leverage separating baseline from frontier inference performance. According to DeepInfer’s technical release notes, cold-start prefill throughput on an isolated SM100 sits at 60,000 tokens per second. Once a prefix cache hits, single-card prefill throughput surges to 1.5 million tokens per second. This 25-fold performance disparity underscores how severely redundant prefix recomputation degrades effective hardware utilization.
Production telemetry reveals an average input context length of 77,000 tokens. Global Time-to-First-Token (TTFT) averages 400 milliseconds, with P99 latency capped tightly under 1,000 ms. To facilitate prompt reuse across extended multi-turn conversations, the engineering team deployed a dedicated, distributed KVCache storage cluster with a 72-hour retention window. Without the lifecycle management provided by this external storage tier, a 96% prefix cache hit rate could never translate into sustained, real-world throughput gains.
Figure: Official DeepInfer Prefill Pareto curve (throughput vs. latency) benchmarked across a 4x SM100 node. Note the steep climb in throughput under cache-hit conditions as concurrency scales, while Time-to-First-Token remains remarkably flat. Source: deepseek-ai/DeepInfer
Cache hit rates directly reshape the unit economics of LLM inference. In DeepSeek’s published pricing schedule, cached input tokens are billed at $0.003 per million tokens (¥0.02), whereas uncached tokens cost $0.14 per million tokens (¥1.00). If hit rates degrade from 96% down to 80%, the effective blended input cost surges from ¥0.06 to ¥0.22 per million tokens. Prefix reuse efficiency has become the definitive cost control lever, rendering raw peak FLOPS a secondary consideration.
Ascend 950 and SM100 Enter as First-Class Citizens
Huawei’s Ascend 950 enters DeepInfer as an official tier-one platform. Developers can build natively for the accelerator simply by passing export DEEPINFER_ACCELERATOR=ascend, compiling directly against CANN 9.2. This milestone marks the transition of domestic Chinese AI silicon from third-party community forks into day-one upstream architecture support.
This breakthrough builds upon extensive full-stack kernel optimization. DeepSeek had previously open-sourced foundational primitives, including the FlashMLA attention operator, the DeepEP communication library, and DeepGEMM for matrix multiplication. On the Ascend 950DT, the Ascend-tailored DeepGEMM achieved 99.8% of theoretical hardware utilization in dense matrix multiplication. Coupled with open model weights, low-level kernels, communication collectives, and top-level orchestration have formed a fully integrated, transparent stack.
A unified runtime normalizes scheduling across heterogeneous compute hardware. Facing identical model workloads, Ascend 950 and SM100 operate under identical execution pipelines. Deploying domestic silicon as a primary workhorse under workloads exceeding hundreds of trillions of daily tokens confirms that China’s AI hardware and software ecosystem has crossed the threshold from functional viability to industrial-grade deployment.
Four Architectural Refactors Eliminate Long-Sequence and Batching Overheads
The autoregressive decoding phase imposes dual constraints: aggregate cluster throughput and per-request generation speed. Within a single 64-GPU SM100 domain, DeepInfer sustains a decode throughput of 14,000 tokens per second per card while maintaining a single-stream output velocity of 230 tokens per second. The physical cluster ratio between prefill and decode nodes remains locked at 1:1, challenging traditional architectures that routinely allocate several decode nodes for every prefill node.
Figure: Official DeepInfer Decode Pareto curve evaluated across a 64x SM100 cluster domain. The curve highlights stable token-per-second velocity under high-concurrency batching, simultaneously balancing total throughput and single-user latency. Source: deepseek-ai/DeepInfer
Long-sequence recomputation penalties are mitigated through a bounded-replay sliding window mechanism. Traditional sliding window architectures often trigger full recomputation of historical context once boundaries are exceeded. DeepInfer enforces a bounded suffix replay restricted to 140 tokens. Regardless of how long the context window expands, the historical recomputation penalty is locked into a predictable, constant-time overhead.
Speculative decoding leverages the DSpark parallel draft module paired with dynamic confidence-based scheduling. The engine dynamically tunes verification draft lengths based on instantaneous cluster load. Under matched throughput, this design delivers a 60% to 85% generation speedup over standard MTP-1 baselines, yielding a net 6% boost to overall cluster throughput.
An asynchronous scheduler pipelines CPU orchestration and GPU execution, completely eliminating batching bubbles. On NVIDIA hardware, enabling batch-invariance guarantees deterministic outputs across identical prompts. Simultaneously, the companion deepseek-recipe library standardizes varied API ingestion patterns, handling prompt formatting and structured schema validation.
Core Engine Release Establishes Production Boundaries
The boundaries of DeepInfer’s open-source release reveal a deliberate engineering distinction. The public release focuses on the high-performance offline inference engine and single-node scaling primitives. DeepSeek’s hyper-scale online infrastructure remains proprietary: neither the distributed 72-hour KVCache storage tier nor the cross-replica global load balancers were included in the open repository.
This architectural division clearly delineates on-premise deployments from hyperscale managed APIs. Enterprise engineering teams can adopt the optimized kernels and batch schedulers to replicate state-of-the-art Pareto curves locally. However, operating highly resilient, globally distributed cloud inference still demands specialized cluster management infrastructure.
The AI price war has decisively transitioned into low-level systems engineering. Open model weights merely demonstrate model capability; the inference orchestration stack determines the true cost of token delivery. As Huawei Ascend 950 and NVIDIA SM100 jointly shoulder daily enterprise workloads of 155 trillion tokens, raw peak hardware specifications are no longer the deciding factor. Full-stack system co-design and production-hardened scheduling represent the true foundation of enterprise-scale AI deployment.
Reference Links:
- DeepInfer Official Repository and Architecture Guide
- deepseek-recipe Protocol Parsing and Formatting Library
- DSpark Speculative Decoding Paper (arXiv:2607.05147)
- Hacker News Community Technical Discussion
