On July 28, 2026, Sebastian Raschka published a short blog post titled “Kimi K3 Architecture Notes.” It was not a lengthy technical paper, just annotations on an architecture diagram—yet once posted, it quickly climbed to the front page of Hacker News, earning over 270 upvotes.
On the very same day, Moonshot AI’s paper from last October, “Kimi Linear: An Expressive, Efficient Attention Architecture,” also surfaced on the HN front page, garnering over 260 upvotes. Having two papers from the same lab featured at the top of the Western tech community on the same day was no coincidence.
Figure: The Kimi K3 architecture diagram analyzed by Sebastian Raschka in his blog post. From Attention Residuals to LatentMoE, each component reflects a distinct design philosophy. Source: Sebastian Raschka
If DeepSeek made the West aware of Chinese AI’s “presence” in early 2025, July 28, 2026 was the day Moonshot AI’s two papers forced Western tech observers to take a serious look at Chinese AI’s “originality."
"They Are Not Just Distilling”
For a long time, Western tech media coverage of Chinese AI has been bound to a single word: distillation. In February of this year, Anthropic publicly accused DeepSeek, Moonshot AI, and MiniMax of using over 24,000 fake accounts and 16 million conversations to systematically distill Claude. The U.S. Treasury Department even threatened sanctions against Moonshot. In the mainstream Western narrative, Chinese AI labs were cast as “tech thieves”—incapable of true innovation and relying entirely on copied data.
That narrative is now crumbling.
One of the top-voted comments on HN came from user constantlm:
“So, unlike what leaders of western labs would like you to believe (that Kimi is just the result of distillation attacks), they are introducing new and novel approaches.”
That single sentence received dozens of upvotes. The comment section was filled with similar technical engagement: researchers discussed the implications of the NoPE architecture, compared the technical differences between DeepSeek V4’s Hyper-Connections and Kimi’s Attention Residuals, and calculated the FLOPs savings brought by Kimi Delta Attention. Nobody was talking about “distillation” anymore.
Kimi Delta Attention: Linear Attention Beats Full Attention For the First Time
To understand the weight of this moment, some technical context is necessary. Linear attention is a research direction pursued for years to solve the core bottleneck of the Transformer architecture: the $O(n^2)$ computational complexity of standard attention mechanisms. Simply put, as context length grows, computational costs scale quadratically. Linear attention attempts to reduce this complexity to $O(n)$, but the trade-off has historically been clear: performance falls short of full attention.
The core contribution of the Kimi Linear paper is that, under a fair comparison, it enabled linear attention to surpass full attention across all dimensions—short text, long text, and even reinforcement learning (RL) scenarios.
Figure: Core concept of Kimi Delta Attention — independently controlling “write weights” and “forget weights” per channel, enabling more effective use of limited memory states. Source: Zhouyao Xie
The core building block achieving this breakthrough is Kimi Delta Attention (KDA). Built upon Gated DeltaNet, it introduces a fine-grained gating mechanism that utilizes limited memory states far more efficiently. While traditional linear attention treats all memory channels equally when writing state, KDA independently controls the “write weight” and “forget weight” for every single channel—determining which information to retain, update, or discard.
This fine-grained control at every layer and channel allowed linear attention to match and exceed full attention quality for the first time.
The paper also introduced an elegant engineering optimization: a specialized variant of Diagonal-Plus-Low-Rank (DPLR) matrices for hardware acceleration. Without sacrificing model performance, KV cache footprint was reduced by up to 75%, and decoding throughput on 1M token contexts increased by 6x.
A crucial engineering insight lies here: countlessly many linear attention papers have been published over the past few years, yet almost none were adopted in real-world production. The reason was “great theory, poor hardware implementation”—mathematically $O(n)$ looks clean, but on actual GPUs, it could not beat highly optimized cuDNN full-attention kernels. KDA specially designed a hardware-aware chunkwise algorithm, making this architecture truly ready for production deployment—rather than remaining confined to academic papers.
Kimi K3: Architectural Philosophy Behind 2.8 Trillion Parameters
If Kimi Linear represents the paper on the “engine,” Kimi K3 is the rollout of the “complete vehicle.”
Kimi K3 is a 2.8-trillion-parameter MoE model released by Moonshot AI on July 16, activating 16 out of 896 experts per token, with native multimodal capabilities and a 1M token context window. It is currently the world’s largest open-weights model—not one of the largest, but the largest.
What prompted Sebastian Raschka to write an architecture breakdown was a series of bold design choices in K3:
Attention Residuals: A unique component in K3. Traditional residual connections simply add input to output to assist gradient flow. Attention Residuals use attention scores to weight how much each layer should contribute to the final output—a weighted aggregation rather than simple addition. According to the technical report, this consistently improved validation loss and downstream performance while adding only ~4% to training costs and ~2% to inference overhead.
NoPE (No Position Encoding): K3 is the first known frontier model architecture to completely discard position encodings across all layers in production. While recent trends used RoPE for local attention and NoPE for global layers, K3 eliminated position encodings entirely across all layers—unprecedented for a production-grade flagship model.
LatentMoE: While not strictly unique to Kimi (Nemotron 3 explored a similar idea), K3 demonstrated the viability of LatentMoE at the 2.8T scale. The core idea compresses large linear layers of expert networks into a low-dimensional latent space before computation, drastically reducing parameter footprint without sacrificing accuracy.
As Raschka summarized in his post: “The overall trend with K3, like Nemotron 3 and DeepSeek V4, is moving toward higher inference efficiency. The core idea is replacing each component with an efficiency-optimized version.”
Same Community, Different Voices
An interesting observation from these two papers reaching the HN front page simultaneously is the dialogue they sparked.
The HN discussion on Kimi Linear (114 comments) leaned heavily academic—researchers debated whether linear attention could truly replace full attention, the applicability of the Bitter Lesson, and scaling laws. This was typical for a technical paper submission.
The discussion on Raschka’s K3 architecture notes (34 comments), while smaller, carried higher information density. Because the post analyzed K3 from an architectural perspective, the comment section immediately transitioned into a debate on whether Chinese AI is merely distillation. User thatsagasey referred to Raschka as a “great LLM researcher/author,” which itself serves as an endorsement—when a Western researcher of Sebastian Raschka’s stature seriously analyzes the architecture of a Chinese lab, the “distillation-only” narrative loses its footing.
Turning Point from “Follower” to “Original Innovator”
Looking at the broader timeline, the trajectory of original architectural contributions from Chinese AI labs is clear:
- 2024: DeepSeek V2 introduced Multi-head Latent Attention (MLA), widely recognized as one of the most significant original contributions in KV cache optimization. Almost all recent efficient long-context models borrow from MLA.
- October 2025: Moonshot AI published Kimi Linear, demonstrating for the first time that linear attention can outperform full attention across all evaluation metrics.
- July 2026: Kimi K3 scaled Kimi Linear from 48B to 2.8T, validating the scalability of these architectural innovations in real-world production environments.
This is not a one-off stroke of luck from a single lab. DeepSeek’s MoE and MLA, alongside Moonshot’s Delta Attention and Attention Residuals, represent independent, original contributions from multiple Chinese labs across different architectural directions.
To be objective, Western researchers hold nuanced views. Some note that while Kimi’s Linear Attention is a clear advance, claims of “beating full attention for the first time” require further independent replication, and Gated DeltaNet research itself originated in Western labs. The degree of “originality” remains debated. Yet the debate itself marks progress—moving the conversation from “they only distill” to “how significant is their contribution” is an implicit acknowledgment of capability.
Industry Signals Behind Two Papers
Beyond technical discussions, Kimi K3’s release triggered broader waves. Bloomberg reported that K3’s release coincided with a tech stock sell-off on Nasdaq, and U.S. Treasury Secretary Bessent even threatened sanctions. A model release from a Chinese AI startup moving U.S. capital markets is a far more tangible signal than any benchmark score.
Equally notable is Moonshot’s adoption of a revenue-tiered licensing model for K3: free open-weights access for SMBs and researchers, and commercial licensing for large enterprises. This marks one of the first times a Chinese AI company has demonstrated a mature commercialization strategy around open source—building a moat at the intersection of technology and business.
Conclusion
Having two papers top Hacker News on the same day may have been a coincidence for Moonshot AI. But for the broader Chinese AI industry, it marks a narrative turning point.
When Sebastian Raschka analyzes K3’s architecture diagram layer by layer, and Western engineers on HN debate the merits of AttnRes and NoPE rather than talking about “distillation,” Chinese AI has crossed the threshold from product follower to architectural innovator. Distillation can help catch up, but surpassing requires originality. Originality does not happen overnight, but when it arrives, it does not pass quietly.
On July 28, 2026, that signal arrived.
References:
- Sebastian Raschka, “Kimi K3 Architecture Notes”, Jul 28, 2026
- Kimi Team, “Kimi Linear: An Expressive, Efficient Attention Architecture”, Oct 2025 (arXiv)
- Hacker News Discussion: Kimi K3 Architecture Notes (item?id=49085698)
- Hacker News Discussion: Kimi Linear (item?id=49082022)
- Moonshot AI, “Kimi K3 Tech Blog: Open Frontier Intelligence”
- Moonshot AI, “Kimi K3 Quickstart”
- Zhouyao Xie, “Behind Kimi K3: Understanding Kimi Delta Attention”, Jul 2026
- GitHub: MoonshotAI/Kimi-K3
- GitHub: MoonshotAI/Kimi-Linear
- Pandaily, “Deconstructing the Kimi K3 Technical Report”, Jul 2026
