Pinning Gates to One: Turning Floating-Point Memory into Physical RAM
Running 18 million consecutive CPU instructions, the error counter remained at zero. Approximately 230,000 output bits matched a standard hardware reference bit-for-bit. This benchmark comes from Qwenputer, an open-source project released by developer Divyansh Agrawal. Bypassing parameter scaling and external interpreters, the author compiled a full 32-bit RISC-V virtual processor directly into the weights of Qwen3.5-4B.
Large language models are notoriously prone to arithmetic errors. Because attention mechanisms rely on high-dimensional probabilistic associations, accumulating long numbers across floating-point weights inevitably drifts. Standard workarounds either rely on brute-force memorization through sheer parameter scale or offload calculations to external Python subprocesses. Qwenputer takes an entirely different engineering path: engineering strict hardware determinism inside the network’s internal topology.
The theoretical foundation stems from the seminal 2021 paper by Schmidhuber et al., which demonstrated that linear Transformers are fundamentally fast weight programmers. In the Gated DeltaNet architecture used by Qwen3.5, hidden states evolve via delta update rules. When the gating is pinned to 1 and the write strength is set to 1, state matrix updates degenerate into direct address-overwrite operations.
With one-hot vectors serving as key-value inputs, reading from and writing to the state matrix becomes exact, row-by-row addressing. Locking the gates alongside orthogonal input vectors transforms the neural network’s state matrix into physical, row-addressable RAM. General-purpose registers, the program counter, and main memory are all allocated to fixed rows within the matrix. Precision stems from structural constraints rather than convergence over statistical probabilities.
Figure: The Gated DeltaNet state matrix restructured as physically addressed memory, with registers, memory, and the program counter mapped to fixed rows. Source: divagr.com
11 Prefix Layers Execute 18 Million Instructions with Zero Errors
In terms of engineering implementation, the author inserted an 11-layer CPU-dedicated prefix network in front of Qwen’s original 32 layers. The base model weights remain frozen, acting purely as a task router and dispatcher. Arithmetic operations execute independently within the prefix layers. The entire system implements the full RV32IM instruction set architecture, supplemented by the Zba extension and a dedicated multi-precision integer unit.
Rigorous validation tests confirm the efficacy of this topological redesign. The system passed the official RISC-V architectural compliance test suite without a single error. In an uninterrupted, unguided run, it executed 18 million instructions, matching QEMU emulator results exactly bit for bit.
On DeepMind’s arithmetic benchmarks, the advantage of structure over raw scale is even more apparent. On the div_big division task, the vanilla Qwen3.5-4B base model achieves only 72.5% accuracy. Adding an external Python tool brings accuracy to 95%, while Qwenputer reaches 97.5%. On mixed long multiplication and division tasks, Qwenputer scores a perfect 100%, outperforming the Python toolchain by 2.5 percentage points.
On the multi-digit integer benchmark NUPA, base model accuracy plunges to 48.6%, whereas Qwenputer achieves a flawless 100%. Achieving perfect arithmetic accuracy does not require moving to massive models; it merely requires providing the right algebraic execution units. Crucially, general language capabilities remain intact: the modified model’s KL divergence on general text corpora is 6e-4, sitting right against the 5e-4 baseline noise floor.
Figure: Performance comparison across arithmetic benchmarks among Qwenputer, base Qwen3.5-4B, and an external Python toolchain. Source: divagr.com
53,000 Tokens in Matrix Slots: State Addressing Replaces Full Context Scans
Since the state matrix can function as overwritable memory, abstracting it into persistent storage is a natural next step. In an experimental branch, the author built a lightweight file system operating entirely within the weight states. The system hashes directory paths into 1024 fixed slots across the DeltaNet state, resolving hash collisions through linear probing.
This storage structure holds up to 512 independent files, roughly equivalent to 53,000 tokens of text. To retrieve a file, the network only executes 2 to 3 state transitions, eliminating the need to scan through historical context token by token. The addressing mechanism compresses long-context retrieval overhead from a linear dependency on context length down to constant-time state stepping.
This physical state persistence reshapes the memory model for multi-turn interactions. At the end of a session, the state containing the file system and RAM can be exported losslessly and reloaded in the next run. In cross-session fact-updating tests, standard context concatenation achieves only 30% accuracy. An external Python memory database scores 78.9%, while Qwenputer hits 90% purely through in-state updates.
Conventional architectures offload conversational state to external vector databases, incurring text parsing and context window filling costs on every turn. Qwenputer solidifies memory inside the recurrent matrix’s physical addresses, turning updates into single row-level incremental writes. This demonstrates that an LLM’s native internal states possess practical engineering potential to host long-term, exact knowledge stores.
Doubling Memory Overhead: Self-Contained Execution Without External Sandboxes
Embedding deterministic hardware logic directly into neural networks demands architectural tradeoffs. The first is storage footprint expansion. To match execution paths, the 11 prefix layers include substantial zero-padded parameters. The modified checkpoint weighs in at 18.8 GB, nearly double the original base model’s size.
Execution latency is another notable performance penalty. In standard inference, each generated token corresponds to final output text. In Qwenputer, each virtual instruction consumes several internal decoding tokens. On a single NVIDIA L4 GPU, the batched execution engine achieves a throughput of roughly 3,200 CPU tokens per second, noticeably slower than native execution.
System engineering complexity also shifts to the driver layer. To prevent intermediate instruction execution traces from polluting the natural language output space, the model cannot simply spit out raw characters. It relies on a custom RvChat driver to inject firmware, intercepting and masking low-level CPU instruction traces so only the final execution state is handed back to the text generation layer.
Trading doubled model size and extra inference cycles for self-contained execution on a single GPU represents a distinct engineering tradeoff. The entire system runs without needing an isolated host sandbox or privileged container runtimes. On a single 24 GB VRAM GPU, the model executes full arbitrary binary programs inside a strictly bounded environment.
Computation Hidden in Matrix States: Breaching Chain-of-Thought Auditing
Beyond leaps in arithmetic precision, this experiment exposes a critical AI safety dilemma. When neural internal states are converted into deterministic addressing hardware, they naturally become opaque information black boxes. The author highlighted how this architecture challenges existing safety and alignment defenses.
Today’s mainstream safety auditing and behavioral monitoring systems rely heavily on Chain-of-Thought (CoT) inspection. Auditing models read intermediate reasoning tokens step by step to catch malicious planning or deception. Under the Qwenputer paradigm, however, multi-step complex reasoning, memory reads, and memory writes occur entirely inside the state matrix. The execution process leaves zero explicit token footprints for external monitors.
External monitoring systems see only benign natural language inputs and final conclusions. Hundreds of thousands of intermediate hardware-level computations leave no trace in the context window. When reasoning shifts from explicit text generation into silent matrix addressing, safety probes dependent on intermediate textual traces become entirely blind. Hiding uninspectable execution logic directly inside model weights transitions from theory to practical reality.
Qwenputer’s core value lies in reframing how we perceive model accuracy. Precise computation does not strictly require stacking parameters or maintaining clunky external code sandboxes. As long as the network topology supports deterministic addressing, a 4B parameter model can deliver zero-error, hardware-grade execution. Structural determinism rewrites the rules of internal information flow—and confronts black-box model auditing with a brand new technical reality.
Reference Links:
- Qwenputer Project Technical Report
- ICML 2021 Paper: Linear Transformers Are Secretly Fast Weight Programmers
- RISC-V Official Instruction Set Architecture Specifications and Compliance Test Suite
