A palm-sized microcontroller development board operates without a game engine or external servers. Inside the chip, it autonomously computes the causal chains of a physical environment. In the open-source tinyworldformer project, a Raspberry Pi Pico 2 W hosts the complete prediction and decision-making loop. Featuring just 520KB of SRAM, the chip simultaneously executes a recurrent Transformer that generates frames and a decision network that chooses actions.
World models are traditionally anchored to energy-hungry cloud clusters, spanning billions of parameters. tinyworldformer takes a radical lightweight path. It compresses networks down to 26k–131k parameters, dropping per-frame computation to 2M–8M MACs. Environments for 5 classic games fit neatly into 4MB Flash and 520KB SRAM.
Reinforcement learning training unfolds entirely within the network’s predicted frames. The agent never interacts with the original game engines. This end-to-end neural simulation eliminates cloud overhead, porting world models and reinforcement learning directly onto microcontrollers. The trade-offs lie in visual fidelity and an 83.6% to 100% frame accuracy, completely bypassing external computing costs.
Mirrored Twin Networks: No Real Game Running on the MCU
The RP2350 powering the Pico 2 W features dual Cortex-M33 cores. The system cleanly delineates display refresh duties from causal inference across physical cores. Core 0 handles USB communication and screen output. Core 1 independently runs two mirrored neural networks, entirely isolated from real game engines.
The world model advances environmental state predictions. Taking the current frame and action as input, it predicts the subsequent frame and associated reward. Its architecture is an int8-quantized recurrent Transformer spanning 26k to 131k parameters. Generating 2M to 8M multiply-accumulate operations per frame (reaching 8.2M MACs in Pac-Man lite), it guarantees on-chip inference keeps pace with the control loop.
The actor senses the visual state and selects actions. Built as a two-layer float32 MLP with just 9k to 16k parameters, it demands only 9k to 16k MACs per frame—an almost negligible computational load. The actor inspects solely the world model’s predicted features, while the world model receives only discrete actions chosen by the actor.
This dual-network setup allows flexible human takeover. The pre-trained onboard actor can be hot-swapped for a stream-x agent learning from scratch. Whenever a developer presses the onboard button, control switches instantly to a human player. As players tap buttons, they navigate purely within fictitious frames hallucinated in real time by the neural network.
Figure: tinyworldformer architecture and overview of twin networks running on the MCU. Source: SheepSeb/tinyworldformer.
Recurrent Weight Reuse: Trading Latency for Storage Footprint
Conventional vision Transformers stack multiple blocks to scale capacity. However, microcontroller SRAM cannot store separate weights for dozens of layers. tinyworldformer adopts a recurrent Transformer architecture, retaining only a single Transformer block globally. The model loops the same weight set 1 to 4 times, re-adding the initial input residual before each iteration.
During training, intermediate outputs from every recurrent pass are supervised against the ground-truth next frame. At runtime, the loop count turns into an adjustable hardware knob trading latency for inference accuracy. On Pico 2 W hardware, the system defaults to 2 recurrent iterations to maintain an interactive frame rate. This architecture places depth configuration directly into the hands of the edge execution environment.
In a standalone Doom-lite test case, recurrent iteration dramatically suppresses visual hallucinations. Under a single recurrent pass, the model hallucinates a non-existent enemy, generating 92 erroneous grid cells across the frame. Bumping iterations to 2 drops errors to 35 cells. A 3rd iteration reduces errors to 12, and 4 iterations eliminate all artifacts entirely.
The project evaluated 5 games under 2 recurrent iterations. Breakout (20×16 grid) achieves 99.99% per-frame accuracy with 26k parameters and 4.3M MACs. Flappy Bird (24×24 grid) hits 100% accuracy with 46k parameters and 2.0M MACs. Pong (20×32 grid) posts 99.45% accuracy with 46k parameters and 2.3M MACs, accurately fitting deterministic 2D physics.
Environments requiring deeper state tracking and pseudo-3D rendering encounter capacity boundaries. Pac-Man lite packs an intricate maze and two ghosts into a 48×48 grid; parameters scale to 131k and compute reaches 8.2M MACs, yielding 86.4% accuracy. Doom-lite renders a 32×32 first-person view across an 8×8 maze, demanding 55k parameters and 4.2M MACs for an 83.6% accuracy. Tracking hidden multi-entity states forces low-parameter models to accept localized grid glitches.
Figure: Consecutive Pac-Man prediction frames generated autonomously by the world model without a real game engine. Source: SheepSeb/tinyworldformer.
Heuristic Decoding: Enforcing Constraints on Probability Distributions
If lightweight neural networks sample freely from unconstrained Softmax distributions, generated physical worlds collapse within steps. Pac-Man can duplicate into ghost-eating twins, while bouncing balls evaporate into thin air. Heuristic decoding gets its own chapter because raw probability distributions do not naturally form valid game frames.
The first rule targets static backgrounds. Tiles like walls, bricks, pellets, and floors independently pick the highest-probability token. Individual argmax evaluation bypasses global graph search costs, locking maze borders to prevent wall creep.
The second rule governs singleton entities. Singletons such as balls and player sprites are assigned exactly once across all spatial grids and classes. Global uniqueness is strictly enforced. For stochastic events like respawns or ball serves, the decoder samples within a 2.2-logit neighborhood of the top candidate, retaining dynamic variety while suppressing mean-blur residue.
The third rule manages moving sprites occupying 2×2 composite cells. For sprites like Pac-Man or ghosts, only the anchor tile maintains latent state, recording mouth direction or ghost-pellet occlusions. The remaining three cells are stamped via deterministic layout templates, skipping point-wise prediction and eliminating sprite tearing.
Full int8 Quantization: Two MACs per SIMD Instruction
Porting Transformers to microcontrollers clashes directly with floating-point matrix multiplication energy budgets. tinyworldformer compresses all world model weight matrices to int8 integers. The network computes scaling factors row-wise per matrix output (scale = max|w| / 127). Activation tensors are also quantized to int8, receiving dedicated scales partitioned by tensor affiliation and recurrent iteration.
The Cortex-M33 core inside the RP2350 provides low-level integer acceleration instructions. A single 32-bit memory read retrieves 4 weights, SXTB16 unpacks bytes into two 16-bit pairs, and SMLAD completes dual multiply-accumulates into a 32-bit register in a single instruction (attention uint8 probabilities × int8 values leverage UXTB16). Once int32 accumulation concludes, the pipeline applies a single dual-scale floating-point rescaling. Coupled with a 1024-entry lookup table for uint8 Softmax exponentials, continuous floating-point overhead is eliminated.
The RP2350’s hardware FPU does not sit idle. LayerNorm, residual additions, and input embeddings stay in float32. These non-GEMM operations account for a tiny fraction of overall compute. Preserving floating-point representation blocks deep numerical drift, bridging fixed-point acceleration with numerical stability.
High fidelity stems from strict quantization alignment between training and deployment. Quantization-aware training (QAT) with straight-through estimators (STE) injects identical integer rounding during training; what gets trained is exactly what gets deployed. The exporter module rustexport/intref.py locks in NumPy operation order. The Rust MCU inference engine replicates reference outputs bit-for-bit, eliminating cross-platform drift.
Figure: Row-wise and per-iteration int8 quantization scheme for matrices and activations. Source: SheepSeb/tinyworldformer.
Zero Heap Allocations: Securing 520KB SRAM for On-Device Simulation
Embedded environments have zero tolerance for out-of-memory crashes. Analyzing ELF binaries with size -A, Breakout consumes 361KB Flash and 299KB RAM; Flappy Bird uses 409KB and 227KB; Pong takes 414KB and 244KB. Doom-lite occupies 471KB and 357KB, while Pac-Man lite reaches 600KB Flash and 460KB RAM. All firmware resides safely within the Pico 2 W’s 4MB Flash and 520KB SRAM envelope.
Memory profiling exposes an unintuitive reality. Quantized int8 weights account for only 19.7KB to 33.9KB of the entire world model. In contrast, float32 embeddings, normalization constants, and scale arrays form the lion’s share of memory, occupying 31.6KB to 91.9KB. Storage footprints concentrate largely in edge adaptation layers rather than the compact compute core.
To ensure long-term stability, firmware is written in no_std Rust using the Embassy async framework. The system achieves zero dynamic heap allocations across its lifecycle. All model weights reside permanently in Flash as static arrays, while intermediate activation buffers and display memory are pre-allocated as fixed-size static RAM arrays. Eradicating dynamic heap allocation removes fragmentation risks and out-of-memory panics.
The onboard stream-x learner requires just 49KB to 106KB for weights and eligibility traces, operating safely within static buffers. A dual-core async pipeline ensures responsive interaction. Core 1 loops through feature extraction, action selection, 2 recurrent inference passes, heuristic decoding, and reward prediction. Inference data flows to Core 0 through a 2-slot channel. Core 0 drops frames when busy, guaranteeing Core 1 inference never stalls.
On desktop CPUs, the Rust engine processes a frame in just 0.7ms (Flappy Bird) to 1.5ms (Breakout). While raw MCU millisecond figures were not benchmarked on the Pico 2 W, pacing interactive human gameplay to 66ms–150ms per frame delivers a responsive experience. All 5 environments are open-sourced under the SheepSeb/tinyworldformer repository, supporting terminal execution via uv alongside a WebAssembly web demo.
World models do not have to live exclusively on massive cloud clusters. By combining temporal weight recurrence with fixed-point SIMD pipelines, tinyworldformer demonstrates autonomous physical simulation directly on microcontrollers. Trading off visual resolution for server independence, a few hundred kilobytes of memory proves sufficient to run dedicated physical sandboxes for autonomous agents.
References:
- SheepSeb Blog: tinyworldformer Technical Deep Dive and Implementation Details
- GitHub Repository: SheepSeb/tinyworldformer
