In July 2026, an open-source project named TurboFieldfare sparked intense discussion on Hacker News, earning over 608 points and 200+ comments. The reason was striking: it allowed a 26-billion-parameter large language model to run on a Mac laptop with only 8GB of total RAM, consuming just around 2GB of memory footprint.
26 billion parameters. 2GB of RAM. Placing these two numbers side-by-side seems counterintuitive.
A large model does not equal a giant GPU. That is the central message of this article.
A Seemingly Impossible Task
First, some background. Gemma 4 is an open model family released by Google DeepMind in early 2026. One particular variant, named “Gemma 4 26B-A4B”, uses a Mixture-of-Experts (MoE) architecture (explained below). Even after 4x compression, this model still requires 14GB of disk space.
Yet, most consumer Mac laptops come with 8GB of RAM—with the operating system itself taking up 3–4GB, leaving very limited space for large models.
Traditional inference tools (such as llama.cpp or MLX) load the entire model into memory before running. A 14GB model simply cannot be fully loaded on an 8GB machine—the OS will not allow any application to hog all system RAM. That path was a dead end from the start.
So the question became: Is there a way to run the model without loading all of it into memory?
Andrey Mikhaylov, creator of TurboFieldfare and an iOS/Metal engineer, answered with a resounding yes. Spending several weeks and conducting 103 experiments, he wrote a custom inference engine completely from scratch—using Swift and Apple’s Metal graphics framework. The result: 5 to 6 tokens per second on an M2 MacBook Air, and up to 31 to 35 tokens per second on the latest M5 Pro.
Not only does it work, but the speed is genuinely practical. This isn’t a theoretical paper—it’s a downloadable, runnable engineering achievement.
The Hidden Advantage of MoE: Tailor-Made for Streaming
To understand this breakthrough, we first need to look at how Mixture-of-Experts (MoE) architecture works.
A traditional LLM acts as a “generalist”—every single parameter participates in computation for every generated token. It’s like a large company where every single employee must review every single task, making it extremely inefficient.
MoE models operate differently. Think of a company with 128 specialized departments. Whenever a new task arrives, a dispatcher called the “router” analyzes the request and activates only the 8 most relevant departments. The remaining 120 departments stay idle.
Out of the total 26 billion parameters, only about 3.8 billion are actually active during any given computation step. This is what “A4B” in “26B-A4B” stands for: Active 4 Billion parameters.
This architectural property dictated TurboFieldfare’s engineering strategy: If less than one-tenth of the experts are used at any time, why keep the other 90% of weights residing in RAM?
Three Core Optimizations to Fit 14GB into 2GB
Traditional inference frameworks load all 128 experts’ weights into memory, keeping them on standby indefinitely—like reserving desks for all 120 idle employees on a prime office floor.
TurboFieldfare takes a radical, direct approach: Only bring weights to memory when they are actively working.
1. 4x Compression: Data Weight Reduction
Model weights contain significant numerical redundancy. Much like a 4K ultra-HD photo that looks virtually indistinguishable when compressed to 1080p, weights can be aggressively quantized.
TurboFieldfare uses 4-bit quantization—compressing parameter precision from 16-bit to 4-bit, cutting the storage requirement down to one-fourth. That is how the 14GB compressed weight size was achieved. The router layers retain 8-bit precision to maintain routing accuracy, while the main expert weights are stored in 4-bit format, preserving acceptable output quality.
2. SSD Streaming: RAM Holds Only Shared State
This is the central engineering design of the project.
TurboFieldfare keeps only the 1.35GB shared component (shared layers and KV cache used by all experts) in system memory. The weights for all 128 experts remain on the SSD. For each token generation cycle, it reads only the 8 required experts directly from the SSD.
However, a major physical bottleneck exists: SSD latency is in milliseconds, while RAM latency is in nanoseconds—a speed difference of tens of thousands of times. If every read simply blocks and waits for the SSD, generation speed becomes intolerably slow.
3. Smart Caching + Time Overlapping
The author introduced three layers of optimization to bridge this speed gap:
Expert Caching. Although 8 experts are required per step, consecutive tokens often reuse the same experts. TurboFieldfare maintains 16 cache slots per layer and uses an LFU (Least Frequently Used) algorithm to retain active experts. Hits in the cache bypass SSD reads entirely. Benchmark data showed this cache reduced average expert fetch time from 166 ms/token down to 88 ms/token.
Parallel Pre-reading. Relying on demand paging (mmap) to let the OS handle page loads sounds elegant, but testing showed disastrous real-world performance—cold starts dropped to 0.5 tok/s. The author switched to explicit concurrent pread calls, actively fetching data blocks in parallel and pushing speed to 3.97 tok/s. This choice was born out of empirical measurement, not theoretical intuition.
Time Overlapping. While the SSD reads expert data, the GPU is not sitting idle—it computes the model’s shared layers. By the time shared computations finish, the SSD data for the experts has arrived. This precise pipelining hides nearly all read latency. Coarse-grained overlapping (batching reads before computation) proved more stable and performant than fine-grained per-expert processing.
103 Experiments, Most of Them Failures
Engineering differs from academic papers. Papers highlight the successful path; engineering requires walking through all the dead ends.
TurboFieldfare’s documentation candidly details results from 103 experiments, outlining ideas that seemed brilliant on paper but failed in practice:
Memory Mapping (mmap) seemed elegant. Letting the OS handle page loading automatically required minimal code. Real test result: 0.5 tok/s on cold start, practically unusable.
SIMD Cooperative Kernels. Having multiple threads collaborate on a single expert aimed for cleaner code structure. Real result: GPU compute time doubled from 230 ms to 527 ms. Scrapped.
Cross-layer Expert Lookahead. If Layer N selects experts A and B, can we speculatively prefetch for Layer N+1? Analysis revealed expert selection across adjacent layers had almost zero correlation—prediction accuracy was a mere 7%.
Fine-grained Asynchronous Execution. Processing each expert immediately as its data arrived introduced synchronization overhead that slowed down total throughput and altered numerical output. A simpler, coarse-grained approach won out.
These failed attempts are as valuable as the success. They highlight a simple truth: The only optimization that matters is the one validated by empirical measurements.
Why This Matters
TurboFieldfare proves an essential point:
LLM inference does not strictly require expensive, high-end GPUs.
Through extreme software engineering—SSD streaming, intelligent caching, and hardware-aware kernel design—everyday consumer laptops can serve as capable AI inference nodes.
Currently, top-tier GPUs are monopolized by a few giants, priced exorbitantly, and facing severe shortages. Projects like TurboFieldfare demonstrate an alternative path: instead of relying on costlier hardware, rewrite physical constraints through superior software.
This mirrors the PC gaming industry of the 1990s. Back then, 3D graphics demanded professional workstations, until consumer 3D accelerators brought immersive games to standard PCs. TurboFieldfare represents an early step in that same evolution, proving to the tech industry that AI inference need not be locked behind hardware gates.
With over 900 stars on GitHub and an active community providing benchmark data, the creator plans to extend support to iPhone and iPad.
Perhaps in the near future, your smartphone—or even your wireless earbuds—will run a tens-of-billions parameter AI assistant in local memory taking up just a few dozen megabytes. TurboFieldfare is one stepping stone on that journey, but it clearly points the way forward.
References:
- GitHub: TurboFieldfare Repository
- HN Discussion (item?id=49098510)
- Gemma 4 Technical Report
- Maarten Grootendorst’s Visual Guide to Gemma 4
- TurboFieldfare System Architecture Spec
- TurboFieldfare Optimization Experiments Log (103 Experiments)
