For years, speech technology has operated on an unspoken premise: for a machine to understand your spoken words, your voice must first leave your device. It had to be packaged up and transmitted to servers owned by a tech giant.
Over the past few years, we grew accustomed to this Faustian bargain, sacrificing privacy in exchange for the accuracy of massive cloud models. But on October 2, 2026, the team at Cactus released an open-source speech model named Whistle—demolishing that premise entirely.
Engineered by Jakub Mroz and Henry Ndubuaku, the project delivers speech-to-text in a self-contained 16.9 MB binary container. Running locally on CPU, it executes faster than the 145.3 MB industry benchmark, Whisper base. With this minuscule footprint, Whistle decisively demonstrates that an active network connection is no longer an engineering prerequisite for speech recognition.
Fitting 16.9 MB into a Smartwatch: Breaking the Always-Online Dogma
Speech recognition has long been framed as an inherently heavy cloud service. Mainstream models typically weigh hundreds of megabytes and require complex dependency chains to execute. Whistle takes the opposite route.
Its official product positioning is unmistakable: mobile phones, wearables, robotics, smart home appliances, automotive systems, and microcontrollers. A 16.9 MB file means the model can easily fit onto a smartwatch chip or into the flash storage of wireless earbuds. It processes 16 kHz mono audio in chunks of up to 30 seconds.
The model ships with built-in support for seven languages (including English and German) alongside automatic language identification. The entire workload runs locally in three distinct stages. The first stage is single-pass transcription.
The second stage outputs word-level timestamps alongside probability scores. The third stage extracts speech feature embeddings encoded at 80 ms per frame. Because audio data is digested strictly on-device, the risk of data leakage during transmission is physically eliminated.
Figure: Comparison of Whistle against Whisper base and Moonshine tiny v2 across word error rate, model size, time-to-first-token, and decoding throughput. Source: Cactus official repository.
Sub-12ms Time-to-First-Token Outpaces Cloud Round-Trips
With such a dramatic difference in footprint, raw execution speed becomes the lightweight model’s sharpest weapon against cloud behemoths. Benchmarked on an Apple M4 Pro running purely on CPU without external acceleration, Whistle achieved a time-to-first-token of just 11.1 milliseconds on a 10-second audio clip.
In comparison, Whisper base exhibited a first-token latency of 73.2 milliseconds under identical conditions. Another popular lightweight model, Moonshine tiny v2, took 22.8 milliseconds. In human perceptual terms, 11.1 milliseconds is nearly instantaneous.
Before a user finishes their opening phrase, the first word is already appearing on the screen. In overall decoding throughput, Whistle clocked an impressive 1,319 tokens per second, dwarfing Whisper base (266 tokens/s), which is 8.6 times larger.
In a fully local runtime unencumbered by network round-trips and packet retransmissions, the lightweight model reclaims the upper hand in conversational responsiveness through sheer compute efficiency.
Moving Past Benchmark Fetishism: Winning in Targeted Domains
At 16.9 MB, no model can sweep every synthetic benchmark. Across word error rate (WER)—the primary metric for transcription accuracy—Whistle and Whisper base trade victories. Whistle performs exceptionally well in everyday and commercial scenarios.
On LibriSpeech, Whistle scored 4.31 on the clean split and 10.49 on the other (noisy) split. It also took the lead on SPGISpeech in financial audio (7.65) and the multilingual FLEURS benchmark (21.4). Meanwhile, Whisper base countered on conversational meeting audio in AMI (26.07), lecture speech in TED-LIUM (7.61), and the multilingual MLS dataset (24.9).
For targeted, single-purpose deployments that do not require universal generalization across every global dialect, compact edge models have reached the operational frontier. Rather than pursuing sweeping dominance on academic leaderboards, Whistle focuses on solving practical problems: not a universal cure for every possible audio stream, but a razor-sharp single-purpose scalpel for specific devices.
Squeezing Out Performance: Ruthless Engineering Simplification
Extracting high performance from a severely constrained footprint requires relentless engineering discipline. Rather than reinventing the wheel, the Cactus team directly repurposed the underlying runtime of Needle, their companion text model. For speech tasks, they simply added a gated cross-attention mechanism into each decoder layer.
When audio input arrives, the model projects the required matrices in a single pass and reuses them across the entire decoding sequence. A five-way parallel beam search consumes only five short text cache buffers, completely bypassing the need to repeatedly recompute heavy audio representations five separate times.
For silent segments, a voice activity detection module immediately returns an empty result, preventing blank audio from entering the expensive decoding pipeline. To boost accuracy for domain-specific proper nouns, the engine applies classical finite-state automata for keyword biasing. Developers can deliberately weight obscure names and terms without forcing the underlying neural network to memorize them.
From Raw Audio to Tool Calls Within a Single Binary
On the deployment front, Whistle has been precompiled for 17 target platforms. These range from desktop environments like macOS and Linux to mobile platforms including Android and iOS, and even WebAssembly (WASI) components inside web browsers.
The engine runtime is remarkably clean, strictly avoiding any reads from system environment variables during execution. All runtime behaviors are governed either by hardcoded defaults at compile time or by explicit parameters provided by the caller. Furthermore, because Whistle shares the same underlying quantization format and C++ engine as Needle, the two integrate seamlessly.
With a single binary, loading a text configuration outputs standard transcripts. Swapping in a configuration equipped with tool definitions bypasses intermediate text entirely, translating raw speech straight into structured executable commands.
Figure: Deployment architecture where a single binary executes distinct tasks based on loaded configuration. Source: Cactus official repository.
Real-World Problem Solving Over Synthetic Benchmarks
On Hacker News, community attention quickly moved past fractional benchmark differences toward practical utility. In one comment, a developer shared the real-world dilemma faced by his 84-year-old father, who struggled with slurred speech following a stroke.
The developer built an offline dictation setup for his father. The elderly man could dictate an entire page of his autobiography in ten minutes, saving days of agonizing manual typing. However, general-purpose speech models faithfully captured every vocal hesitation, cough, and verbal correction.
When the father dictated “an elephant no a monkey climbed up a tree,” standard models transcribed every single word. What his father truly needed was a specialized dictation engine capable of filtering out self-corrections on the fly and outputting the intended outcome directly.
For long-form dictation, users do not need an omnipotent cloud service trying to achieve perfection everywhere. Pairing a tiny local engine for instantaneous capture with a low-cost model for downstream cleanup is far more practical than forcing users to repeatedly re-record their thoughts.
Whistle provides concrete proof that an AI model small enough to fit inside a smartwatch can make local processing viable again. Instead of participating in the cloud’s arms race toward hundreds of billions of parameters, it pushes efficiency to its absolute limit in disconnected corners.
When a disconnected legacy watch can understand spoken commands in 11 milliseconds, cloud-tethered voice processing ceases to be an inevitable technical necessity.
Reference Links:
- Cactus Official Release and Benchmarks
- Hacker News Community Discussion
