The Database Inside Every App Had a 16-Year-Old Bug Hidden in Plain Sight

In August 2026, Tailscale, a network security software company, published a postmortem revealing that their core database had suffered 19 unexplained corruptions over the preceding six months. Each incident caused brief network outages for a subset of users, taking over an hour to recover each time. After months of painstaking investigation, engineers tracked down the root cause: a race condition hidden inside the world’s most widely deployed database engine for at least 16 years.

Tailscale builds software that connects devices securely into private networks. Even if you haven’t heard of them, this story affects you directly—because the database software at the center of the mystery is running on your smartphone right now.

The Most Ubiquitous Software You’ve Never Noticed

First, the protagonist. SQLite is the most widely deployed database in the world, running on billions of devices. Your mobile apps, web browsers, operating systems, and games all rely on it to store data.

What makes SQLite unique is its embedded design. While enterprise databases resemble bank vaults requiring dedicated guards and separate servers, SQLite is like a tiny, portable safe bundled directly inside each app—no installation, no daemon administration, and zero maintenance. Developers plug it into their software, and it operates reliably for decades.

This effortless reliability made SQLite the invisible foundation of the digital world. Because billions of devices execute billions of queries on it every day, its stability has been tested continuously, driving bugs down to infinitesimal probabilities—so low that even its creators assumed certain core paths were virtually bug-free.

A Flaw Hidden in the Foundation

Beginning in 2022, Tailscale stored all core state in SQLite because it was “boring technology”—in engineering terms, a compliment meaning predictable and bulletproof. From early 2023 through mid-2025, everything ran without a hitch.

Then in August 2025, an automated backup pipeline threw an alert: a database file was corrupt. The team repaired the database and investigated, but found no obvious explanation. Then a second corruption occurred, followed by a third… totaling 19 incidents within six months. In large-scale systems engineering, an event with an astronomically low single-occurrence probability will inevitably become a routine event once scale and transaction counts grow large enough.

Making matters worse, the 19 corruptions shared no common pattern: different servers, different user clusters, different times of day, and varying workload patterns. The engineering team was left with no actionable leads and no way to reproduce the bug in a local test environment.

”Notepad First, Ledger Later”: The WAL Mode

Understanding the resolution requires understanding Write-Ahead Logging (WAL), an optional SQLite concurrency mode.

Imagine managing a general ledger. In default journal mode, every transaction is written directly into the main ledger book. In WAL mode, new entries are quickly noted onto a stack of scratch pads first. Once enough pad pages accumulate, they are consolidated and transcribed into the main ledger. The scratch pad is the WAL file, the ledger is the main database file, and the consolidation process is called a checkpoint.

Figure: New data is written to the WAL file first before being checkpointed back to the main database file. Source: tailscale.com

The advantages of WAL mode are clear: writing to a scratch pad is much faster than mutating the main ledger file, checkpoints can happen asynchronously during idle periods, and readers can access the main ledger without blocking writers. Many performance-conscious applications use WAL mode. Tailscale took this a step further by manually triggering checkpoints at very high frequencies to facilitate frequent database backups—a design decision that unwittingly set the stage for the bug.

Race Condition: A Collision Between Writing and Resetting

The bug’s mechanism comes down to a timing conflict: a write transaction and a checkpoint ran simultaneously, and their order of operations broke down. In software terms, a race condition occurred.

Here is how it happened. A checkpoint process counted 10 dirty pages in the WAL file and began copying them back to the main database file. Halfway through the checkpoint, a concurrent transaction appended a new write and reset the WAL file’s frame counter—giving rise to the bug’s name: “WAL-Reset”. Unaware of the reset, the checkpoint process continued copying data using stale page offsets. As a result, one of the modified data pages was mistakenly treated as already checkpointed when it had actually been skipped.

Figure: Checkpoints copy data pages from the WAL back to the main database file. Source: tailscale.com

That transaction’s data vanished without leaving a trace. Worse, index pages in the main database file still pointed to the missing data block, leading SQLite to declare the entire database corrupted. Tailscale engineers later noticed a glaring inconsistency in their logs: the WAL file contained only 10 pages, yet the checkpoint reported writing 20 pages. The extra 10 pages were a phantom artifact of the stale checkpoint iteration—providing the critical clue needed to crack the case.

How It Stayed Hidden for 16 Years

SQLite developers estimate this race condition had existed in the codebase for at least 16 years. It avoided detection for so long because triggering it required an extraordinarily rare convergence of conditions: specific library version combinations, a write transaction striking at an exact microsecond window during checkpointing, and specific OS filesystem semantics. The probability of all three conditions aligning in standard workloads was, in the words of SQLite’s author, “virtually impossible in ordinary use.”

In fact, to verify their bug fix, SQLite maintainers had to inject synthetic test instrumentation into the core engine to artificially induce the timing collision—something they had never done for any bug before.

Most applications use default checkpointing schedules and never encounter the bug. Tailscale encountered it 19 times because their custom high-frequency checkpointing strategy multiplied the statistical exposure by orders of magnitude, turning an impossible black-swan event into an inevitable occurrence.

How the Detectives Solved It

Tailscale’s postmortem stands out for its analytical rigour.

Initial efforts yielded no progress for months. Code reviews revealed no logic errors, and the lack of reproducible state stalled debugging. At one point, a six-week quiet period without incidents offered a false sense of security—reminding engineers that the absence of failure does not imply the presence of correctness.

To break the stalemate, Tailscale began recording every raw database mutation command to a separate audit log. That audit trail provided the breakthrough. During two subsequent corruptions, replaying the log revealed that a committed transaction simply disappeared from the database file without throwing an error. In database systems, a committed write vanishing silently violates fundamental ACID guarantees.

Tailscale then engaged SQLite’s core development team through a commercial support agreement. Together, they built a custom diagnostic layer named tmstmpvfs shim. SQLite’s architecture consists of three main layers: the upper SQL parser, the middle B-tree engine, and the lower Virtual File System (VFS) interface responsible for raw disk I/O. The shim wrapped the VFS layer, logging every disk read and write like a high-speed camera outside a vault.

Figure: SQLite’s low-level storage interface wrapped with a monitoring layer (shim). Source: tailscale.com

With the VFS shim active, they waited for another corruption. When it occurred, the exact sequence of disk operations was captured and handed to SQLite maintainers, catching the WAL-Reset race condition red-handed.

Simultaneously, software testing firm Antithesis independently reproduced the bug using property-based testing. By generating massive streams of randomized concurrent transactions and checkpoints, Antithesis tested two invariant rules: “committed writes must never be lost” and “databases must never become corrupt.” The automated test harness triggered the failure on unpatched versions and passed on patched builds—demonstrating how machine-generated edge cases can uncover subtle concurrency bugs that evade manual code inspection.

The Patch and a False Alarm

The patch itself required modifying just one location: adding an explicit check inside the checkpoint routine to detect whether another thread had reset the WAL state, aborting the checkpoint safely if so. The fix was officially released in SQLite 3.51.3.

However, the rollout presented one final twist. When Tailscale deployed the updated build, their monitoring dashboards flashed red with corruption alerts across multiple nodes. It turned out to be a false alarm: an older, subtle index anomaly produced by earlier versions was surfaced by an optimization in the new release. The SQLite team promptly yanked that version and released a clean build containing only the WAL-Reset fix, while Tailscale adjusted its storage layer to bypass the secondary anomaly.

To conclusively verify that the fix stopped the real-world bug, Tailscale added production telemetry to log whenever a WAL reset collided with an active checkpoint. After two quiet months, the alarm triggered—confirming that the 16-year-old race condition had indeed struck again in production, but was successfully neutralized by the fix. Over the following four months, zero database corruptions occurred.

The Most Trusted Components Warrant the Deepest Scrutiny

This investigation offers a valuable lesson for software engineering. SQLite is among the most trusted software components in existence, accepted as reliable by default across the technology industry. Yet that implicit trust allowed a critical race condition to remain unnoticed for 16 years across billions of devices. When foundational components fail, every abstraction built on top of them crumbles.

Tailscale’s proactive approach—funding official commercial support, contributing to open-source debugging tooling, and sharing a detailed postmortem—received widespread praise from the engineering community. By investing in diagnostic infrastructure, they ensured that developers worldwide benefit from a safer, more resilient database foundation.

A 16-year-old bug has been laid to rest. But in a world built on complex software stacks, the next hidden edge case may already be lurking inside the core libraries we trust most.

References:

• Tailscale: Postmortem on SQLite WAL-Reset Bug
• Antithesis: Breaking the WAL
• HN Discussion (item?id=49272832)