After a Decade of Parquet Dominance, Can WASM Pry It Open?
You’ve got a Parquet table that’s been sitting around for eight years. Not huge, not tiny — about 300 GB. One day you get a new requirement: point-query this table — not a full scan, just pick a few dozen rows by primary key. You try it. You find that Parquet can do it, kind of, but every time it has to dig through a row group’s column chunk to find the target page, and page granularity is hundreds of thousands of rows. The I/O overhead doesn’t match the requirement at all.
You think to yourself: columnar storage formats have been around for over a decade — how can they still not handle decent random access?
This is exactly the starting point for CMU’s database group publishing F3 (Future File Format) at SIGMOD. And it’s also exactly the starting point that steps on the most sensitive nerve in the format wars.
A New Format, an Old Problem
The problem F3 aims to solve can be summed up in one sentence: existing columnar storage formats (Parquet, ORC) were born in the Hadoop era; their storage layout and evolution mechanisms no longer fit today’s hardware and workloads. Parquet’s row groups are coarse-grained, its metadata hierarchy is flat, and column encodings are baked into the spec — for a new encoding to land, every reader implementation has to update. And a data point cited in the F3 paper is particularly telling: the most widely used Parquet version today is still v1, from 2013.
Parquet couldn’t even replace Parquet itself.
F3’s approach is a two-pronged attack. At the layout level, it introduces a more fine-grained hierarchy: IOUnit (basic I/O unit) → EncUnit (encoding unit, default 64K rows) → optional sub-EncUnit vectors. This hierarchy lets readers do finer-grained projection at read time — want just a few thousand rows from a single column? Traverse the EncUnit index, skip irrelevant blocks, done.
At the extensibility level, F3’s core creative move is embedding decoders as WASM binaries inside the file itself. Each EncUnit can be tagged with a WASM ID pointing to a decoder stored at the end of the file. If a reader doesn’t natively recognize that encoding, it just loads the WASM module and decodes — no reader version upgrade needed, no waiting for community consensus. The paper claims WASM decoder sizes are in the kilobyte range, a “negligible storage cost.”
These are F3’s two cards: finer-grained random access, and using WASM to break the compatibility deadlock.
What’s Behind the Numbers
F3’s benchmarks compare against Parquet, ORC, Vortex, Lance, and Nimble. A few trends emerge from the paper’s experiments:
- Random access: F3’s point-query latency is significantly lower than Parquet’s, especially in scenarios requiring only a few columns and a small number of rows. This isn’t magic — the EncUnit hierarchy naturally supports smaller I/O granularity.
- Compression ratio and decompression speed: Roughly on par with Parquet. F3 defaults to Cascade encoding (similar to Vortex’s default) per 64K-row EncUnit group, combined with Zstd/LZ4 compression. Didn’t win, but didn’t lose either.
- WASM decoding overhead: WASM-based decoding is a notch slower than native decoding, but the paper argues the gap is acceptable. An engineering judgment is needed here: the WASM decoder’s purpose is to guarantee “the file is readable.” It’s a fallback, not an accelerator.
Taken together, F3’s benchmarks present a posture of “improved in some dimensions, doesn’t lose to Parquet overall.” For a SIGMOD paper, this result is adequate. For a format-replacement war, it’s not enough.
Compatibility: The Real Moat
The top-voted HN comment came from vouwfietsman, and it’s brutal but hard to refute:
Parquet is unfortunately very good just by virtue of being first, and so widely supported.
How entrenched is Parquet’s ecosystem position? A few facts make it clear: Spark, DuckDB, Pandas, Polars, Snowflake, BigQuery, Redshift Spectrum, AWS Athena, Trino, Presto, ClickHouse (external tables) — virtually every data tool with a name natively supports Parquet. Its spec is open, but through the repeated friction of twenty-plus major implementations, it has formed a de facto standard. The Parquet file you generate can be read by any tool — that’s the accumulated product of a decade of community bug fixes and interoperability tuning.
This leads to a paradox: F3 tries to use WASM to solve the compatibility problem of “new encodings can’t be recognized by old readers,” but what really blocks a new format is the ecosystem integration cost.
What does a company need to do to switch to F3?
- Add an F3 reader to every downstream query engine (WASM fallback can only decode EncUnits, it can’t replace a full reader implementation — file header parsing, metadata traversal, predicate pushdown, projection pruning all need native code).
- Support F3 writers in every data pipeline (ETL/ELT).
- Get every data governance tool (catalogs, schema registries, lineage tracking) to parse F3 metadata.
- Ensure external data-sharing partners can read F3.
This isn’t something a WASM decoder can solve. Parquet’s moat is a decade of accumulated ecosystem fabric.
The Tension in the WASM Approach
F3’s WASM design triggered an intense sub-discussion on HN, centered on three layers.
Layer one: security. Embedding executable code inside files — even with a mature WASM sandbox — naturally triggers engineers’ security instincts. Someone drew an analogy to JavaScript in PDFs: the standard designed the capability, but every sane viewer disables it by default. F3’s supporters countered that WASM decoders are pure functions with no I/O capability, and the sandbox can cap instruction count and memory. But data engineering workflows often involve data files from untrusted sources, and allowing arbitrary WASM execution is still an option many security teams won’t accept.
Layer two: performance positioning. vouwfietsman pointed out with surgical precision: the core value of a columnar storage format is trading random access for analytical performance through sequential scans. F3 makes improved random access its main selling point, but random access was never the design goal of columnar formats. If you optimize random access while making full-table scans slower (even just on the WASM decoding path), you’re trading your core strength for a secondary capability.
Layer three: technical coherence. F3’s metadata layer uses Google’s FlatBuffers to serialize schema and file layout information. WASM decoders need to shuttle data back and forth between the host language and WASM memory, and FlatBuffers parsing itself comes with some overhead. Some commenters argued that introducing the combination of a WASM runtime + FlatBuffers serialization/deserialization adds two layers of abstraction overhead to the read path — precisely the kind of thing columnar formats want to keep as lean as possible.
These critiques don’t mean F3’s design is wrong. But they point to a core proposition: F3 is trying to solve a secondary contradiction in format evolution, not the primary one. The primary contradiction is “how to make everyone willing to switch,” not “how to land new encodings.”
Echoes of History
In the HN comments, someone posted xkcd #927 (“Standards”). Someone brought up the fate of OpenDoc — a technically superior file format that ultimately lost to network effects. Others argued there’s no need for such pessimism: if F3 provides value that Parquet can’t in certain niche scenarios (like online feature stores needing frequent random access, or vertical domains requiring custom encodings), it doesn’t need to win the entire market — just carve out its own niche.
I lean toward the view that these two judgments aren’t mutually exclusive. Format-replacement history does overwhelmingly favor the “compatibility first” thesis, but history has never seen a design like “embed decoders inside the file.” WASM has changed the cost structure of cross-platform executable code — a decade ago, embedding a sandboxed execution environment inside a file was unthinkable; today it’s one line of wasmtime::Module::new().
F3 probably won’t replace Parquet, but the WASM-decoder paradigm it proposes could very well be absorbed by Parquet or other formats. The best outcome isn’t replacement — it’s contamination. Let the old format learn your good ideas, while you move on to the next uncharted territory.
Looking at Current Trends
F3 is still a research prototype — the README opens with a declaration that it “should not be used in production,” the GitHub repo has only 4 commits, and the benchmark reproduction scripts aren’t yet complete. It’s a long engineering distance from “can be seriously evaluated as an alternative by engineering teams.”
And from an industry trend perspective, Parquet’s position is nearly unshakable in the short term. The rise of open table formats like Iceberg, Delta Lake, and Hudi has further cemented Parquet at the bottom of the lakehouse architecture — the table format wars are moving upward, and the file format is actually getting more locked in. You’re unlikely to switch Iceberg and switch the underlying file format at the same time — that’s double the migration cost.
But the question F3 raises is valuable. Parquet’s evolution bottleneck is real — v1 dominating the world for a decade without movement is not a healthy state. The WASM-decoder approach, even if it doesn’t ultimately succeed as F3, could succeed as a spec revision of some other format.
In other words: this isn’t Parquet’s funeral, but it might be the first kick of the next-generation columnar storage format.
References: F3 SIGMOD Paper · GitHub Repository · HN Discussion