An AI Dark Horse Among 183 Top Engineers
In a top-tier GPU kernel optimization competition featuring 183 developers, the top ranks were almost exclusively occupied by principal NVIDIA engineers. However, sankalp, a developer with just one year of GPU programming experience, finished in 12th place. His secret weapon for achieving outstanding results over the 14-day contest was allowing the Codex large language model to autonomously execute over 1,500 “test-analyze-modify-verify” cycles in the background.
The contest challenge was to implement batched square compact Householder QR decomposition. This matrix operation is critical in low-level computing; for instance, training processes for LLM optimization algorithms like Muon and Kimi rely on related techniques.
1,500 Automated Submissions: Putting AI on Cruise Control
To enable Codex to evolve autonomously, sankalp built a comprehensive performance evaluation and profiling framework for the model. Operating on this baseline, Codex made more than 1,500 code submissions over the 14-day contest. The benchmark runtime dropped continuously from an initial torch.geqrf baseline (taking ~419,000 microseconds, with an initial kernel baseline of 108,803 microseconds) down to a final 1,805 microseconds, achieving a 232x performance improvement.
sankalp only needed to check the search direction every 2 to 3 hours using non-disruptive, lightweight prompts, allowing the model to run completely unsupervised overnight. This demonstrates that once large language models possess strong code generation capabilities, pairing them with robust automated evaluation suites enables them to independently handle exceptionally heavy low-level code iteration and long-horizon testing.
Figure: Log of Codex autonomously running benchmark tests and performance profiling in the background. Source: sankalp.bearblog.dev
Bottleneck Shift: Why AI Got Stuck Before 1,800 Microseconds
During the optimization process, Codex demonstrated a clear trajectory of architectural evolution. The code progressed from initial torch.geqrf calls to blocked WY QR algorithms, then to using Triton panels, Cholesky-ORHR replacements, CUDA graph bindings, fused assemblies, split16 panels, fixed-shape specializations, and ultimately evolved into a complex combination of super-panels and custom Cholesky decompositions.
Figure: The 10-step algorithmic structural evolution process reducing kernel runtime from 108,803 microseconds to 1,805 microseconds. Source: sankalp.bearblog.dev
However, as execution time was squeezed into the 3,000 to 1,800 microsecond window, fine-tuning compute instructions alone failed to deliver further breakthroughs. At this point, the core optimization bottleneck shifted away from GPU compute capacity or memory bandwidth to GPU launch overhead and panel overhead.
At this critical juncture, Codex fell into a classic local optima trap. The model made repetitive minor tweaks within the existing code structure, achieving no substantial speedups across dozens of consecutive iterations. When optimization enters the deep waters of low-level scheduling mechanisms, an LLM’s stochastic search mechanism easily gets stuck in local sub-optimal solutions.
Beam of Candidates: Human Navigation at the Crossroads
To break the stalemate, sankalp adjusted his prompting strategy for the AI. He introduced a “beam of candidates” mechanism, instructing the model to maintain 3 to 5 candidate branches from different algorithm families simultaneously. This prevented the AI from prematurely abandoning a potential technical route due to a single compilation error.
Figure: The developer guiding Codex’s algorithmic optimization search direction through interactive dialogue. Source: sankalp.bearblog.dev
Under this guidance, Codex was pushed toward entirely new algorithmic directions, successfully adopting the Cholesky-ORHR architecture and super-panel optimizations. The value of human developer experience lies in enforcing a paradigm shift when the model stagnates in local search, drawing upon domain knowledge. Directions chosen at the algorithmic level proved to be the decisive force in breaking microsecond-level bottlenecks.
Community Echoes: The True Reality Behind Full Automation
Similar automated research experiments have echoed across the technical community. Developer Almondsetat used DeepSeek v4 to run automated optimization loops on video codecs, producing parallel instruction implementations for SSE and AVX within just a few hours and doubling single-core performance.
Another developer, poizan42, relied on Opus 5 to write NEON kernels, achieving real-time 4K HEVC video transcoding on a Raspberry Pi 4. Yet this developer candidly admitted that the entire optimization process still required extensive manual steering.
These practices across different domains reveal a consistent pattern. In the LLM era, so-called “automated research” demonstrates impressive execution efficiency, but remains heavily reliant on humans to define boundaries and dimensions. Without human direction, AI can rarely cross key evolutionary thresholds in algorithms on its own.
The Destination Belongs to Those Holding the Compass
The 232x performance gain demonstrates the immense potential of AI automated research. Codex took on the vast majority of tedious labor, from reading algorithm papers to debugging low-level kernels. However, at every critical decision point of the 14-day marathon, human directional choices remained irreplaceable.
While large language models can now run code benchmarks all night long, they still cannot foresee which algorithmic path will lead to the peak. On the frontier of technical exploration, AI is an indefatigable executor, while humans remain the navigators holding the compass.
Reference Links:
- sankalp BEARBLOG Post
- HN Discussion (item?id=49309549)
