Karpathy Retires the Pelican Benchmark: Why AI “Graduation” Isn’t Real Mastery

On August 2nd, one of the most famous benchmarks in AI image generation was declared ready for retirement. Andrej Karpathy—former Director of AI at Tesla and co-founder of OpenAI—posted on X: “We are starting to move past the era of testing AI with ‘a pelican riding a bicycle.’” The post generated over 2.74 million views within 24 hours, sparking a debate on Hacker News that climbed to nearly 400 points and over 300 comments.

However, Hacker News commenter Morromist poured cold water on the sentiment: “I haven’t seen any AI reliably draw a pelican riding a bicycle. Even the best ones, if you look closely, always have flaws—both legs on the same side of the bike, or two beaks.”

The benchmark has been declared retired, yet pelicans are still crashing. What is really happening here?

The Story Behind the “Pelican Test”

In late 2024, developer Simon Willison offhandedly created a simple test: prompt an AI image generator to draw “a pelican riding a bicycle.” What started almost as a joke evolved over two years into one of the AI community’s most iconic informal benchmarks—cited in over 130 articles, with virtually every new vision model put through the test upon release.

Why is this prompt so challenging? Breaking it down reveals the spatial puzzle: a pelican (anatomy), a bicycle (mechanical assembly), and “riding it” (spatial relationships). Combining all three pushes image generators to their limits.

AI image models do not understand physics; they operate on pattern recognition. An AI has seen millions of images of birds and millions of images of bikes across the internet. But humanity has produced almost no images of pelicans actually riding bicycles. The model cannot simply copy from training data; it must synthesize unseen components according to spatial rules. Synthesizing novel mechanical interactions is precisely where diffusion and generative models fail: while they excel at mimicking familiar scenes, asking them to assemble unfamiliar parts according to physical constraints exposes their structural weakness. It is like asking someone who has never ridden a bike to draw a cyclist: the person looks right, the bike looks right, but the pedals, chain, and handlebars end up in nonsensical places.

Two Years of Blooper Reels

Over the past two years, the “pelican test” produced a hall of fame of failures: pelicans pedaling with both legs on one side of the bike (a posture that would cause a fall within seconds), birds growing two beaks, wheels without spokes, bike frames hovering in mid-air, and handlebars attached to nothing.

Image: A default mode generation from a leading model. The pelican isn’t riding the bike; it’s floating above it. Source: simonwillison.net

Smaller models yielded near-abstract art: two blue circles, a brown rectangle, and an orange blob passed off as a pelican riding a bike.

Image: An ultra-small model’s response to the same prompt, with barely recognizable birds or bikes. Source: simonwillison.net

Just late last month, when Simon Willison tested the latest models, default mode still rendered hovering pelicans and broken frames. Only when turning up the “extended thinking” or reasoning toggle did models produce plausible images. Commenters noted that Claude’s pelicans showed no dramatic improvement over an 18-month span.

Why Declare “Graduation” Now?

Karpathy’s argument is less about perfect mastery and more about benchmark saturation: the prompt can no longer differentiate between models effectively.

In evaluation design, a test’s value depends on its discriminative power. When every student scores 30%, the test highlights weaknesses; when every student scores 95%, the exam becomes useless for ranking performance. The pelican benchmark reached this saturation point: most top-tier models can now generate a pelican bike image that looks convincing at first glance. Discussions shifted from “which model failed” to “whose pelican looks more aesthetically pleasing”—turning a capability test into a subjective taste debate.

Instead, Karpathy proposed a far more demanding task: feed an AI the opening passage of The Lord of the Rings, allocate 1 million thinking tokens (roughly $10 in compute), and instruct it to program a 3D interactive scene from scratch. After two hours of processing and 5,500 lines of code, the AI constructed an interactive 3D Hobbiton, complete with ambient music featuring Tom Bombadil’s song. Karpathy quipped: “Watch out, ‘GTA Hobbiton’ might ship before GTA VI.”

Image: The same model and prompt under extended thinking mode, showing a dramatic leap from the default attempt above. Source: simonwillison.net

High Scores vs. True Understanding

There is a subtle trap here that is easy to miss: passing an exam is not the same as mastering the skill.

First, there is the “test contamination” effect. After two years of viral pelican tests across the web, training datasets inevitably contained numerous synthetic and curated examples of pelicans riding bikes. A high score might reflect memorization rather than spatial reasoning. This isn’t just speculation: researchers tested 7 leading models across 48 “animal + vehicle” permutations (flamingo on a skateboard, heron steering a boat, etc.) and found that models performed no better on pelicans than on other unscripted combinations. If a model had genuinely acquired physical spatial reasoning, why would it only demonstrate it on the pelican? It suggests the specific prompt was overfitted.

Second, when a public benchmark becomes widespread, “genuine improvement” and “memorizing the answer key” become indistinguishable in benchmark scores. Once a test leaks into the public domain, it devolves from a capability measure into a familiarity index. Karpathy’s pivot is an implicit acknowledgment that the pelican test dataset is contaminated.

Third, fundamental generative flaws persist. Karpathy acknowledged that during 3D generation, the AI could not visually inspect its own creation directly—it relied on taking periodic screenshots, often misinterpreting what it saw. This inability to self-evaluate visual output is the root cause of pelican failures: the AI lacks a feedback loop to realize that “two legs shouldn’t be on the same pedal.”

Implications for Users

The next time you evaluate AI-generated imagery—especially scenes involving complex mechanics, coordinated action, or multi-object interaction—it pays to look closely. For atmospheric artwork, landscapes, and posters, AI generation is mature and impressive. But for structural diagrams, mechanical assemblies, or physical interactions requiring precise spatial awareness, subtle flaws remain hiding in plain sight.

Two years ago, when the pelican benchmark emerged, people marveled at how AI struggled with such a simple concept. Today, as the benchmark is retired, the debate centers on whether AI actually understands what it draws. The true takeaway from the pelican era is clear: benchmark scores can graduate, but fundamental capability requires continuous, rigorous scrutiny. The next proctor is already waiting at the exam door.

References:

• Andrej Karpathy’s post on X (retiring the pelican benchmark & LOTR 3D demo)
• Hacker News discussion (“Karpathy’s Pelican”, ~400 points, 300+ comments)
• Simon Willison: pelican-riding-a-bicycle tag page (130+ posts)
• Simon Willison: Kimi K3, and what else we can learn from the pelican benchmark
• Dylan Castillo: Are AI labs secretly pelicanmaxxing?