After receiving AI advice, people’s accuracy on questions dropped from 27% to 9%. Meanwhile, their confidence surged from 30% to 76%.
These are the real numbers from a peer-reviewed study just published on arXiv[1] — a paper with a telling title: AI advice suppresses people’s willingness to say “I don’t know”, even when the advice is wrong and accuracy is incentivized. Researchers from three French and Italian universities conducted five experiments with 3,132 participants, reaching an unsettling conclusion: AI isn’t just helping you find answers — it’s systematically eroding your ability to recognize what you don’t know.
A Carefully Designed Trap
The researchers didn’t use standard trivia questions. They deliberately chose areas where AI models are most prone to errors — visual details from film scenes. Questions like “What color were the team uniforms in Bend It Like Beckham?” They used the Step 3.5 Flash model, which is largely wrong on these questions.
There was a reason for this design choice.
If the AI gave correct answers most of the time and people performed better, that would just be a study about how useful the tool is. But by deliberately having the AI give wrong answers, the researchers could test whether people would still accept them without question — revealing a problem in human judgment itself, not AI capability.
The results were stark.
Without AI assistance, 44% of participants would say “I don’t know” when uncertain. Once AI advice appeared on screen — even just displayed, not actively requested — that number plummeted to 3%. Accuracy fell from 27% to 9%. Confidence rose from 30% to 76%.
As Valerio Capraro, associate professor at the University of Milan-Bicocca and one of the study’s authors, put it: “People performed terribly — their accuracy dropped to a third, yet their confidence doubled.”
In other words, AI makes you wrong more often, and more certain of your wrongness.
Even Money Didn’t Help
The researchers also tested whether financial incentives — rewarding correct answers and penalizing wrong ones — could make people more rational.
It helped a little: those willing to say “I don’t know” rose from 3% to 8%, and accuracy from 9% to 16%. But both numbers remained far below the 44% and 27% seen without AI.
In other words, even with real money on the line, people struggle to resist AI’s halo effect. This finding is particularly relevant for the workplace, where many companies are already using AI for critical decisions — and financial incentives don’t effectively prevent blind acceptance of bad AI advice.
Cognitive Surrender: A Spreading Phenomenon
This isn’t the first study to identify this problem. Earlier this year, Wharton researchers Steven D. Shaw and Gideon Nave introduced a precise concept — Cognitive Surrender.
Their experiments found that people directly accept AI-provided answers 73% to 80% of the time — even when those answers are wrong. More strikingly, those who cognitively surrender are more confident than those who think independently. They are “confidently wrong.”
The Wharton researchers describe three cognitive layers: the intuitive system (fast thinking), the rational system (slow thinking), and a third layer specific to AI interaction — cognitive surrender. At this level, people neither follow intuition nor engage in reasoning. They simply adopt the AI’s output as their own judgment.
It’s a profound cognitive transfer: you don’t even realize you’ve stopped thinking.
Where Does the Bias Lie?
So is this a flaw in AI itself, or in how we use it?
The optimist’s view: this is only because current AI isn’t good enough. As model accuracy improves, human accuracy will naturally follow. And if people actively remind themselves that “AI might be wrong,” they can avoid blind acceptance.
There’s truth to this — the AI in the experiment did give wrong answers. But the key insight is that AI’s mere presence changes people’s cognitive state.
Without AI, people engage their own knowledge, experience, and logic to make judgments. Even when wrong, they are “wrong after thinking.” With AI, people bypass the thinking process entirely. Worse, they don’t even realize they’ve bypassed it — because they feel so confident in their answers.
The study also reveals a more subtle mechanism: even when AI advice is simply displayed alongside rather than actively sought, people subconsciously abandon their own judgment. This points to a core design logic of modern technology: AI products are designed to always have an answer, not to sometimes say I don’t know.
Google’s AI search has replaced link results with confident AI summaries — Common Sense Media this week classified it as an “unacceptable risk” for students. ChatGPT, Claude, Copilot — none of them say “I’m not sure.”
When “always right” becomes a product’s default posture, users are unconsciously trained into a pattern of never questioning.
The Children Factor
Capraro singled out one group — children.
“For children, whose critical thinking abilities are still developing, the consequences of exposure to these systems could be even more profound,” he said.
Think about it — a child who is still learning how to learn, who grows up accustomed to having AI provide answers from the start. When will they learn to recognize the moment of “I don’t know”? And knowing what you don’t know is precisely the starting point of intellectual growth.
What Can We Do?
Before going further, a clarification: the point of this article is not to tell you to stop using AI. AI is a powerful tool, like calculators or search engines.
But the difference is that calculators don’t make decisions for you, and search engines don’t tell you “this is enough.” Today’s AI delivers final answers, skipping every intermediate step of thought.
The researchers’ recommendations are simple in concept but hard in practice:
First, establish a discipline of thinking first. Before asking AI, try to answer the question yourself. Even getting 50% of the way there is more valuable than jumping straight to the answer.
Second, treat AI as a challenger, not an answer machine. Actively look for where AI might be wrong, rather than validating how right it is.
Third, be suspicious of AI’s confidence. The more certain an AI sounds, the more skeptical you should be. There is no inherent connection between a model’s fluency and its factual accuracy.
Fourth, protect children’s independent thinking. Limit AI tool use until their critical thinking abilities are developed, or always use it under adult guidance.
A Deeper Question
Wharton researcher Shaw wrote in an article: “The most dangerous thing about cognitive surrender is that AI replaces thoughts you were about to have but hadn’t yet articulated.”
When AI makes a decision for you in that moment, you lose the opportunity to discover that you were wrong — and that might be more dangerous than missing a correct answer.
And that may be the last line of defense for independent thought.
References:
[1] Marcoccia, Quattrociocchi, Capraro. “AI advice suppresses people’s willingness to say ‘I don’t know’, even when the advice is wrong and accuracy is incentivized.” arXiv:2607.13562, July 2026.
[2] Shaw & Nave. “Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender.” Wharton School, University of Pennsylvania, February 2026.
[3] TNW report: AI advice made people three times less accurate but twice as confident, researchers found
[4] The Register report: Using AI makes people less likely to admit they don’t know something
[5] Hacker News discussion: AI advice made people 3x less accurate but 2x confident, researchers found
