Study finds AI advice reduces accuracy and raises confidence

Access to AI advice can make people less accurate while dramatically increasing their confidence, according to a study by researchers from universities in France and Italy. Participants were far less likely to acknowledge uncertainty after receiving AI input, even when the model gave unreliable answers.

Ana Maria Constantin reports for The Next Web that the researchers measured a decline in both accuracy and judgment suspension, meaning the willingness to answer “I don’t know.” Without AI advice, participants withheld an answer in 44% of cases and answered correctly 27% of the time. With AI advice, the rate of withheld answers fell to 3%, while accuracy dropped to 9%. Confidence rose from 30% to 76%.

The team, led by Valerio Capraro of the University of Milano-Bicocca, designed questions that language models often answer incorrectly. They included visual details from films, such as the colour of a sports uniform. The researchers used Step 3.5 Flash, a model that was generally wrong on these questions, to rule out the possibility that people were simply delegating tasks to a more capable system.

Some participants who would have been correct without assistance consulted the AI and then selected its incorrect response.

Incentives improve results, but do not close the gap

Financial rewards for correct answers modestly improved performance. Accuracy increased from 9% to 16%, and the share of participants admitting uncertainty rose from 3% to 8%. Both figures remained well below the results achieved without AI.

The findings align with the concept of “cognitive surrender,” previously used by Wharton researchers to describe people accepting false AI outputs while becoming more confident in them. For people using generative AI at work, the study highlights the value of checking outputs against source material and preserving the option to withhold judgment when evidence is weak.