When AI Explains Itself, Patients and Defendants Still Pay the Price of Blind Trust

Picture a doctor staring at an AI tool that flags a scan as cancerous, or a judge reviewing a risk score that recommends denying bail. In theory, pairing human judgment with AI should produce better outcomes than either acting alone. In practice, something else keeps happening: people accept the machine's answer even when it's wrong.

Researchers call this AI overreliance, and it's not a minor glitch. It shows up in exactly the settings where mistakes carry real weight: medical diagnosis, criminal justice, hiring decisions. For years, the proposed fix has been simple. Make the AI explain itself, and people will catch its errors. But a growing body of research has found that explanations often don't help at all.

"It's a result that doesn't align with our intuition," says Helena Vasconcelos, a Stanford undergraduate studying symbolic systems and lead author of a new study on the problem. Working with Ranjay Krishna of the University of Washington and Stanford HAI affiliates Michael Bernstein and Tobias Gerstenberg, Vasconcelos set out to understand why explainability so often fails to reduce overreliance, and whether that failure was fixable.

Their answer reframes the whole question. It's not that explanations are useless. It's that most explanations demand just as much mental effort to verify as doing the task from scratch, so people skip the verification and just trust the machine.

What the Maze Experiments Revealed

To test this, the team turned to something deceptively simple: mazes. Crowdworkers recruited through the platform Prolific were asked to solve maze puzzles at three difficulty levels, sometimes alone and sometimes with an AI's suggested solution attached. That solution came with varying levels of explanation: none at all, turn-by-turn written directions, a highlighted path image, a highlighted path that stopped abruptly if it hit a dead end, or a path colored red up to the error and blue afterward, visually flagging exactly where the AI went wrong.

The turn-by-turn directions, the most detailed explanation, did nothing to reduce overreliance at any difficulty level. Checking them required just as much mental work as solving the maze independently, so people simply didn't bother. Highlighted images alone fared little better when the maze was easy, since people could solve it unaided anyway and had no need for an explanation at all.

But when researchers showed workers a highlighted route that visibly crossed a barrier, something changed. Overreliance dropped, and it dropped further as the visual cue became more intuitive, with the color-coded version outperforming the plainer highlight. The effect was strongest on the hardest mazes, precisely the situation where good judgment matters most.

The underlying pattern echoes something familiar from everyday life. "If my professor is doing something complicated and then explains it in a complicated way, it's not that helpful," Vasconcelos says. "But if there is an easy-to-understand explanation, I'm more likely to scrutinize it to make sure it makes sense." The same logic applies to machines. An explanation only earns scrutiny if understanding it costs less effort than solving the problem yourself.

The team also tested what happens when the stakes rise. In one experiment, workers chose between completing mazes solo for a set payment or accepting less money in exchange for AI assistance. People valued the AI's help more as tasks got harder, and valued simple explanations over complex ones. In a separate test, researchers offered a cash bonus for correct answers. As that incentive grew, overreliance shrank. "If there's a benefit to be gained, people are going to put more effort toward avoiding mistakes," Vasconcelos says.

That finding matters beyond the lab. Financial bonuses won't translate neatly into hospitals or courtrooms, but Vasconcelos sees them as a stand-in for other motivators: intrinsic satisfaction, gamified feedback, or simply making the review process feel worthwhile. "We can motivate people with other mechanisms, using the same underlying principle of increasing benefit," she says.

Looking back at earlier research that found explanations ineffective, the team spotted a pattern others had missed. In easy tasks, like judging whether a movie review is positive or negative, explanations don't matter because people can just do the task themselves without help. In hard tasks, like a reading comprehension exercise one prior study used, the AI's explanation was often just as complicated as reading the passage directly. Either way, people had no real incentive to engage, so they defaulted to trusting the AI outright.

Why It Matters

This research reframes a debate that has frustrated AI safety advocates for years. The problem was never that explainability is a dead end. It's that most explainability tools weren't built with the actual cognitive math of verification in mind. An explanation that takes as long to check as the original decision offers no real protection, no matter how transparent it looks on paper.

For developers building AI tools meant to support, not replace, human judgment, the implication is concrete: simplicity and proportional effort need to be design priorities, not afterthoughts. Vasconcelos puts it plainly: "Maybe we need to build some," referring to explanation techniques that genuinely demand less work than doing the task manually, since few currently exist.

There's also a policy dimension worth sitting with. As regulators push for AI transparency requirements in high-stakes domains like healthcare and criminal justice, this study suggests that transparency alone isn't a safeguard. A system can be fully explainable and still invite blind trust if the explanation isn't built to be quickly and intuitively checked. Vasconcelos hopes the finding reinvigorates researchers in explainable AI, pointing them toward a clearer goal: when a task is hard, design explanations that make the AI's errors easy to spot, not just explanations that document what the AI did. That distinction, subtle as it sounds, may be the difference between a tool people genuinely scrutinize and one they simply, dangerously, trust.