
Share
Healthcare AI vendors love to tout exam scores, but those numbers often hide stale data, contaminated test sets, and untested real-world judgment. Here's what buyers should actually be asking before they trust a model with patient care.
Imagine hiring a surgeon based solely on their medical school entrance exam score. You'd want to know more: how they handle a complication at 2 a.m., whether they can navigate a messy chart, whether they recover gracefully when something goes wrong. Yet that's roughly the level of assurance many hospitals get when an AI vendor leads a pitch with "our model passed a medical licensing exam."
It sounds impressive. It tells you almost nothing about whether that system can safely do the actual work of clinical care.
For health systems to adopt AI responsibly, and at scale, they need evaluation that's open to scrutiny, reproducible by outsiders, and updated continuously as models change. That kind of testing infrastructure is harder to build in healthcare than in most other AI applications, for three interlocking reasons. Patient data is locked down for good reason. Public benchmarks go stale fast. And AI agents are now doing far more than answering questions, which means there's far more to test.
A benchmark score is only as trustworthy as the method behind it, and the method depends on access to real clinical data that privacy law was designed to protect.
Take MedHELM, a clinician-validated framework covering 121 clinical tasks across five categories, built with a public methodology. It's held up as a strong example of rigorous benchmarking. It's also a clear illustration of the underlying problem. Of the 37 datasets that feed into MedHELM, 14 are private, held inside the institutions that created them. Seven more are gated. Outside researchers can only partially verify the published rankings, because full replication would require access to restricted patient records that most of them will never get.
Even the best available dataset has real limits. MIMIC-IV underpins a huge share of academic healthcare AI research, but using it requires individual credentialing, ethics training, and a data use agreement that bars redistribution and blocks sending the data to third-party commercial APIs. The dataset itself only covers 2008 to 2019, draws from a single Boston academic medical center, and focuses almost entirely on emergency and intensive care. Outpatient clinics, community hospitals, rural settings, regional diversity: largely absent.
These restrictions exist to protect patient confidentiality under HIPAA, and they should stay in place. But they also mean clinical benchmarks can't become the kind of turnkey, open-source test suites that a hospital IT team could simply download and run before deploying a new model. The main workaround right now is synthetic data, built and validated to match real records in difficulty, score distribution, and how models rank against each other. Getting that right takes sustained investment, not a one-time engineering sprint.
Then there's the contamination problem, which cuts the opposite direction. Frontier AI models train on enormous amounts of public internet data. Once a benchmark test set gets published, there's a real risk it ends up inside a future model's training data. A newer model might then score higher not because it reasons better, but because it has effectively seen the answer key in advance.

A study in Nature evaluating BiasMedQA, a benchmark of 1,273 clinical vignettes designed to test how leading prompts sway a model's medical judgment, put a number on that gap. A frontier model scored between 80% and 89% on publicly available prompts. On unpublished prompts built using the same underlying logic, accuracy dropped as low as 47%. That's nearly a 40 point swing, driven by what the model had already seen rather than what it could actually figure out.
Benchmark builders are stuck with a genuine trade-off. Private test sets resist contamination but can't be independently checked. Open test sets stay transparent but need constant refreshing and ongoing funding to stay useful. Neither option is free, and neither fully solves the problem on its own.
Medicine is also asking more of these systems than it used to. AI is moving from chatbots that answer clinical questions to agents that review charts, query electronic health records, place orders, and navigate payer portals on their own. Evaluating that kind of system means judging an entire chain of decisions, not a single response. Did the agent form a sound clinical plan? Did it pick the right tools and use them correctly? Did it recognize and recover from its own errors? Did it finish the task safely, and legally?
New benchmarks are starting to catch up to this shift. PhysicianBench, a Stanford-led benchmark published in May 2026, places AI agents inside a simulated EHR environment with real patient records and grades them on 100 long-horizon tasks adapted from actual consultations between primary care physicians and specialists, spanning 21 specialties. The best-performing model succeeded on only 46% of tasks on its first attempt. Open-source models topped out at just 19%.
HealthAdminBench, another Stanford effort, targets the administrative side of medicine, a category that accounts for over $1 trillion in annual U.S. spending. It simulates an EHR, two payer portals, and a fax system across 135 tasks covering things like prior authorization and appeals and denials management. Agents handled individual steps reasonably well, with the top subtask success rate hitting 82.8%. But the best-performing agent completed only 36.3% of full tasks end to end. That gap between isolated steps and full workflows is exactly where reliability tends to collapse.
Interactive environments like these are much harder for a model to simply memorize its way through, which gives agentic benchmarks some natural resistance to contamination. They're also far more expensive to build and host than a static question set, which is part of why they remain rare.
None of this means buyers should ignore exam scores entirely, but they shouldn't stop there either. Evaluation isn't a box to check once during procurement and forget about. Hospitals and health systems should be asking vendors whether a benchmark's data and scenarios actually resemble their own clinical and administrative workflows, when the model was last tested and how leakage is prevented, whether an independent third party could rerun the evaluation and get the same results, and whether the testing measures static recall or actual end-to-end task completion in something resembling real conditions.
No single vendor, university, or hospital system can solve the privacy, contamination, and agent-testing problems alone. Building evaluation infrastructure that's genuinely trustworthy, open enough to verify and current enough to matter, will take health systems, researchers, and AI companies funding and maintaining that work together, on an ongoing basis. Patients are the ones who bear the risk when that work doesn't get done.
Tags
Original Sources
Your AI Vendor's Medical Exam Score Doesn't Mean What You Think It Does - MedCity News
↗ https://medcitynews.com/2026/11/your-ai-vendors-medical-exam-score-doesnt-mean-what-you-think-it-does
About the author
Amara's entry point into AI was an epidemiology role at a London research hospital, where she spent five years studying how digital health tools reached — or conspicuously failed to reach — underserved communities. Watching early algorithmic systems in healthcare quietly entrench existing inequalities, she redirected her career toward the systemic consequences of AI at scale. She covers AI through an unflinching lens: who benefits, who bears the cost, and what evidence actually says versus what the press release claims. Her writing is calm and precise, but she doesn't mistake balance for neutrality.
More from The Steward →This Week's Edition
18 November 2026
29 articles
Related Articles

Merck Taps Isomorphic Labs for AI-Designed Oncology Drug in Latest Pharma-AI Tie-Up
Health & Science · 5 min

FDA Approves First Drug to Treat Brain Symptoms of Type 3 Gaucher Disease
Health & Science · 6 min

Stanford Pairs AI With Brain Science to Tackle Depression, Memory, and How the Mind Develops
Health & Science · 6 min
Related Articles

Merck Taps Isomorphic Labs for AI-Designed Oncology Drug in Latest Pharma-AI Tie-Up
Health & Science · 5 min

FDA Approves First Drug to Treat Brain Symptoms of Type 3 Gaucher Disease
Health & Science · 6 min

Stanford Pairs AI With Brain Science to Tackle Depression, Memory, and How the Mind Develops
Health & Science · 6 min
More Stories
© 2026 Cedar & Bloom. All rights reserved.