Your AI Vendor’s Medical Exam Score Doesn’t Mean What You Think It Does
Healthcare AI companies often begin their sales presentations with a familiar claim: their AI model passed a medical licensing exam. This sounds reassuring, but it tells buyers very little about whether the model can safely handle real medical work.
For healthcare systems to use AI widely, they need tests that are open, can be repeated, and are regularly updated. Building these tests is much harder in healthcare than in general AI for three reasons: patient data is restricted, public tests become outdated, and AI agents are taking on more complex tasks that need to be tested.
Why openness matters and where it falls short
A test score is only as reliable as the method used to produce it. MedHELM is a good example of a strong testing system that has been checked by doctors. It covers 121 medical tasks in five groups and uses a public testing method.
But it also shows the main problem. MedHELM uses 37 datasets. Fourteen are private and kept by the institutions that created them. Another seven have limited access. This means people outside those institutions can only partly check the published rankings. To repeat the tests fully, they would need access to restricted patient records.
Even the best available dataset has limits. MIMIC-IV supports much of the academic research into healthcare AI. But people must get individual approval, complete ethics training, and sign an agreement about how they can use the data. The agreement does not allow them to share the data or send it to outside commercial APIs. The dataset also covers only 2008–2019, comes from one academic medical center in Boston, and focuses on emergency and intensive care. It includes very little information about outpatient care, community hospitals, or differences between regions.
These limits help protect patient privacy under HIPAA, and they are necessary. But they also make it difficult to create ready-to-use, open tests that companies can run themselves before using an AI model. One possible solution is to create artificial patient data that has been checked to make sure it matches real records in difficulty, score patterns, and model rankings. Creating and checking this data takes a long-term investment.
The problem of test data becoming familiar
Making a test public creates another problem. Leading AI models are trained on huge amounts of information from the public internet. Once a test is published, its questions may become part of the data used to train future models. Newer models may then get higher scores because they have already seen the questions, not because they are better at reasoning.
A study published in Nature looked at BiasMedQA. This test contains 1,273 medical case examples. It was designed to see how strongly the way a question is written can influence a model's medical judgment. The study showed the problem clearly. A leading AI model scored between 80% and 89% on questions that were publicly available. But when researchers tested it with unpublished questions built in the same way, its accuracy fell as low as 47%.
This creates a difficult choice for people who build tests. Private tests are less likely to be included in AI training data, but outside researchers cannot fully check them. Public tests are easier to inspect, but they need regular updates and ongoing funding.
Moving from answers to actions
Medical AI is moving beyond chatbots to AI agents that can review medical records, search electronic health records (EHRs), place orders, and work through insurance company websites. Testing these systems means checking a whole series of decisions, not just one answer. Did the agent make a sound medical plan? Did it choose and use the right tools? Could it handle mistakes and unusual situations? Did it complete the task safely and legally?
New tests are beginning to cover these abilities. PhysicianBench, a test led by Stanford researchers and published in May 2026, puts AI agents in an EHR system containing real patient records. It tests 100 complex tasks based on real discussions between primary care doctors and specialists in 21 medical fields. Even the best-performing model completed only 46% of the tasks on its first try. Open-source models completed no more than 19%.
HealthAdminBench, another test created by Stanford researchers, focuses on administrative healthcare work. This type of work accounts for more than $1 trillion in spending each year. The test simulates an EHR system, two insurance company websites, and a fax system. It includes 135 tasks, such as getting prior approval for treatment, handling appeals, and managing denied claims. AI agents did well on individual steps. The best success rate for a single task step was 82.8%. But the best agent completed only 36.3% of entire tasks from beginning to end. This difference shows that making AI reliable throughout a complete process is still difficult.
Interactive testing environments are harder for AI models to memorize, so tests of AI agents have some natural protection against training data contamination. However, these environments cost much more to build and operate. Testing should also not be treated as a one-time box to check when buying AI. Buyers should ask vendors these questions:
- Does the test match the work? What data and situations are included in the test? Do they match the medical and administrative tasks our organization actually handles?
- Is the test up to date? When was the model last updated? Which versions were tested? What steps are taken to stop test data from leaking into training data?
- Can the results be repeated? Can we, or an independent outside group, run the test and check the results in a controlled setting?
- Does the test measure real behavior? Does it only check whether the model remembers information, or does it test whether the AI agent can complete full tasks in an interactive environment?
In the end, no single company or university can solve the problems of patient privacy, test data contamination, and testing AI agents on its own. Building a reliable testing system will require healthcare organizations, research institutions, and AI vendors to work together to fund, build, and maintain open tests that can change as healthcare and AI develop.
Photo: J Studios, Getty Images
David Talby, PhD, MBA, is the chief technology officer (CTO) of John Snow Labs. He has spent his career using AI, big data, and data science to solve real-world problems in healthcare, life sciences, and related fields.
This article is part of the MedCity Influencers program. Anyone can share their views on business and innovation in healthcare through MedCity Influencers on MedCity News. Click here to learn how.

