MS 'AI Doctor' Has Twice More Diagnostic Accuracy Than Real Physicians

An artificial intelligence (AI) system made by Microsoft (MS) was more than twice as accurate as real doctors at finding difficult diseases in tests. The AI works a little like a team of doctors discussing a patient. It asks questions, decides which medical tests are needed, and then makes a diagnosis, meaning it tries to find out what disease the patient has. However, much of the improvement in accuracy came from the basic AI models used by the system, including OpenAI's GPT-5.

On October 22, local time, Microsoft AI researchers announced that their system, called the "MS AI Diagnostic Orchestra (MAI-DxO)," correctly diagnosed 83.2% of 304 difficult medical cases. General practitioners, who are doctors who treat many common health problems, correctly diagnosed only 36.1% of the same cases. The researchers first shared these findings in June last year. After other experts checked the research, it was published as an official scientific paper in the international journal Science.

The researchers used medical cases published in the New England Journal of Medicine (NEJM), one of the world's most respected medical journals. General practitioners correctly diagnosed 36.1% of cases when they could use textbooks and the internet for help. Without those resources, they correctly diagnosed only 22.3%.

What makes this study different is that the AI had to work through each case much like a real doctor. In earlier tests of medical AI, the system was usually given both the patient's symptoms and medical test results, then asked to name the disease. In Microsoft's study, the AI was given only the patient's first symptoms. It then had to decide what questions to ask and which medical tests to order.

To do this, the researchers created a new testing method called the "sequential diagnosis benchmark." This means the AI receives medical information one step at a time. When the AI asks for a test, a separate system gives it only the result of that test. Every test also has a price. This allows researchers to measure both how accurate the AI is and how much its tests cost, instead of letting it order as many tests as it wants.

MAI-DxO works by giving one AI model several different doctor roles. Each AI doctor suggests a possible disease and challenges the opinions of the others. Together, they decide whether more tests are needed. This is similar to how several specialist doctors at a hospital might meet to discuss a patient's health problem.

The AI and human doctors also worked differently. Before ordering their first medical test, general practitioners asked an average of 5.6 questions. The AI asked an average of 25.5 questions. This means the AI gathered much more information about the patient before trying to make a diagnosis. Even when the AI gave the wrong diagnosis, it often chose a disease similar to the correct one.

The researchers also tested how well the AI could diagnose emergency room patients. They studied 325 cases involving patients admitted to emergency rooms, using a U.S. medical database. The AI correctly diagnosed 80.3% of these cases. This was about 1.7 times the 46.8% accuracy reported for the comparison group. The results showed that the AI was accurate not only with rare and difficult diseases but also with more common emergency room cases.

However, the AI did not always make medical testing cheaper. In one setup designed to improve accuracy without putting a limit on testing costs, the AI's average testing cost was $5,431 per case, higher than the comparison average of $4,725. But when the researchers limited how much the AI could spend, the average testing cost fell to $1,303 while accuracy stayed at 79.6%. These numbers are not the medical bills that patients actually pay. The researchers calculated them using standard hospital fees in the United States.

Another important finding was that the AI system's results depended heavily on the basic AI model running it. GPT-5 alone correctly diagnosed 78.9% of the difficult cases, even without Microsoft's extra diagnosis system. Adding Microsoft's system increased accuracy by about 4 percentage points, to 83.2%. It also lowered testing costs by about $3,000 per case.

The results were even better when newer AI models were used. With GPT-5.5, which was released in April this year, accuracy on difficult medical cases increased from 72% to nearly 87% when the Microsoft system was added. But for emergency room cases, accuracy stayed at around 85% whether the extra system was used or not. Google's Gemini 2.5 Flash showed a much bigger improvement when the system was added, with accuracy rising from 46.4% to about 69%.

These findings suggest that when a basic AI model is already very good, adding Microsoft's diagnosis system makes a smaller difference to accuracy. However, Microsoft's system can still be useful because it may reduce the cost of medical tests while keeping diagnoses accurate.

Still, this study does not prove that AI can replace human doctors. The researchers compared AI with general practitioners, not specialist doctors. They also did not test important treatment decisions or whether patients would be safe during real medical care. Another limitation is that the AI used only written medical information. It did not examine medical images or pictures of tissue samples taken during biopsies.

Microsoft is now starting to test its system with real patients. The company plans to work with Beth Israel Deaconess Medical Center (BIDMC), a major hospital in Boston. Patients will continue receiving their usual medical care while also talking with the AI system. Researchers will then compare the AI's diagnoses with the decisions made by real doctors. Microsoft AI CEO Mustafa Suleiman said the company plans to work with Mayo Clinic to create an AI model specially designed for medicine and introduce it next year. He also said, "In 2027, we will have a breakthrough like coding in the medical record field." [Silicon Valley correspondent Wonho-seop]