Microsoft says its medical AI system can diagnose difficult cases more than twice as accurately as practicing doctors — and this time, the claim has survived peer review.
The Microsoft AI Diagnostic Orchestrator (MAI-DxO) "substantially outperformed practicing generalist physicians" in diagnostic accuracy while ordering fewer tests and procedures, according to a study titled "Sequential Diagnosis with Language Models." The work first appeared on arXiv last year as a non-peer-reviewed preprint. On Thursday, the findings were formally published in the journal Science, a step Microsoft argues moves the system from impressive demo territory toward clinical credibility. For more context on this story, see our ongoing AI news.
In tests built from complex case records published in the New England Journal of Medicine, MAI-DxO achieved 83.2 percent diagnostic accuracy, compared with 36.1 percent for generalist physicians who had access to reference materials. When doctors were denied those resources, their accuracy fell to roughly 22 percent — putting the AI system nearly four times as accurate under those conditions. In a separate test based on emergency department admissions, the AI again outperformed generalist physicians, though by a narrower margin of 80.3 percent to 46.8 percent.
A System That Diagnoses the Way Doctors Do
What distinguishes MAI-DxO from conventional symptom checkers is that it works sequentially. Patients do not hand the model a complete file of symptoms, history, and test results. Instead, the system starts with a chief complaint and must decide which follow-up questions to ask and which tests to order before committing to a diagnosis — much closer to how a real physician works under uncertainty.
The researchers say prior benchmarks failed to evaluate large language models' diagnostic abilities in this sequential fashion, so they built a new evaluation called the Sequential Diagnosis Benchmark, or SDBench.
Under the hood, MAI-DxO "simulates a panel of physicians," according to the paper, continually revising a differential diagnosis and selecting cost-effective tests. Different AI agents take on distinct roles: proposing diagnoses, ordering tests, challenging assumptions, and weighing costs. The researchers suggest this structure helps counter cognitive biases that affect individual doctors, such as anchoring on an initial diagnosis or overlooking alternative explanations.
Built on OpenAI's GPT Models
The system achieved its strongest results running on OpenAI's GPT-5, with additional testing showing further improvements using GPT-5.5. Microsoft also emphasizes the economics: the company says MAI-DxO reaches the right diagnosis at half or even one third of the cost by ordering fewer unnecessary tests and procedures.
That cost claim comes with a caveat that Microsoft itself flags — the savings are modeled on estimated costs in simulated cases, not real-world medical bills. The researchers also acknowledge they would expect the performance gap to narrow if the system were compared against expert specialists rather than generalist physicians.
From Benchmark to Bedside
Publication in a top journal is one thing; treating patients is another. Microsoft's AI chief Mustafa Suleyman told The Verge that any real-world deployment would still require clinical testing — and the company already has plans for it. Microsoft's AI health lead, Dr. Dominic King, said testing will begin at Beth Israel Deaconess Medical Center in Boston within the next couple of months.
That staged approach matters, because the field has seen this movie before. Earlier this year, researchers from Harvard University and Beth Israel Deaconess Medical Center reported that OpenAI's o1-preview reasoning model also beat doctors at diagnostic tasks, feeding a wave of headlines about AI replacing physicians. What those studies could not establish was safety in live clinical settings, where mistakes carry real consequences and the patient population is far messier than curated journal cases.
Microsoft has leaned into ambitious framing, publishing its own announcement under the banner of "building medical superintelligence." Suleyman has positioned diagnostic reasoning as one of the clearest domains where AI can exceed human expert performance in the near term.
Why the Details Matter
The NEJM cases used in the benchmark are, by design, diagnostically difficult — the kind of puzzles that get written up precisely because they stumped someone. Day-to-day primary care involves enormous volumes of routine presentations where generalist accuracy is naturally much higher. A system that doubles accuracy on rare, complex cases is valuable, but it is not the same as doubling accuracy across all of medicine.
There is also the question of accountability. If an AI orchestrator orders a test cascade that misses a fatal condition, liability questions remain unresolved in most jurisdictions. Clinical trials at Beth Israel Deaconess are meant to generate exactly the kind of prospective evidence regulators and hospital systems would need.
Still, the trajectory is hard to ignore. A peer-reviewed result in Science, a defined clinical testing plan, and cost pressure on health systems all point the same direction: diagnostic AI is moving from research demonstrations toward regulated medical practice. The next milestone will be those Boston trial results — the first test of whether benchmark dominance translates into better outcomes for actual patients.
For now, the paper's own framing is the fairest summary: on sequential diagnosis of complex cases, a simulated panel of AI physicians outperformed practicing generalists by a wide margin, at lower estimated cost. Turning that into approved clinical practice is the harder problem Microsoft has now signed up for.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
