Internal Medicine · General Practice · 2 h ago
Language-model system outperforms physicians in sequential diagnosis benchmark
A study in Science evaluated an AI system that iteratively updates differential diagnoses and selects tests. It achieved higher diagnostic accuracy than generalist physicians on two benchmark datasets, though results come from a controlled setting rather than clinical deployment.
- AI system reached 83.2% accuracy on conference cases.
- Accuracy was 80.3% on MIMIC-IV emergency cases.
- Generalist physician results were 36.1% and 46.8%.
- Performance was assessed in controlled benchmarks, not clinical deployment.
Researchers developed an interactive benchmark for sequential diagnosis, in which a system receives a brief clinical presentation, requests additional information or tests, and revises its differential diagnosis. Their AI system simulated a panel of physicians and was designed to select cost-effective tests. The evaluation used New England Journal of Medicine clinicopathological conference cases and emergency department admissions from MIMIC-IV; sample counts were not provided in the supplied abstract.
The system achieved 83.2% diagnostic accuracy on the conference cases and 80.3% on MIMIC-IV emergency department admissions. The reported accuracy for generalist physicians was 36.1% and 46.8%, respectively. The benchmark also assessed workup cost, although the abstract provides no specific cost results.
The findings suggest potential for language models to support iterative diagnostic reasoning, but they do not establish performance or safety in routine care. Results were obtained in a controlled benchmark setting, and the supplied abstract does not detail case counts, evaluation methods, or how costs compared. Clinical usefulness therefore remains to be tested in real-world settings.
In practice
The study does not support changing diagnostic practice or relying on language models for clinical decisions. Its system achieved higher accuracy than generalist physicians on two controlled benchmark datasets, but the supplied abstract does not provide case counts, detailed cost findings, or evidence from real-world clinical deployment. Clinicians and health systems should regard the results as preliminary evidence of potential decision support, not proof of safety, effectiveness, or lower-cost diagnosis in practice.
AI-generated clinical interpretation · Not editor-reviewed
Is this summary clinically accurate?
Help fellow clinicians: your rating sends inaccurate summaries straight to our editors.
Sign in to rate this summary →Source
Science: Sequential diagnosis with language models ↗This is an automated AI-condensed summary that has not yet been reviewed by an editor. Always consult the full item at the original source.