Internal Medicine · Neurology ·
Delirium Risk Model Maintains Performance Across Two Hospitals
A quality improvement study evaluated live deployment of a machine-learning delirium risk model in 33,095 non-ICU admissions among adults aged 60 years or older at two hospitals. Discrimination remained high at both sites, while deployment at the external-validation hospital was associated with a higher monthly delirium detection rate and a substantial false-positive burden.
- External-site AUROC was 0.91 without model retraining.
- Deployment coincided with higher monthly delirium detection.
- Positive predictive value was 19.4% at the external site.
This quality improvement study assessed a multimodal machine-learning model incorporating natural language processing for delirium risk stratification in patients aged 60 years or older admitted to non-intensive care units. The model was deployed at Mount Sinai Hospital and externally evaluated in live clinical operations at Mount Sinai Morningside Hospital between December 2024 and December 2025. The analysis included 33,095 admissions and examined area under the receiver operating characteristic curve (AUROC) and delirium detection rates before and after deployment.
The model’s AUROC was 0.94 (95% CI, 0.93-0.94) at the development hospital and 0.91 (95% CI, 0.90-0.92) at the external site, without retraining. At Morningside, mean monthly delirium detection increased from 9.23% (95% CI, 8.07%-10.38%) before deployment to 19.59% (95% CI, 17.32%-21.85%) afterward (P<.001). Positive predictive value was 19.4% at Morningside and 25.9% at Mount Sinai.
The findings suggest the model retained discrimination across these two hospitals and deployment coincided with increased case detection. However, the low positive predictive values indicate many alerts were false positives. The quality improvement design and differences between patient populations also limit conclusions about effects on patient outcomes or wider generalizability.
In practice
For hospitals caring for older adults on non-ICU units, this study suggests a multimodal delirium risk model can retain discrimination when deployed at a second hospital without retraining, and its use was associated with increased delirium detection there. The evidence does not establish improved patient outcomes or prove that the model caused the increase. Positive predictive value at the external site was 19.4%, indicating a substantial false-positive burden; the findings come from two hospitals with differing patient populations and may not generalize to other settings.
AI-generated clinical interpretation · Not editor-reviewed
Is this summary clinically accurate?
Help fellow clinicians: your rating sends inaccurate summaries straight to our editors.
Sign in to rate this summary →Source
JAMA Network Open: External Validation of a Machine Learning Model for Delirium Risk ↗This is an automated AI-condensed summary that has not yet been reviewed by an editor. Always consult the full item at the original source.