Pediatrics · 15 h ago
Kawasaki disease resistance model showed reduced performance over time
A multicenter retrospective cohort study of 6,443 children with Kawasaki disease found that an IVIG-resistance prediction model’s discrimination and calibration declined across calendar periods. Calendar-year adjustment had one of the most favorable performance profiles among the updating strategies tested, but the study did not assess effects on patient outcomes.
- Cohort included 6,443 children treated from 2006 to 2024.
- Model discrimination and calibration declined in later cohorts.
- Calendar-year adjustment performed favorably among tested updating strategies.
- Patient outcomes from model-guided care were not assessed.
Prediction models can lose accuracy as patient populations and clinical settings change. Researchers evaluated temporal performance of a model for resistance to intravenous immunoglobulin (IVIG) in children with Kawasaki disease, using retrospective data from 6,443 children treated at multiple centers between 2006 and 2024. They developed models using data from 2006–2019 and assessed performance in cohorts from 2020–2022 and 2023–2024. Five algorithms were compared; the random forest model was selected for recalibration and updating analyses. Performance measures included discrimination, calibration, Brier score and decision-curve net benefit.
IVIG resistance occurred in 7.0% of the pre-COVID cohort, 13.9% of the during-COVID cohort and 10.1% of the after-COVID cohort. The random forest model’s area under the receiver operating characteristic curve fell from 0.776 during COVID to 0.629 after COVID; its calibration slope declined from 0.894 to 0.431. Calendar-year-adjusted and three-year rolling-window models each had a mean AUC of 0.804. The calendar-year-adjusted model had the lowest Brier score, an observed-to-expected ratio closest to 1, and slightly higher net benefit across most thresholds.
The findings indicate that model performance may change over time and support monitoring and reassessment when such tools are used. However, this retrospective study compared prediction metrics, not clinical outcomes; it does not establish that using an updated model improves treatment decisions or patient outcomes.
In practice
For clinicians or health systems considering an IVIG-resistance prediction model in children with Kawasaki disease, these results caution against assuming that performance remains stable after development. In this retrospective multicenter cohort, discrimination and calibration declined in later calendar periods, while calendar-year adjustment performed favorably among the strategies tested. The study supports monitoring model performance over time and reassessing models before and during implementation, but it did not test whether model-guided decisions improve treatment selection or patient outcomes. The findings alone do not justify changing care.
AI-generated clinical interpretation · Not editor-reviewed
Is this summary clinically accurate?
Help fellow clinicians: your rating sends inaccurate summaries straight to our editors.
Sign in to rate this summary →Source
BMC Medicine: Temporal drift, calibration monitoring, and adaptive updating of an IVIG resistance prediction model in Kawasaki disease: a multicenter pediatric cohort study ↗This is an automated AI-condensed summary that has not yet been reviewed by an editor. Always consult the full item at the original source.