Agentic AI improves chart review for immune-related adverse events

Researchers developed an agentic language-model system to identify six immune-related adverse events associated with immune checkpoint inhibitors—myocarditis, dermatitis, thyroiditis, hepatitis, colitis and pneumonitis—and extract event timing, severity, treatment attribution and attribution certainty. Retrospective validation used 289 expert-annotated notes, with 263 in the held-out test set. The study also included prospective silent deployment across 884 notes and a randomized crossover evaluation involving 17 clinical trial staff and 20 notes.

On retrospective testing, macro-averaged F1 was 0.92 for current-event detection and 0.66 for multi-class severity grading; attribution detection scored 0.94 when timing was not considered. During silent deployment, binary F1 ranged from 0.72 to 0.79. With AI assistance, annotation took 2.71 minutes less, complete-match accuracy increased (odds ratio 1.45, 95% CI 1.01–2.08), and interannotator agreement rose from 0.22–0.51 to 0.82–0.85. The best-performing configuration cost approximately US$0.02 per note.

The findings suggest potential to support adverse-event curation for research and pharmacovigilance, but do not establish improved patient outcomes or validate use as a clinical decision tool. Prospective testing was silent, and severity grading was less accurate than event detection; the results support further evaluation rather than a change in clinical practice.