Review finds incomplete methods and harms reporting in medical LLM trials

A systematic review and meta-research analysis published in The BMJ examined reporting completeness in clinical trials of large language model (LLM) interventions in medicine. Researchers searched bibliographic databases and trial registries for records published from January 2017 to December 2025. The analysis included 68 trial reports, 15 protocols, and 378 registrations, assessed against reporting guidelines matched to study design and purpose. Eligible trials prospectively assigned human participants to interventions and evaluated health-related biomedical or behavioural outcomes.

Trial reports were predominantly small, single-site randomised studies published in 2024–2025, mainly evaluating patient-facing education and management recommendations. Mean reporting completeness for trial reports was 40% under TRIPOD-LLM, 44% under CONSORT 2025, 59% under CONSORT-AI, and 52% under CHART. Protocol completeness averaged 51% under SPIRIT 2025 and 57% under SPIRIT-AI; registration completeness averaged 61% against the WHO dataset.

Frequently missing details included model versions, access methods, inference parameters, input and output processing, and access to protocols, data, and code. Harms planning, monitoring, and reporting were also often incomplete. These gaps limit replication, risk-of-bias assessment, and clinicians’ ability to appraise safety before implementation. The findings concern reporting quality rather than directly demonstrating whether LLM interventions are effective or safe; the supplied abstract does not provide intervention-level outcomes.