Vision transformer classifies non-informative frames in surgical videos

Researchers retrospectively analyzed 300,832 frames from 120 minimally invasive surgery videos performed by 30 surgeons. They evaluated an EndoViT-initialized vision transformer using leave-six-surgeons-out cross-validation and compared it with conventional vision models. GPT-4o-mini, LLaVA and Doubao were assessed separately using zero- and few-shot prompts on a balanced 400-frame subset. The study also examined whether surgeon experience was associated with the proportion of non-informative frames.

Among the trained vision models, the EndoViT-initialized transformer achieved aggregate accuracy and an F1-score of 0.942. In the separate exploratory benchmark, GPT-4o-mini achieved accuracy of 0.890 and an F1-score of 0.880 with few-shot prompting. The surgeon-level analysis found no statistically significant association between experience and non-informative-frame ratio (r = −0.421; 95% CI, −0.83 to 0.28; p = 0.226).

The results suggest automated frame triage merits further evaluation for surgical video management, but do not establish clinical readiness. The model assessments used different datasets and sample sizes, and separation between few-shot examples and test frames could not be confirmed. The authors call for matched evaluation and external validation; the study was conducted at a single institution.