LLMs extract sarcoma pathology data with high accuracy in pilot study

Researchers retrospectively evaluated an LLM-based workflow using 1,106 surgical pathology notes from patients who underwent bone or soft tissue sarcoma resection at three sites within one institution. They divided notes into training (n=353) and testing (n=753) sets and used prompt engineering and fine-tuning with Llama3-70B-Instruct and Gemma4-31B-Instruct. The workflow extracted diagnosis, greatest tumor dimension, closest margin, grade, necrosis and viability; accuracy, F1 score and specificity were assessed.

In the test set, both models achieved accuracy of at least 0.85 for all six data points and specificity of at least 0.84 when identifying information as “unknown” because it was absent from the note. Gemma had higher sensitivity than Llama for detecting positive margins (0.92 vs 0.78). Each model took less than one second on average to extract each data point from a note.

The findings suggest that LLMs could help scale sarcoma registry abstraction, which is often labor-intensive. This was a pilot evaluation using notes from a single institution, so results do not establish generalizability to other sites, documentation styles or models. The study supports feasibility, not replacement of human review or a change in clinical care.