Activities

Background: Large language models (LLMs) have potential for structuring clinical data, yet extracting structured variables from lengthy EHRs, especially in non-English settings, remains challenging. We evaluated the reliability of open-source LLMs for this task using complex oncological EHRs while ensuring data privacy through on-premise deployment.

Methods: We studied a dataset from St. Olavs University Hospital, including 62 patients treated for stage IV colorectal cancer from 2010 to 2020, with EHRs ranging from 180,000 to 1.7 million characters. We manually extracted fifty variables related to patient characteristics and treatments to establish a gold standard, benchmarking extraction strategies with models under 100B parameters.

Results: The sequential approach achieved an F1 score of 0.74 and an accuracy of 0.73, with categorical variables like BRAF mutation status (F1 = 0.983) performing best, while temporal and numerical variables were more challenging. We developed a retrieval framework using MCP search, resulting in roughly 50-fold faster processing and improved accuracy for temporal variables.

Conclusions: Locally deployed LLMs can effectively extract structured oncology variables from complex, non-English EHRs. Improved retrieval architectures demonstrate potential for enhancing speed and performance, with ongoing work aimed at optimizing methods and expanding the dataset.