Abstract
Background/Aims The overarching goal of the Studying Colorectal Cancer: Effectiveness of Screening Strategies (SuCCESS) project at Group Health (GH) is to develop evidence to inform personalized colorectal cancer (CRC) screening recommendations. Specifically, we aim to study the comparative effectiveness of screening as practiced, evaluate the potential for personalizing screening and surveillance recommendations, and model the long-term comparative effectiveness of screening in a cohort of GH members enrolled between 1993 and 2015. To accomplish these goals, we used Natural Language Processing (NLP) to collect detailed information from colonoscopy reports in GH’s electronic medical record (EMR). Specifically, we extended an existing NLP system to identify whether signs or symptoms related to CRC were reported at the time of colonoscopy.
Methods To prepare the NLP system to process all colonoscopy reports available in the EMR during the study period, we used a development set of 248 documents. The reports were randomly selected from colonoscopies performed in 2011 for which there was a corresponding pathology report on the same day. Trained medical record reviewers created the development set gold standard. The NLP system, developed in GATE (an open source text processing architecture), was an extension of a system created by Harkema et al. that used MetaMap as a resource to process all documents before sending the appropriate reports through a set of colonoscopy extraction rules. Colonoscopy results were consolidated by a post-processing and evaluation tool written in Python by the authors.
Results The system performed admirably on CRC signs and symptoms. Aggregate sensitivity and specificity were 0.885 and 0.980, respectively, and positive predictive value (PPV), or precision, was 0.826, resulting in an F-score of 0.855.
Conclusions In order to execute the SuCCESS project’s ambitious research aims we need high-quality data from tens of thousands of colonoscopy procedures. This information is not captured in a structured way in GH’s EMR, and manual abstraction of this information is not feasible, but our results show that NLP can reliably extract detailed information from the text reports. Future work includes improving the system’s precision, extracting patient and family history, and extracting results from the associated pathology reports.
