Accurate and Reliable Classification of Unstructured Reports on Their Diagnostic Goal Using BERT Models

• Published in: Diagnostics (Basel) Vol. 13; no. 7; p. 1251
• Main Authors: Rietberg, Max Tigo, Nguyen, Van Bach, Geerdink, Jeroen, Vijlbrief, Onno, Seifert, Christin
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 27.03.2023; MDPI
• Subjects: Artificial intelligence; BERT; Classification; Diagnosis; Health aspects; health informatics; Language; Machine learning; Magnetic resonance imaging; Medical records; Multiple sclerosis; Natural language processing; Neural networks; Patients; Technology application; text classification; Netherlands; Germany; BERT; health informatics; text classification; natural language processing
• ISSN: 2075-4418; 2075-4418
• DOI: 10.3390/diagnostics13071251

Understanding the diagnostic goal of medical reports is valuable information for understanding patient flows. This work focuses on extracting the reason for taking an MRI scan of Multiple Sclerosis (MS) patients using the attached free-form reports: Diagnosis, Progression or Monitoring. We investigate the performance of domain-dependent and general state-of-the-art language models and their alignment with domain expertise. To this end, eXplainable Artificial Intelligence (XAI) techniques are used to acquire insight into the inner workings of the model, which are verified on their trustworthiness. The verified XAI explanations are then compared with explanations from a domain expert, to indirectly determine the reliability of the model. BERTje, a Dutch Bidirectional Encoder Representations from Transformers (BERT) model, outperforms RobBERT and MedRoBERTa.nl in both accuracy and reliability. The latter model (MedRoBERTa.nl) is a domain-specific model, while BERTje is a generic model, showing that domain-specific models are not always superior. Our validation of BERTje in a small prospective study shows promising results for the potential uptake of the model in a practical setting.