Deep learning in generating radiology reports: A survey

• Published in: Artificial intelligence in medicine Vol. 106; p. 101878
• Main Authors: Monshi, Maram Mahmoud A., Poon, Josiah, Chung, Vera
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.06.2020
• Subjects: Convolutional neural network; Deep learning; Natural language processing; Radiology; Recurrent neural network; Deep learning; Radiology; Natural language processing; Recurrent neural network; Convolutional neural network
• ISSN: 0933-3657; 1873-2860; 1873-2860
• DOI: 10.1016/j.artmed.2020.101878

•Deep Learning algorithms showed promising results in generating radiology reports.•We categorize state of the art models into three levels: word, sentence and paragraph.•We review the publicly available datasets of radiology images and linked reports.•We compare results of the generated reports through quantitative evaluation matrices.•Researchers integrate convolutional neural network and recurrent neural network. Substantial progress has been made towards implementing automated radiology reporting models based on deep learning (DL). This is due to the introduction of large medical text/image datasets. Generating radiology coherent paragraphs that do more than traditional medical image annotation, or single sentence-based description, has been the subject of recent academic attention. This presents a more practical and challenging application and moves towards bridging visual medical features and radiologist text. So far, the most common approach has been to utilize publicly available datasets and develop DL models that integrate convolutional neural networks (CNN) for image analysis alongside recurrent neural networks (RNN) for natural language processing (NLP) and natural language generation (NLG). This is an area of research that we anticipate will grow in the near future. We focus our investigation on the following critical challenges: understanding radiology text/image structures and datasets, applying DL algorithms (mainly CNN and RNN), generating radiology text, and improving existing DL based models and evaluation metrics. Lastly, we include a critical discussion and future research recommendations. This survey will be useful for researchers interested in DL, particularly those interested in applying DL to radiology reporting.