Automatic medical image interpretation: State of the art and future directions

• Published in: Pattern recognition Vol. 114; p. 107856
• Main Authors: Ayesha, Hareem, Iqbal, Sajid, Tariq, Mehreen, Abrar, Muhammad, Sanaullah, Muhammad, Abbas, Ishaq, Rehman, Amjad, Niazi, Muhammad Farooq Khan, Hussain, Shafiq
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.06.2021
• Subjects: Attention mechanism; Automatic captioning; Convolutional neural network (cnn); Deep learning; Encoder-decoder framework; Image captioning; Long-Short-Term-Memory (LSTM); Medical image caption; Deep learning; Attention mechanism; Automatic captioning; Long-Short-Term-Memory (LSTM); Convolutional neural network (cnn); Image captioning; Medical image caption; Encoder-decoder framework
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2021.107856

•Image interpretation is an emerging field of artificial intelligence.•A good amount of research has been published with different titles that may include caption generation, image interpretation, video captioning, deep captioning.•For medical image analysis and interpretation, the work is little at present and need attention of researchers to produce high performance algorithms in order to apply these methods in clinical practices.•This work reviews recent advances in describing medical images in natural and medical language.•The work compares and discusses the strengths and short coming of state of the art work and also proposes the dimensions that can be explored for future work. Automatic Natural language interpretation of medical images is an emerging field of Artificial Intelligence (AI). The task combines two fields of AI; computer vision and natural language processing. This is a challenging task that goes beyond object detection, segmentation, and classification because it also requires the understanding of the relationship between different objects of an image and the actions performed by these objects as visual representations. Image interpretation is helpful in many tasks like helping visually impaired persons, information retrieval, early childhood learning, producing human like natural interaction between robots, and many more applications. Recently this work fascinated researchers to use the same approach by using more complex biomedical images. It has been applied from generating single sentence captions to multi sentence paragraph descriptions. Medical image captioning can assist and speed up the diagnosis process of medical professionals and generated report can be used for many further tasks. This is a comprehensive review of recent years’ research of medical image captioning published in different international conferences and journals. Their common parameters are extracted to compare their methods, performance, strengths, limitations, and our recommendations are discussed. Further publicly available datasets and evaluation measures used for deep-learning based captioning of medical images are also discussed.