Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation

• Published in: IEEE transactions on multimedia Vol. 25; pp. 167 - 178
• Main Authors: Yang, Yan, Yu, Jun, Zhang, Jian, Han, Weidong, Jiang, Hanliang, Huang, Qingming
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Coders; Decoding; deep neural network; Diagnosis; Embedding; encoder-decoder framework; Encoders-Decoders; Encoding; Feature extraction; Medical diagnostic imaging; Medical image report generation; Medical imaging; Natural language (computers); Semantics; Sentences; Task analysis; visual-semantic joint embedding; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2021.3122542

Medical image report generation (MeIRG) aims at generating associated diagnosis descriptions with natural language sentences from medical images, which is essential in the computer-aided diagnosis system. Nevertheless, this task remains challenging in that medical images and linguistic expressions should be understood jointly which however show great discrepancies in the modality. To fill this visual-to-semantic gap, we propose a novel framework that follows the encoder-decoder pipeline. Our framework is characterized by encoding both deep visual and semantic embeddings through a triple-branch network (TriNet) during the encoding phase. The visual attention branch captures attended visual embeddings from medical images with the soft-attention mechanism. The medical report (MeRP) embedding branch predicts semantic report embeddings. The embedding branch of medical subject headings (MeSH) obtains semantic embeddings of related medical tags as complementary information. Then, outputs of these branches are fused and fed into a decoder for the report generation. Experimental results on two benchmark datasets have demonstrated the excellent performance of our method. Related codes are available at https://github.com/yangyan22/Medical-Report-Generation-TriNet .