Medical image report generation method and system based on convolution and circulation network

• Main Authors: ZHANG HEYE, QU YAWEI, HAN JUNWEI, XU CHENCHU, HAN LONGFEI, WU YINGJIE, ZHANG DINGWEN
• Format: Patent
• Language: Chinese; English
• Published: 03.02.2023
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATIONTECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING ORPROCESSING OF MEDICAL OR HEALTHCARE DATA; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTEDFOR SPECIFIC APPLICATION FIELDS; PHYSICS

The invention discloses a medical image report generation method and system based on a convolution and circulation network. The method comprises the steps that an extraction tool MESH extracts keywords of all texts as labels to train a classification network; performing multi-label detection on all keywords by using the classification network trained in the previous stage to generate text features; a visual attention network is trained by utilizing visual features, a text attention network is trained by utilizing text features, full utilization of image texts is realized by utilizing subsequent LSTM (recurrent neural network) combination, output of two attention networks in a single-layer LSTM is taken as input, subject term vector output of this time is generated, and the output of the subject term vector is realized. The current output is applied to the two attention networks; and constructing and forming a hierarchical LSTM (Long Short Term Memory) as a final report. According to the invention, the review