Locating X-ray coronary angiogram keyframes via long short-term spatiotemporal attention with image-to-patch contrastive learning

• Published in: IEEE transactions on medical imaging Vol. 43; no. 1; p. 1
• Main Authors: Zhang, Ruipeng, Qin, Binjie, Zhao, Jun, Zhu, Yueqi, Lv, Yisong, Ding, Song
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Angiography; Cardiovascular Diseases; Contrast agents; Contrast media; contrastive learning; Coronary Angiography; Feature extraction; Humans; Image contrast; Image segmentation; Learning; Location awareness; Long short-term memory; Medical imaging; Modules; Proposals; Radiography; Source code; spatiotemporal attention; Spatiotemporal phenomena; temporal action localization; Transformer; Transformers; Video sequences; X-ray coronary angiography; X-Rays
• ISSN: 0278-0062; 1558-254X
• DOI: 10.1109/TMI.2023.3286859

Locating the start, apex and end keyframes of moving contrast agents for keyframe counting in X-ray coronary angiography (XCA) is very important for the diagnosis and treatment of cardiovascular diseases. To locate these keyframes from the class-imbalanced and boundary-agnostic foreground vessel actions that overlap complex backgrounds, we propose long short-term spatiotemporal attention by integrating a convolutional long short-term memory (CLSTM) network into a multiscale Transformer to learn the segment-and sequence-level dependencies in the consecutive-frame-based deep features. Image-to-patch contrastive learning is further embedded between the CLSTM-based long-term spatiotemporal attention and Transformer-based short-term attention modules. The imagewise contrastive module reuses the long-term attention to contrast image-level foreground/background of XCA sequence, while patchwise contrastive projection selects the random patches of backgrounds as convolution kernels to project foreground/background frames into different latent spaces. A new XCA video dataset is collected to evaluate the proposed method. The experimental results show that the proposed method achieves a mAP (mean average precision) of 72.45% and a F-score of 0.8296, considerably outperforming the state-of-the-art methods. The source code and dataset are available at https://github.com/Binjie-Qin/STA-IPCon.