Multi-attention bidirectional contrastive learning method for unpaired image-to-image translation

• Published in: PloS one Vol. 19; no. 4; p. e0301580
• Main Authors: Yang, Benchen, Liu, Xuzhao, Li, Yize, Jin, Haibo, Qu, Yetian
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 16.04.2024; Public Library of Science (PLoS)
• Subjects: Biology and Life Sciences; Contrast media; Health aspects; Image transmission; Physical Sciences; Research and Analysis Methods; Social Sciences; China
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0301580

Unpaired image-to-image translation (I2IT) involves establishing an effective mapping between the source and target domains to enable cross-domain image transformation. Previous contrastive learning methods inadequately accounted for the variations in features between two domains and the interrelatedness of elements within the features. Consequently, this can result in challenges encompassing model instability and the blurring of image edge features. To this end, we propose a multi-attention bidirectional contrastive learning method for unpaired I2IT, referred to as MabCUT. We design separate embedding blocks for each domain based on depthwise separable convolutions and train them simultaneously from both the source and target domains. Then we utilize a pixel-level multi-attention extractor to query images from embedding blocks in order to select feature blocks with crucial information, thus preserving essential features from the source domain. To enhance the feature representation capability of the model, we incorporate depthwise separable convolutions for the generator. We conducted comprehensive evaluations using three datasets, demonstrating that our approach enhances the quality of unpaired I2IT while avoiding the issue of mode collapse-related image blurring.