GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion

• Published in: IEEE transactions on instrumentation and measurement Vol. 70; pp. 1 - 14
• Main Authors: Ma, Jiayi, Zhang, Hao, Shao, Zhenfeng, Liang, Pengwei, Xu, Han
• Format: Journal Article
• Language: English
• Published: New York IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Computer vision; Data mining; Deep learning; Gallium nitride; generative adversarial network (GAN); Generative adversarial networks; Generators; Image contrast; Image fusion; Image processing; infrared; Infrared imagery; multiclassification; Neural networks; Task analysis; Texture
• ISSN: 0018-9456; 1557-9662
• DOI: 10.1109/TIM.2020.3038013

Visible images contain rich texture information, whereas infrared images have significant contrast. It is advantageous to combine these two kinds of information into a single image so that it not only has good contrast but also contains rich texture details. In general, previous fusion methods cannot achieve this goal well, where the fused results are inclined to either a visible or an infrared image. To address this challenge, a new fusion framework called generative adversarial network with multiclassification constraints (GANMcC) is proposed, which transforms image fusion into a multidistribution simultaneous estimation problem to fuse infrared and visible images in a more reasonable way. We adopt a generative adversarial network with multiclassification to estimate the distributions of visible light and infrared domains at the same time, in which the game of multiclassification discrimination will make the fused result to have these two distributions in a more balanced manner, so as to have significant contrast and rich texture details. In addition, we design a specific content loss to constrain the generator, which introduces the idea of main and auxiliary into the extraction of gradient and intensity information, which will enable the generator to extract more sufficient information from source images in a complementary manner. Extensive experiments demonstrate the advantages of our GANMcC over the state-of-the-art methods in terms of both qualitative effect and quantitative metric. Moreover, our method can achieve good fused results even the visible image is overexposed. Our code is publicly available at https://github.com/jiayi-ma/GANMcC.