AM³Net: Adaptive Mutual-Learning-Based Multimodal Data Fusion Network

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 8; pp. 5411 - 5426
• Main Authors: Wang, Jinping, Li, Jun, Shi, Yanli, Lai, Jianhuang, Tan, Xiaojun
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: adaptive mutual-learning; Adaptive systems; and multimodal data classification; Channels; Classification; Convolution; Convolutional neural networks; data fusion; Data integration; Feature extraction; Hyperspectral imaging; Involution networks; Kernel; Laser radar; Learning; Lidar; Modules; Object recognition; Radar data; Spatial data; Synthetic aperture radar; Tensors
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2022.3148257

Multimodal data fusion, e.g., hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion, plays an important role in object recognition and classification tasks. However, existing methods pay little attention to the specificity of HSI spectral channels and the complementarity of HSI and LiDAR spatial information. In addition, the utilized feature extraction modules tend to consider the feature transmission processes among different modalities independently. Therefore, a new data fusion network named AM 3 Net is proposed for multimodal data classification; it includes three parts. First, an involution operator slides over the input HSI's spectral channels, which can independently measure the contribution rate of the spectral channel of each pixel to the spectral feature tensor construction. Furthermore, the spatial information of HSI and LiDAR data is integrated and excavated in an adaptively fused, modality-oriented manner. Second, a spectral-spatial mutual-guided module is designed for the feature collaborative transmission among spectral features and spatial information, which can increase the semantic relatedness connection through adaptive, multiscale, and mutual-learning transmission. Finally, the fused spatial-spectral features are embedded into a classification module to obtain the final results, which determines whether to continue updating the network weights. Experimental evaluations on HSI-LiDAR datasets indicate that AM 3 Net possesses a better feature representation ability than the state-of-the-art methods. Additionally, AM 3 Net still maintains considerable performance when its input is replaced with multispectral and synthetic aperture radar data. The result indicates that the proposed data fusion framework is compatible with diversified data types.