Correspondence Between SWIR and MWIR Images Using Augmentation and Preprocessing for Registration

• Published in: TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON) pp. 998 - 1003
• Main Authors: Inoue, Takeru, Kibe, Michiya, Miyamoto, Ryusuke
• Format: Conference Proceeding
• Language: English
• Published: IEEE 31.10.2023
• Subjects: Deep learning; domain gap; Feature extraction; feature matching; Image color analysis; Image registration; Image sensors; infrared dataset; Sensor phenomena and characterization; Training data
• ISSN: 2159-3450
• DOI: 10.1109/TENCON58879.2023.10322404

Multispectral sensors are used to ensure visibility in various applications. However, when multiple sensors are used for capturing images, a misalignment may occur between the images taken by each sensor unless special care is taken. To correct such misalignments, image registration based on feature matching is conducted. However, the features captured by each sensor differ, thereby complicating the registration process. In this study, we develop an approach to overcome these challenges and to improve the registration accuracy between short-wave infrared and mid-wave infrared (SWIR and MWIR, respectively) images. First, we compare and validate SiLK, a detector-based feature matching method, and LoFTR, a detector-free feature matching method. The results clearly demonstrate the superior accuracy of LoFTR. Moreover, SWIR and MWIR images exhibit a characteristic color inversion according to Kirchhoff' s law. Therefore, by inverting the color of a SWIR image and aligning the color tone between image pairs, we can improve the matching accuracy. Furthermore, by diversifying the color tones of the training data through augmentation, we can handle the domain gap between SWIR and MWIR images, thereby further enhancing the matching accuracy.