Multi-layer fusion techniques using a CNN for multispectral pedestrian detection

• Published in: IET computer vision Vol. 12; no. 8; pp. 1179 - 1187
• Main Authors: Chen, Yunfan, Xie, Han, Shin, Hyunchul
• Format: Journal Article
• Language: English
• Published: The Institution of Engineering and Technology 01.12.2018; Wiley
• Subjects: adverse illumination; adverse illumination circumstances; CNN; convolutional layers; deep learning; detection miss rate reduction; feature extraction; feature maps; feedforward neural nets; fused ROI pooling layers; image fusion; image matching; KAIST multispectral pedestrian dataset; learning (artificial intelligence); MLF region proposal network; multilayer fused convolution neural network; multilayer fusion techniques; multispectral pedestrian detection; nighttime; object detection; overexposure; pedestrian detectors; pedestrians; Research Article; scale matching; shadows; summation fusion method; thermal information; visible information; scale matching; CNN; image fusion; adverse illumination; object detection; overexposure; MLF region proposal network; nighttime; summation fusion method; deep learning; feature extraction; feature maps; multilayer fusion techniques; thermal information; learning (artificial intelligence); convolutional layers; KAIST multispectral pedestrian dataset; pedestrians; detection miss rate reduction; fused ROI pooling layers; image matching; pedestrian detectors; adverse illumination circumstances; visible information; multispectral pedestrian detection; shadows; multilayer fused convolution neural network; feedforward neural nets
• ISSN: 1751-9632; 1751-9640; 1751-9640
• DOI: 10.1049/iet-cvi.2018.5315

In this study, a novel multi-layer fused convolution neural network (MLF-CNN) is proposed for detecting pedestrians under adverse illumination conditions. Currently, most existing pedestrian detectors are very likely to be stuck under adverse illumination circumstances such as shadows, overexposure, or nighttime. To detect pedestrians under such conditions, the authors apply deep learning for effective fusion of the visible and thermal information in multispectral images. The MLF-CNN consists of a proposal generation stage and a detection stage. In the first stage, they design an MLF region proposal network and propose to use summation fusion method for integration of the two convolutional layers. This combination can detect pedestrians in different scales, even in adverse illumination. Furthermore, instead of extracting features from a single layer, they extract features from three feature maps and match the scale using the fused ROI pooling layers. This new multiple-layer fusion technique can significantly reduce the detection miss rate. Extensive evaluations of several challenging datasets well demonstrate that their approach achieves state-of-the-art performance. For example, their method performs 28.62% better than the baseline method and 11.35% better than the well-known faster R-CNN halfway fusion method in detection accuracy on KAIST multispectral pedestrian dataset.