Drone-Based RGB-Infrared Cross-Modality Vehicle Detection Via Uncertainty-Aware Learning

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 10; pp. 6700 - 6713
• Main Authors: Sun, Yiming, Cao, Bing, Zhu, Pengfei, Hu, Qinghua
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.10.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cross-modality; Datasets; Disaster relief; Drone vehicles; drone-based vehicle detection; Drones; feature fusion; Illumination; Infrared imagery; Learning; Lighting; Object detection; Object recognition; Residential areas; Sun; Surveillance; Traffic management; Uncertainty; uncertainty-aware; Vehicle detection
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2022.3168279

Drone-based vehicle detection aims at detecting vehicle locations and categories in aerial images. It empowers smart city traffic management and disaster relief. Researchers have made a great deal of effort in this area and achieved considerable progress. However, because of the paucity of data under extreme conditions, drone-based vehicle detection remains a challenge when objects are difficult to distinguish, particularly in low-light conditions. To fill this gap, we constructed a large-scale drone-based RGB-infrared vehicle detection dataset called DroneVehicle, which contains 28, 439 RGB-infrared image pairs covering urban roads, residential areas, parking lots, and other scenarios from day to night. Cross-modal images provide complementary information for vehicle detection, but also introduce redundant information. To handle this dilemma, we further propose an uncertainty-aware cross-modality vehicle detection (UA-CMDet) framework to improve detection performance in complex environments. Specifically, we design an uncertainty-aware module using cross-modal intersection over union and illumination estimation to quantify the uncertainty of each object. Our method takes uncertainty as a weight to boost model learning more effectively while reducing bias caused by high-uncertainty objects. For more robust cross-modal integration, we further perform illumination-aware non-maximum suppression during inference. Extensive experiments on our DroneVehicle and two challenging RGB-infrared object detection datasets demonstrated the advanced flexibility and superior performance of UA-CMDet over competing methods. Our code and DroneVehicle will be available: https://github.com/VisDrone/DroneVehicle .