MMW-Carry: Enhancing Carry Object Detection Through Millimeter-Wave Radar-Camera Fusion

• Published in: IEEE sensors journal Vol. 24; no. 9; pp. 15091 - 15100
• Main Authors: Gao, Xiangyu, Luo, Youchen, Alansari, Ali, Sun, Yaping
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Camera; Cameras; carry object; Imaging; Indoor environments; Knowledge management; Millimeter wave communication; Millimeter waves; millimeter-wave (MMWave) radar; multiple observations; Neural networks; object detection; Object recognition; Radar; Radar detection; Radar imaging; Radar tracking; Sensors; tracking; Zooming
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2024.3378571

This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, specifically tailored for non-super-sensitive scenarios. Overall, MMW-Carry achieves significant advancements in two crucial aspects. First, it addresses localization challenges in complex indoor environments caused by multipath reflections, enhancing the system's overall robustness. This is accomplished by the integration of camera-based human detection, tracking, and the radar-camera plane transformation for obtaining subjects' spatial occupancy region, followed by a zooming-in operation on the radar images. Second, the system performance is elevated by leveraging long-term observation of a subject. This is realized through the intelligent fusion of neural network results from multiple different-view radar images of an in-track moving subject and their carried objects, facilitated by a proposed knowledge-transfer module. Our experiment results demonstrate that MMW-Carry detects objects with an average error rate of 25.22% false positives and a 21.71% missing rate (MR) for individuals moving randomly in a large indoor space, carrying the common-in-everyday-life objects, both in open carry or concealed ways. These findings affirm MMW-Carry's potential to extend its capabilities to detect a broader range of objects for diverse applications.