MMW-Carry: Enhancing Carry Object Detection Through Millimeter-Wave Radar-Camera Fusion

This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, s...

Full description

Saved in:
Bibliographic Details
Published inIEEE sensors journal Vol. 24; no. 9; pp. 15091 - 15100
Main Authors Gao, Xiangyu, Luo, Youchen, Alansari, Ali, Sun, Yaping
Format Journal Article
LanguageEnglish
Published New York IEEE 01.05.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Camera
Cameras
carry object
Imaging
Indoor environments
Knowledge management
Millimeter wave communication
Millimeter waves
millimeter-wave (MMWave) radar
multiple observations
Neural networks
object detection
Object recognition
Radar
Radar detection
Radar imaging
Radar tracking
Sensors
tracking
Zooming
Online AccessGet full text

Cover

Abstract This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, specifically tailored for non-super-sensitive scenarios. Overall, MMW-Carry achieves significant advancements in two crucial aspects. First, it addresses localization challenges in complex indoor environments caused by multipath reflections, enhancing the system's overall robustness. This is accomplished by the integration of camera-based human detection, tracking, and the radar-camera plane transformation for obtaining subjects' spatial occupancy region, followed by a zooming-in operation on the radar images. Second, the system performance is elevated by leveraging long-term observation of a subject. This is realized through the intelligent fusion of neural network results from multiple different-view radar images of an in-track moving subject and their carried objects, facilitated by a proposed knowledge-transfer module. Our experiment results demonstrate that MMW-Carry detects objects with an average error rate of 25.22% false positives and a 21.71% missing rate (MR) for individuals moving randomly in a large indoor space, carrying the common-in-everyday-life objects, both in open carry or concealed ways. These findings affirm MMW-Carry's potential to extend its capabilities to detect a broader range of objects for diverse applications.
AbstractList This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, specifically tailored for non-super-sensitive scenarios. Overall, MMW-Carry achieves significant advancements in two crucial aspects. First, it addresses localization challenges in complex indoor environments caused by multipath reflections, enhancing the system’s overall robustness. This is accomplished by the integration of camera-based human detection, tracking, and the radar–camera plane transformation for obtaining subjects’ spatial occupancy region, followed by a zooming-in operation on the radar images. Second, the system performance is elevated by leveraging long-term observation of a subject. This is realized through the intelligent fusion of neural network results from multiple different-view radar images of an in-track moving subject and their carried objects, facilitated by a proposed knowledge-transfer module. Our experiment results demonstrate that MMW-Carry detects objects with an average error rate of 25.22% false positives and a 21.71% missing rate (MR) for individuals moving randomly in a large indoor space, carrying the common-in-everyday-life objects, both in open carry or concealed ways. These findings affirm MMW-Carry’s potential to extend its capabilities to detect a broader range of objects for diverse applications.
Author Luo, Youchen
Gao, Xiangyu
Alansari, Ali
Sun, Yaping
Author_xml – sequence: 1
  givenname: Xiangyu
  orcidid: 0000-0002-7552-6192
  surname: Gao
  fullname: Gao, Xiangyu
  email: xygao@uw.edu
  organization: Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
– sequence: 2
  givenname: Youchen
  orcidid: 0000-0003-0375-5903
  surname: Luo
  fullname: Luo, Youchen
  email: yluo6@uw.edu
  organization: Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
– sequence: 3
  givenname: Ali
  surname: Alansari
  fullname: Alansari, Ali
  organization: Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
– sequence: 4
  givenname: Yaping
  orcidid: 0000-0001-6284-1639
  surname: Sun
  fullname: Sun, Yaping
  email: sunyp@pcl.ac.cn
  organization: Department of Broadband Communication, Peng Cheng Laboratory, Shenzhen, China
BookMark eNp9kE1LwzAYx4NMcJt-AMFDwXNn0iRN6k3m5gubA53MW3jaJFvG1s60Ffbtbd0O4sHT_-Hh_wK_HurkRW4QuiR4QAhObp7fRi-DCEdsQKmQXJAT1CWcy5AIJjvtTXHIqPg4Q72yXGNMEsFFFy2m00U4BO_3t8EoX0GeuXwZ_DyCWbo2WRXcm6oRV-TBfOWLerkKpm6zcdvm7cMFfJngFTT4pmVrPATjumy85-jUwqY0F0fto_fxaD58DCezh6fh3STMooRVYSIZzWIbxQKnXEAsIQULmmqutRXCYgFGJjwCSKW1jHMdMxKlmOnUYppq2kfXh96dLz5rU1ZqXdQ-byYVxSxhSSSFaFzk4Mp8UZbeWLXzbgt-rwhWLT_V8lMtP3Xk12TEn0zmKmg5VB7c5t_k1SHpjDG_lpiQFMf0G4I8f9Y
CODEN ISJEAZ
CitedBy_id crossref_primary_10_1007_s12239_025_00236_6
crossref_primary_10_1109_JSEN_2024_3524441
Cites_doi 10.3390/app11198926
10.1109/TENCON.2018.8650148
10.1109/79.526899
10.1109/MSP.2005.1406480
10.1109/IEEECONF44664.2019.9048939
10.1109/JSEN.2020.3040354
10.1109/ICARCV.2018.8581329
10.1007/BF02278710
10.1109/ICAS49788.2021.9551127
10.1109/ROBOT.2010.5509644
10.1109/TGRS.2010.2053038
10.1109/TVT.2021.3092355
10.1115/1.3662552
10.1109/ACCESS.2021.3059170
10.1016/B978-0-12-411597-2.00014-X
10.1109/JSTSP.2022.3171168
10.1109/ICIP.1999.817171
10.1109/IUCS.2010.5666180
10.1007/978-3-030-58452-8_13
10.1109/JSEN.2018.2879223
10.1109/TAP.1986.1143830
10.1109/JSEN.2020.3036047
10.1109/TMTT.2008.2007081
10.1109/IROS45743.2020.9341164
10.1109/JSEN.2020.3028362
10.5121/ijcseit.2012.2216
10.1109/CVPR.2016.90
10.1109/TPAMI.2009.167
10.1145/3447993.3483258
10.1109/ICPR.2004.1334537
10.1109/JSEN.2023.3295574
10.1109/TIE.2019.2893843
10.1109/TAES.1972.309463
10.1109/MSP.2007.904812
10.1109/iwem.2018.8536660
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID 97E
RIA
RIE
AAYXX
CITATION
7SP
7U5
8FD
L7M
DOI 10.1109/JSEN.2024.3378571
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Electronics & Communications Abstracts
Solid State and Superconductivity Abstracts
Technology Research Database
Advanced Technologies Database with Aerospace
DatabaseTitle CrossRef
Solid State and Superconductivity Abstracts
Technology Research Database
Advanced Technologies Database with Aerospace
Electronics & Communications Abstracts
DatabaseTitleList Solid State and Superconductivity Abstracts
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Geography
Engineering
EISSN 1558-1748
EndPage 15100
ExternalDocumentID 10_1109_JSEN_2024_3378571
10478306
Genre orig-research
GroupedDBID -~X
0R~
29I
4.4
5GY
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFO
ACGFS
ACIWK
AENEX
AGQYO
AHBIQ
AJQPL
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
EBS
F5P
HZ~
IFIPE
IPLJI
JAVBF
LAI
M43
O9-
OCL
P2P
RIA
RIE
RNS
TWZ
AAYXX
CITATION
7SP
7U5
8FD
L7M
ID FETCH-LOGICAL-c294t-9843c6f2670b57a68abafad3d5ddf77f07ae8952aab8ff455d6412b04dbf03bd3
IEDL.DBID RIE
ISSN 1530-437X
IngestDate Mon Jun 30 10:22:30 EDT 2025
Tue Jul 01 04:27:31 EDT 2025
Thu Apr 24 22:50:47 EDT 2025
Wed Aug 27 02:06:29 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 9
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c294t-9843c6f2670b57a68abafad3d5ddf77f07ae8952aab8ff455d6412b04dbf03bd3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0003-0375-5903
0000-0002-7552-6192
0000-0001-6284-1639
PQID 3049492877
PQPubID 75733
PageCount 10
ParticipantIDs ieee_primary_10478306
proquest_journals_3049492877
crossref_citationtrail_10_1109_JSEN_2024_3378571
crossref_primary_10_1109_JSEN_2024_3378571
ProviderPackageCode CITATION
AAYXX
PublicationCentury 2000
PublicationDate 2024-05-01
PublicationDateYYYYMMDD 2024-05-01
PublicationDate_xml – month: 05
  year: 2024
  text: 2024-05-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE sensors journal
PublicationTitleAbbrev JSEN
PublicationYear 2024
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref35
ref12
ref34
ref15
ref37
ref14
ref36
ref31
ref11
ref10
(ref29) 2019
Richards (ref42) 2005
ref1
(ref2) 2022
Rao (ref17) 2017
ref39
ref16
ref38
ref19
ref18
Richards (ref26) 2014
McWirter (ref33)
ref24
ref23
ref20
ref41
ref22
ref44
ref21
(ref25) 2018
ref43
Chung (ref32) 2014; 3
ref28
ref27
ref8
ref7
ref9
ref4
ref3
ref5
Bandyopadhyay (ref6) 2012
ref40
Gao (ref30) 2023
Gao (ref45) 2022
References_xml – ident: ref10
  doi: 10.3390/app11198926
– ident: ref12
  doi: 10.1109/TENCON.2018.8650148
– ident: ref31
  doi: 10.1109/79.526899
– ident: ref1
  doi: 10.1109/MSP.2005.1406480
– ident: ref16
  doi: 10.1109/IEEECONF44664.2019.9048939
– ident: ref22
  doi: 10.1109/JSEN.2020.3040354
– ident: ref40
  doi: 10.1109/ICARCV.2018.8581329
– year: 2023
  ident: ref30
  article-title: Static background removal in vehicular radar: Filtering in azimuth-elevation-Doppler domain
  publication-title: arXiv:2307.01444
– ident: ref38
  doi: 10.1007/BF02278710
– ident: ref15
  doi: 10.1109/ICAS49788.2021.9551127
– ident: ref28
  doi: 10.1109/ROBOT.2010.5509644
– ident: ref11
  doi: 10.1109/TGRS.2010.2053038
– ident: ref14
  doi: 10.1109/TVT.2021.3092355
– volume-title: White Paper: Imaging Radar Using Cascaded mmWave Sensor Reference Design
  year: 2019
  ident: ref29
– ident: ref36
  doi: 10.1115/1.3662552
– ident: ref23
  doi: 10.1109/ACCESS.2021.3059170
– volume-title: 5G; Nr; User Equipment (UE) Radio Transmission and Reception, Part 2: Range 2 Standalone, 2 Version 15.2.0 Release 15
  year: 2018
  ident: ref25
– volume: 3
  start-page: 599
  volume-title: Academic Press Library in Signal Processing
  year: 2014
  ident: ref32
  article-title: Chapter 14—DOA estimation methods and algorithms
  doi: 10.1016/B978-0-12-411597-2.00014-X
– ident: ref8
  doi: 10.1109/JSTSP.2022.3171168
– ident: ref4
  doi: 10.1109/ICIP.1999.817171
– ident: ref9
  doi: 10.1109/IUCS.2010.5666180
– year: 2012
  ident: ref6
  article-title: Identifications of concealed weapon in a human body
  publication-title: arXiv:1210.5653
– ident: ref44
  doi: 10.1007/978-3-030-58452-8_13
– ident: ref18
  doi: 10.1109/JSEN.2018.2879223
– ident: ref34
  doi: 10.1109/TAP.1986.1143830
– ident: ref7
  doi: 10.1109/JSEN.2020.3036047
– ident: ref5
  doi: 10.1109/TMTT.2008.2007081
– volume-title: Fundamentals of Radar Signal Processing
  year: 2014
  ident: ref26
– start-page: 1
  volume-title: Proc. IEE F-Radar Signal Process.
  ident: ref33
  article-title: Systolic array processor for MVDR beamforming
– year: 2022
  ident: ref45
  article-title: Raw ADC data of 2D-MIMO mmWave radar for carry object detection
– ident: ref37
  doi: 10.1109/IROS45743.2020.9341164
– ident: ref24
  doi: 10.1109/JSEN.2020.3028362
– volume-title: White Paper: MIMO Radar. Texas Instrum
  year: 2017
  ident: ref17
– ident: ref3
  doi: 10.5121/ijcseit.2012.2216
– ident: ref43
  doi: 10.1109/CVPR.2016.90
– ident: ref35
  doi: 10.1109/TPAMI.2009.167
– ident: ref13
  doi: 10.1145/3447993.3483258
– volume-title: Fundamentals of Radar Signal Processing
  year: 2005
  ident: ref42
– year: 2022
  ident: ref2
  publication-title: TSA
– ident: ref39
  doi: 10.1109/ICPR.2004.1334537
– ident: ref20
  doi: 10.1109/JSEN.2023.3295574
– ident: ref21
  doi: 10.1109/TIE.2019.2893843
– ident: ref27
  doi: 10.1109/TAES.1972.309463
– ident: ref41
  doi: 10.1109/MSP.2007.904812
– ident: ref19
  doi: 10.1109/iwem.2018.8536660
SSID ssj0019757
Score 2.4138873
Snippet This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 15091
SubjectTerms Camera
Cameras
carry object
Imaging
Indoor environments
Knowledge management
Millimeter wave communication
Millimeter waves
millimeter-wave (MMWave) radar
multiple observations
Neural networks
object detection
Object recognition
Radar
Radar detection
Radar imaging
Radar tracking
Sensors
tracking
Zooming
Title MMW-Carry: Enhancing Carry Object Detection Through Millimeter-Wave Radar-Camera Fusion
URI https://ieeexplore.ieee.org/document/10478306
https://www.proquest.com/docview/3049492877
Volume 24
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3LSxwxGA_Wi-3BVy1uayWHngoZs5N3b2J3WYRdoV3ZvQ3JJKmlOso4U7B_fZPMrFiL4m0YvoTAL4_v-fsA-ESc5tjnGuVE5IgyQZASJUeOYys8K60nsd55OuOTc3q6ZMu-WD3VwjjnUvKZy-JniuXb67KNrrKjRCVDIsH2q2C5dcVa9yEDJRKtZzjBGFEiln0Ic4jV0en30SyYgjnNCBGSieE_j1DqqvLfVZzel_EWmK1W1qWV_MraxmTln0ekjS9e-jbY7DVNeNxtjR2w5qpd8OYB_-Au2OhboF_cvQWL6XSBTnRd332Bo-oi0nBUP2D6Ac9M9NbAr65JiVsVnHfdfWCsJPx5FTNq0EL_dvCbtroOs0RPFxy30RO3B87Ho_nJBPVdF1CZK9ogJSkpuc-5wIYJzaU22mtLLLPWC-Gx0E4qlmttpPeUMcvpMDeYWuMxMZa8A-vVdeX2AWRMDrX1LAg4WqpgTHIpDZPKKiWCojIAeAVDUfaU5LEzxmWRTBOsiohcEZEreuQG4PP9kJuOj-M54b2IxAPBDoQBOFiBXfRH9raI8UaqggEp3j8x7AN4HWfv0h0PwHpTt-5jUEkac5i24l9xLNsk
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwzV1LbxMxEB6Vcig98ChFBAr4ABckp44f6zUSB9QmSh8JEqRKbou9tmkFbFGagMJ_4a_w27C9m6iA4FaJ22ple7We8Xie3wA8ZU5nxFONKZMUcyEZVrLMsMuIlV6U1rNY7zwYZv0TfjgRkzX4vqqFcc6l5DPXjo8plm_Py3l0le0mKJmg4zY5lEdu8TVYaBcvD_YDOZ9R2uuO9vq4aSKAS6r4DKucszLzNJPECKmzXBvttWVWWOul9ERqlytBtTa591wIm_EONYRb4wkzloV1r8H1oGgIWpeHrYIUSiYg0SAzCOZMTpqgaYeo3cO33WEwPilvMyZzITu_XHupj8sfwj_daL1b8GO5F3Uiy4f2fGba5bffYCL_2826DTcbXRq9qpn_Dqy5ags2LyEsbsFG0-T9dHEXxoPBGO_p6XTxAnWr0wg0Ur1H6QV6baI_Cu27WUpNq9Co7l-EYq3k2aeYM4TH-otDb7TV07BK9OWh3jz6Grfh5Er-8h6sV-eVuw9IiLyjrRdhgOOlCuZyludG5MoqJYMq1gKyJHtRNqDrsffHxyIZX0QVkVOKyClFwykteL6a8rlGHPnX4O1I-UsDa6K3YGfJXEUjlC6KGFHlKpjI8sFfpj2Bjf5ocFwcHwyPHsKN-KU6uXMH1mfTuXsUFLCZeZyOAYJ3V81KPwFP0Ds9
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MMW-Carry%3A+Enhancing+Carry+Object+Detection+Through+Millimeter-Wave+Radar%E2%80%93Camera+Fusion&rft.jtitle=IEEE+sensors+journal&rft.au=Gao%2C+Xiangyu&rft.au=Luo%2C+Youchen&rft.au=Alansari%2C+Ali&rft.au=Sun%2C+Yaping&rft.date=2024-05-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=1530-437X&rft.eissn=1558-1748&rft.volume=24&rft.issue=9&rft.spage=15091&rft_id=info:doi/10.1109%2FJSEN.2024.3378571&rft.externalDBID=NO_FULL_TEXT
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1530-437X&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1530-437X&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1530-437X&client=summon