Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition

Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these sh...

Full description

Saved in:
Bibliographic Details
Published inSensors (Basel, Switzerland) Vol. 22; no. 15; p. 5787
Main Authors Yuan, Liangqi, Andrews, Jack, Mu, Huaizheng, Vakil, Asad, Ewing, Robert, Blasch, Erik, Li, Jia
Format Journal Article
LanguageEnglish
Published Basel MDPI AG 03.08.2022
MDPI
Subjects
Accuracy
activity recognition
Classification
Decision making
Deep learning
human identification
Human subjects
passive infrared
passive radio frequency
sensor fusion
Sensors
Software
Online AccessGet full text

Cover

More Information
Summary:Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
These authors contributed equally to this work.
Current address: 115 Library Dr., Engineering Center 422, Rochester, MI 48309, USA.
ISSN:1424-8220
1424-8220
DOI:10.3390/s22155787