Privacy-preserving activity recognition using multimodal sensors in smart office

• Published in: Future generation computer systems Vol. 148; pp. 27 - 38
• Main Authors: Zhang, Xiangying, Zheng, Pai, Peng, Tao, Li, Dai, Zhang, Xujun, Tang, Renzhong
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.11.2023
• Subjects: Activity recognition; Infrared array sensor; Multimodal data; Privacy-preserving method; Sensing chair; Smartphone; Activity recognition; Privacy-preserving method; Multimodal data; Smartphone; Infrared array sensor; Sensing chair
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2023.05.023

Developing a human activity recognition (HAR) system for employees is essential to incorporate intelligence into smart office environments, enabling various human-centered applications to enhance employees’ well-being. Although remarkable progress has been made for the HAR in the smart office, several issues still exist, including lacking a privacy-preserving and unobtrusive method and demanding enhanced generalization performance across users. Therefore, a novel privacy-preserving HAR method based on multimodal sensors is investigated in this study, containing an infrared array sensor, a sensing chair, and a triaxial accelerometer built in a smartphone. The effectiveness of different multimodal combinations is examined. Moreover, a deep learning model is developed for multimodal data fusion to enhance the generalization performances across users. The model contains a residual 3D convolutional neural network (CNN) and 1D CNN for learning spatial–temporal feature representation of different modalities. Additionally, external memory units and an adaptive decision fusion operation are utilized for multimodal data fusion. Finally, extensive experiments are conducted to examine the performance of the proposed model using a self-collected dataset and the leave-one-subject-out cross-validation approach. The results verify the effectiveness of the proposed model. •Propose a novel privacy-preserving and unobtrusive human activity recognition method.•Integrate external memory units and adaptive score fusion for enhancing generalization performance of the method.•Analyze detailed performances of different multimodal solutions.•Evaluate the classification performance of the proposed method on unfamiliar users.