VirtualHAR: Virtual Sensing Device and Correlation-Based Learning Approach for Multiwearable Sensing Device-Based Human Activity Recognition

• Published in: IEEE internet of things journal Vol. 12; no. 13; pp. 23577 - 23597
• Main Authors: Ahmad, Nafees, Leung, Ho-Fung, Farnia, Farzan
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.07.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Body parts; Computational modeling; Convolutional neural networks; Correlation; Data mining; Deep learning; Devices; Effectiveness; Feature extraction; Human activity recognition; human activity recognition (HAR); Human motion; Internet of Things; Learning; Legged locomotion; Manuals; Modules; Performance evaluation; Qualitative analysis; Representations; Soft sensors; ubiquitous and mobile computing; Ubiquitous computing; Virtual sensors; wearable sensing devices
• DOI: 10.1109/JIOT.2025.3555799
• ISSN: 2327-4662

Human activity recognition (HAR) is a prominent research direction in ubiquitous computing. Current state-of-the-art HAR models achieve great success by learning the correlations between the regions of the body parts by using the attached sensing devices for feature extraction. However, explicitly computing the correlations between whole body parts and whole sub-body parts, which is crucial for extracting discriminatory features for some activities, has not been investigated due to the lack of sensing devices that capture the movements of the whole (sub-)body parts. This study proposes an effective yet lightweight VirtualHAR framework, which automatically models correlations between the whole body parts, whole sub-body parts, and regions based on the concept of virtual sensing devices. The VirtualHAR framework mainly encompasses three modules. The backbone feature extraction (BEF) module extracts the features from a physical sensing device, based on which the Multipurpose Correlations Learning module constructs virtual sensing devices for body parts and sub-body parts and then exploits the explicit correlations between body parts, sub-body parts as well as in regions by using their attached physical sensing devices. Finally, the global aggregation (GA) module learns the GA representation for each physical sensing device by collecting the learned correlated representation from each virtual sensing device and physical sensing device. Comprehensive experiments on benchmark HAR datasets and a resource-constrained device confirm that VirtualHAR outperforms SOTA models in recognition performance and computational complexity. Through thorough quantitative and qualitative analysis, we validate the proposed VirtualHAR framework's effectiveness and efficiency.