Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist

• Published in: PloS one Vol. 15; no. 1; p. e0227039
• Main Authors: Siddiqui, Nabeel, Chan, Rosa H M
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 01.01.2020; Public Library of Science (PLoS)
• Subjects: Accelerometers; Accuracy; Acoustic measurement; Acoustics; Adult; Algorithms; Arm; Biology and Life Sciences; Discriminant analysis; Electrical engineering; Engineering and Technology; Female; Gesture recognition; Gestures; Hand; Hand - physiology; Humans; Inertial platforms; Locomotion; Male; Medicine and Health Sciences; Microphones; Model accuracy; Movement; Pattern Recognition, Physiological; Physical Sciences; Redundancy; Sensors; Skin; Social Sciences; Studies; Units of measurement; Wearable Electronic Devices; Wrist; Wrist Joint - physiology; Young Adult; Hong Kong China; United States > US; China
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0227039

To facilitate hand gesture recognition, we investigated the use of acoustic signals with an accelerometer and gyroscope at the human wrist. As a proof-of-concept, the prototype consisted of 10 microphone units in contact with the skin placed around the wrist along with an inertial measurement unit (IMU). The gesture recognition performance was evaluated through the identification of 13 gestures used in daily life. The optimal area for acoustic sensor placement at the wrist was examined using the minimum redundancy and maximum relevance feature selection algorithm. We recruited 10 subjects to perform over 10 trials for each set of hand gestures. The accuracy was 75% for a general model with the top 25 features selected, and the intra-subject average classification accuracy was over 80% with the same features using one microphone unit at the mid-anterior wrist and an IMU. These results indicate that acoustic signatures from the human wrist can aid IMU sensing for hand gesture recognition, and the selection of a few common features for all subjects could help with building a general model. The proposed multimodal framework helps address the single IMU sensing bottleneck for hand gestures during arm movement and/or locomotion.