A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors

• Published in: IEEE journal of biomedical and health informatics Vol. 20; no. 5; pp. 1281 - 1290
• Main Authors: Jian Wu, Lu Sun, Jafari, Roozbeh
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accelerometers; American Sign Language (ASL) recognition; Arm - physiology; Assistive technology; Communication Aids for Disabled; Electromyography - methods; Electronics; Equipment Design; Feature extraction; feature selection; Female; Gesture recognition; Humans; inertial measurement unit (IMU) sensor; Male; Muscle, Skeletal - physiology; Pattern Recognition, Automated - methods; Real-time systems; sensor fusion; Sensor systems; Sign Language; Signal Processing, Computer-Assisted - instrumentation; Speech Recognition Software; surface EMG (sEMG)
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2016.2598302

A sign language recognition system translates signs performed by deaf individuals into text/speech in real time. Inertial measurement unit and surface electromyography (sEMG) are both useful modalities to detect hand/arm gestures. They are able to capture signs and the fusion of these two complementary sensor modalities will enhance system performance. In this paper, a wearable system for recognizing American Sign Language (ASL) in real time is proposed, fusing information from an inertial sensor and sEMG sensors. An information gain-based feature selection scheme is used to select the best subset of features from a broad range of well-established features. Four popular classification algorithms are evaluated for 80 commonly used ASL signs on four subjects. The experimental results show 96.16% and 85.24% average accuracies for intra-subject and intra-subject cross session evaluation, respectively, with the selected feature subset and a support vector machine classifier. The significance of adding sEMG for ASL recognition is explored and the best channel of sEMG is highlighted.