Multimodal Speech Capture System for Speech Rehabilitation and Learning

• Published in: IEEE transactions on biomedical engineering Vol. 64; no. 11; pp. 2639 - 2649
• Main Authors: Sebkhi, Nordine, Desai, Dhyey, Islam, Mohammad, Lu, Jun, Wilson, Kimberly, Ghovanloo, Maysam
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adult; Algorithms; electromagnetic articulography (EMA); Feedback; Humans; Imaging, Three-Dimensional - methods; Kinematics; Language; lip reading; Lipreading; Lips; Magnetic dipole localization; Magnetic fields; Magnetic sensors; Magnetometers; Male; motor speech disorders; Patients; Pattern matching; Rehabilitation; Speech; Speech - physiology; Speech Disorders - diagnostic imaging; Speech Disorders - physiopathology; Speech Disorders - rehabilitation; Speech-Language Pathology - instrumentation; Speech-Language Pathology - methods; Tongue; Tongue - diagnostic imaging; Tongue - physiology; tongue tracking; visualization
• ISSN: 0018-9294; 1558-2531
• DOI: 10.1109/TBME.2017.2654361

Speech-language pathologists (SLPs) are trained to correct articulation of people diagnosed with motor speech disorders by analyzing articulators' motion and assessing speech outcome while patients speak. To assist SLPs in this task, we are presenting the multimodal speech capture system (MSCS) that records and displays kinematics of key speech articulators, the tongue and lips, along with voice, using unobtrusive methods. Collected speech modalities, tongue motion, lips gestures, and voice are visualized not only in real-time to provide patients with instant feedback but also offline to allow SLPs to perform post-analysis of articulators' motion, particularly the tongue, with its prominent but hardly visible role in articulation. We describe the MSCS hardware and software components, and demonstrate its basic visualization capabilities by a healthy individual repeating the words "Hello World." A proof-of-concept prototype has been successfully developed for this purpose, and will be used in future clinical studies to evaluate its potential impact on accelerating speech rehabilitation by enabling patients to speak naturally. Pattern matching algorithms to be applied to the collected data can provide patients with quantitative and objective feedback on their speech performance, unlike current methods that are mostly subjective, and may vary from one SLP to another.