MagTrack: A Wearable Tongue Motion Tracking System for Silent Speech Interfaces

• Published in: Journal of speech, language, and hearing research Vol. 66; no. 8S; pp. 3206 - 3221
• Main Authors: Cao, Beiming, Ravi, Shravan, Sebkhi, Nordine, Bhavsar, Arpan, Inan, Omer T, Xu, Wen, Wang, Jun
• Format: Journal Article
• Language: English
• Published: United States American Speech-Language-Hearing Association 17.08.2023
• Subjects: Adult; Feedback, Sensory; Humans; Motion; Phonetics; Special Issue: Select Papers From the 2022 Conference on Motor Speech—Basic Science and Clinical Innovation; Speech; Speech Perception; Tongue
• ISSN: 1092-4388; 1558-9102
• DOI: 10.1044/2023_JSLHR-22-00319

Current electromagnetic tongue tracking devices are not amenable for daily use and thus not suitable for silent speech interface and other applications. We have recently developed MagTrack, a novel wearable electromagnetic articulograph tongue tracking device. This study aimed to validate MagTrack for potential silent speech interface applications. We conducted two experiments: (a) classification of eight isolated vowels in consonant-vowel-consonant form and (b) continuous silent speech recognition. In these experiments, we used data from healthy adult speakers collected with MagTrack. The performance of vowel classification was measured by accuracies. The continuous silent speech recognition was measured by phoneme error rates. The performance was then compared with results using data collected with commercial electromagnetic articulograph in a prior study. The isolated vowel classification using MagTrack achieved an average accuracy of 89.74% when leveraging all MagTrack signals ( , , coordinates; orientation; and magnetic signals), which outperformed the accuracy using commercial electromagnetic articulograph data (only , coordinates) in our previous study. The continuous speech recognition from two subjects using MagTrack achieved phoneme error rates of 73.92% and 66.73%, respectively. The commercial electromagnetic articulograph achieved 64.53% from the same subject (66.73% using MagTrack data). MagTrack showed comparable results with the commercial electromagnetic articulograph when using the same localized information. Adding raw magnetic signals would improve the performance of MagTrack. Our preliminary testing demonstrated the potential for silent speech interface as a lightweight wearable device. This work also lays the foundation to support MagTrack's potential for other applications including visual feedback-based speech therapy and second language learning.