End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting

• Published in: Computer speech & language Vol. 75; p. 101369
• Main Authors: Desot, Thierry, Portet, François, Vacher, Michel
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2022; Elsevier
• Subjects: Artificial Intelligence; Computation and Language; Computer Science; End-to-end (E2E) learning; Explainable AI (XAI); Machine Learning; Natural Language Understanding (NLU); Smart home; Spoken Language Understanding (SLU); Spoken Language Understanding (SLU); End-to-end (E2E) learning; Natural Language Understanding (NLU); Explainable AI (XAI); Smart home
• ISSN: 0885-2308; 1095-8363
• DOI: 10.1016/j.csl.2022.101369

Spoken Language Understanding (SLU) is a core task in most human–machine interaction systems . With the emergence of smart homes, smart phones and smart speakers, SLU has become a key technology for the industry. In a classical SLU approach, an Automatic Speech Recognition (ASR) module transcribes the speech signal into a textual representation from which a Natural Language Understanding (NLU) module extracts semantic information. Recently End-to-End SLU (E2E SLU) based on Deep Neural Networks has gained momentum since it benefits from the joint optimisation of the ASR and the NLU parts, hence limiting the cascade of error effect of the pipeline architecture. However, little is known about the actual linguistic properties used by E2E models to predict concepts and intents from speech input. In this paper, we present a study identifying the signal features and other linguistic properties used by an E2E model to perform the SLU task. The study is carried out in the application domain of a smart home that has to handle non-English (here French) voice commands. The results show that a good E2E SLU performance does not always require a perfect ASR capability. Furthermore, the results show the superior capabilities of the E2E model in handling background noise and syntactic variation compared to the pipeline model. Finally, a finer-grained analysis suggests that the E2E model uses the pitch information of the input signal to identify voice command concepts. The results and methodology outlined in this paper provide a springboard for further analyses of E2E models in speech processing. •1st paper studying acoustic and linguistic features used by E2E DNN models for SLU.•Joint learning of ASR and NLU in E2E SLU does fight the cascade of errors effect.•E2E SLU models exploit prosodic information to predict intents and concepts.•E2E models are more robust than pipeline models to noise and grammatical variation.•Transfer learning and artificial training data are efficient for low resource tasks.