Very Deep Self-Attention Networks for End-to-End Speech Recognition

Recently, end-to-end sequence-to-sequence models for speech recognition have gained significant interest in the research community. While previous architecture choices revolve around time-delay neural networks (TDNN) and long short-term memory (LSTM) recurrent neural networks, we propose to use self...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Ngoc-Quan Pham, Thai-Son, Nguyen, Niehues, Jan, Müller, Markus, Stüker, Sebastian, Waibel, Alexander
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 03.05.2019
Subjects
Decoders
Hybrid systems
Neural networks
Recurrent neural networks
Speech recognition
Switching theory
Test sets
Training
Voice recognition
Online AccessGet full text

Cover

More Information
Summary:Recently, end-to-end sequence-to-sequence models for speech recognition have gained significant interest in the research community. While previous architecture choices revolve around time-delay neural networks (TDNN) and long short-term memory (LSTM) recurrent neural networks, we propose to use self-attention via the Transformer architecture as an alternative. Our analysis shows that deep Transformer networks with high learning capacity are able to exceed performance from previous end-to-end approaches and even match the conventional hybrid systems. Moreover, we trained very deep models with up to 48 Transformer layers for both encoder and decoders combined with stochastic residual connections, which greatly improve generalizability and training efficiency. The resulting models outperform all previous end-to-end ASR approaches on the Switchboard benchmark. An ensemble of these models achieve 9.9% and 17.7% WER on Switchboard and CallHome test sets respectively. This finding brings our end-to-end models to competitive levels with previous hybrid systems. Further, with model ensembling the Transformers can outperform certain hybrid systems, which are more complicated in terms of both structure and training procedure.
ISSN:2331-8422