Exploring Model Units and Training Strategies for End-to-End Speech Recognition

In this work, we explore end-to-end speech recognition models (CTC, RNN-Transducer and attention-based models) with different model units (character, wordpiece and word) and various training strategies. We show that wordpiece unit outperforms character unit for all end-to-end systems on the Switchbo...

Full description

Saved in:
Bibliographic Details
Published in2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) pp. 524 - 531
Main Authors Huang, Mingkun, Lu, Yizhou, Wang, Lan, Qian, Yanmin, Yu, Kai
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.12.2019
Subjects
Acoustics
Data models
Decoding
end-to-end
Hidden Markov models
Predictive models
sequence-to-sequence models
Speech recognition
Training
word piece
Online AccessGet full text

Cover

More Information
Summary:In this work, we explore end-to-end speech recognition models (CTC, RNN-Transducer and attention-based models) with different model units (character, wordpiece and word) and various training strategies. We show that wordpiece unit outperforms character unit for all end-to-end systems on the Switchboard Hub5'00 benchmark. To improve the performance of end-to-end systems, we propose a multi-stage pretraining strategy, which gives 25.0% and 18.0% relative improvements over training from scratch for attention and RNN-T models respectively with wordpiece units. We achieve state-of-the-art performance on the Switchboard+Fisher-2000h task, outperforming all prior work. Together with other training strategies such as label smoothing and data augmentation, we achieve 5.9%/12.1% WER on the Switch-board/CallHome test set without using any external language models. This is a new performance milestone for a single end-to-end system, and it is also much better than the previous published best hybrid system, which is 6.7%/12.5% on each set individually.
DOI:10.1109/ASRU46091.2019.9003834