A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition

• Published in: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 6538 - 6542
• Main Authors: Jiang, Dongwei, Li, Wubo, Zhang, Ruixiong, Cao, Miao, Luo, Ne, Han, Yang, Zou, Wei, Han, Kun, Li, Xiangang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 06.06.2021
• Subjects: Adaptation models; Conferences; Data models; knowledge transfer; Predictive coding; Signal processing; speaking style; Speech recognition; streaming speech recognition; Training; transformer; unsupervised pretraining
• ISSN: 2379-190X
• DOI: 10.1109/ICASSP39728.2021.9414539

The construction of an effective good speech recognition system typically requires large amounts of transcribed data, which is expensive to collect. To overcome this problem, many unsupervised pretraining methods have been proposed. Among these methods, Masked Predictive Coding achieved significant improvements on various speech recognition datasets with BERT-like Masked Reconstruction loss and transformer backbone. However, many aspects of MPC have yet to be fully investigated. In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pretraining data speaking style, its extension on streaming model, and strategies for better transferring learned knowledge from pretraining stage to downstream tasks. The experimental results demonstrated that pretraining data with a matching speaking style is more useful on downstream recognition tasks. A unified training objective with APC and MPC provided an 8.46% relative error reduction on the streaming model trained on HKUST. Additionally, the combination of target data adaption and layerwise discriminative training facilitated the knowledge transfer of MPC, which realized 3.99% relative error reduction on AISHELL over a strong baseline.