Improving CTC-Based Speech Recognition Via Knowledge Transferring from Pre-Trained Language Models

• Published in: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 8517 - 8521
• Main Authors: Deng, Keqi, Cao, Songjun, Zhang, Yike, Ma, Long, Cheng, Gaofeng, Xu, Ji, Zhang, Pengyuan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 23.05.2022
• Subjects: Acoustics; Benchmark testing; Bit error rate; Conferences; connectionist temporal classification; knowledge transfer; pre-trained language model; Representation learning; Signal processing; Speech processing; speech recognition
• ISSN: 2379-190X
• DOI: 10.1109/ICASSP43922.2022.9747887

Recently, end-to-end automatic speech recognition models based on connectionist temporal classification (CTC) have achieved impressive results, especially when fine-tuned from wav2vec2.0 models. Due to the conditional independence assumption, CTC-based models are always weaker than attention-based encoder-decoder models and require the assistance of external language models (LMs). To solve this issue, we propose two knowledge transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to improve CTC-based models. The first method is based on representation learning, in which the CTC-based models use the representation produced by BERT as an auxiliary learning target. The second method is based on joint classification learning, which combines GPT2 for text modeling with a hybrid CTC/attention architecture. Experiment on AISHELL-1 corpus yields a character error rate (CER) of 4.2% on the test set. When compared to the vanilla CTC-based models fine-tuned from the wav2vec2.0 models, our knowledge transferring method reduces CER by 16.1% relatively without external LMs.