Improving N-gram Language Models with Pre-trained Deep Transformer

Although n-gram language models (LMs) have been outperformed by the state-of-the-art neural LMs, they are still widely used in speech recognition due to its high efficiency in inference. In this paper, we demonstrate that n-gram LM can be improved by neural LMs through a text generation based data a...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Wang, Yiren, Huang, Hongzhao, Liu, Zhe, Pang, Yutong, Wang, Yongqiang, Zhai, ChengXiang, Peng, Fuchun
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 22.11.2019
Subjects
Domains
Error reduction
Speech recognition
Transformers
Voice recognition
Online AccessGet full text

Cover

More Information
Summary:Although n-gram language models (LMs) have been outperformed by the state-of-the-art neural LMs, they are still widely used in speech recognition due to its high efficiency in inference. In this paper, we demonstrate that n-gram LM can be improved by neural LMs through a text generation based data augmentation method. In contrast to previous approaches, we employ a large-scale general domain pre-training followed by in-domain fine-tuning strategy to construct deep Transformer based neural LMs. Large amount of in-domain text data is generated with the well trained deep Transformer to construct new n-gram LMs, which are then interpolated with baseline n-gram systems. Empirical studies on different speech recognition tasks show that the proposed approach can effectively improve recognition accuracy. In particular, our proposed approach brings significant relative word error rate reduction up to 6.0% for domains with limited in-domain data.
ISSN:2331-8422