LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under low-resource settings. To this end, we propose LightHuBERT, a o...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Wang, Rui, Bai, Qibing, Ao, Junyi, Long, Zhou, Xiong, Zhixiang, Wei, Zhihua, Zhang, Yu, Ko, Tom, Li, Haizhou
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 18.06.2022
Subjects
Automatic speech recognition
Compression ratio
Distillation
Learning
Mathematical models
Parameters
Representations
Speech processing
Transformers
Online AccessGet full text

Cover

More Information
Summary:Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under low-resource settings. To this end, we propose LightHuBERT, a once-for-all Transformer compression framework, to find the desired architectures automatically by pruning structured parameters. More precisely, we create a Transformer-based supernet that is nested with thousands of weight-sharing subnets and design a two-stage distillation strategy to leverage the contextualized latent representations from HuBERT. Experiments on automatic speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT enables over \(10^9\) architectures concerning the embedding dimension, attention dimension, head number, feed-forward network ratio, and network depth. LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a \(3.5\times\) compression ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss. The code and pre-trained models are available at https://github.com/mechanicalsea/lighthubert.
ISSN:2331-8422