On The Usefulness of Self-Attention for Automatic Speech Recognition with Transformers

Self-attention models such as Transformers, which can capture temporal relationships without being limited by the distance between events, have given competitive speech recognition results. However, we note the range of the learned context increases from the lower to upper self-attention layers, whi...

Full description

Saved in:
Bibliographic Details
Published in2021 IEEE Spoken Language Technology Workshop (SLT) pp. 89 - 96
Main Authors Zhang, Shucong, Loweimi, Erfan, Bell, Peter, Renals, Steve
Format Conference Proceeding
LanguageEnglish
Published IEEE 19.01.2021
Subjects
Acoustics
Conferences
end-to-end
Error analysis
Measurement
Network architecture
self-attention
speech recognition
Switches
Training
transformer
Online AccessGet full text

Cover

More Information
Summary:Self-attention models such as Transformers, which can capture temporal relationships without being limited by the distance between events, have given competitive speech recognition results. However, we note the range of the learned context increases from the lower to upper self-attention layers, whilst acoustic events often happen within short time spans in a left-to-right order. This leads to a question: for speech recognition, is a global view of the entire sequence useful for the upper self-attention encoder layers in Transformers? To investigate this, we train models with lower self-attention/upper feed-forward layers encoders on Wall Street Journal and Switchboard. Compared to baseline Transformers, no performance drop but minor gains are observed. We further developed a novel metric of the diagonality of attention matrices and found the learned diagonality indeed increases from the lower to upper encoder self-attention layers. We conclude the global view is unnecessary in training upper encoder layers.
DOI:10.1109/SLT48900.2021.9383521