Deep multimodal learning for Audio-Visual Speech Recognition

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature...

Full description

Saved in:
Bibliographic Details
Published in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 2130 - 2134
Main Authors Mroueh, Youssef, Marcheret, Etienne, Goel, Vaibhava
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2015
Subjects
Audio-Visual Automatic Speech Recognition (AV-ASR)
Correlation
Deep Neural Networks
Error analysis
Joints
Multimodal Learning
Speech
Speech recognition
Training
Visualization
Online AccessGet full text

Cover

More Information
Summary:In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of 41% under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of 35.83% demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of 34.03%.
ISSN:1520-6149
DOI:10.1109/ICASSP.2015.7178347