Can DNNs Learn to Lipread Full Sentences?

Finding visual features and suitable models for lipreading tasks that are more complex than a well-constrained vocabulary has proven challenging. This paper explores state-of-the-art Deep Neural Network architectures for lipreading based on a Sequence to Sequence Recurrent Neural Network. We report...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Sterpu, George, Saam, Christian, Harte, Naomi
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 29.05.2018
Subjects
Artificial neural networks
Lipreading
Markov chains
Neural networks
Recurrent neural networks
Sentences
Task complexity
Online AccessGet full text

Cover

More Information
Summary:Finding visual features and suitable models for lipreading tasks that are more complex than a well-constrained vocabulary has proven challenging. This paper explores state-of-the-art Deep Neural Network architectures for lipreading based on a Sequence to Sequence Recurrent Neural Network. We report results for both hand-crafted and 2D/3D Convolutional Neural Network visual front-ends, online monotonic attention, and a joint Connectionist Temporal Classification-Sequence-to-Sequence loss. The system is evaluated on the publicly available TCD-TIMIT dataset, with 59 speakers and a vocabulary of over 6000 words. Results show a major improvement on a Hidden Markov Model framework. A fuller analysis of performance across visemes demonstrates that the network is not only learning the language model, but actually learning to lipread.
ISSN:2331-8422