The impact of removing head movements on audio-visual speech enhancement

This paper investigates the impact of head movements on audio-visual speech enhancement (AVSE). Although being a common conversational feature, head movements have been ignored by past and recent studies: they challenge today's learning-based methods as they often degrade the performance of mod...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Kang, Zhiqi, Sadeghi, Mostafa, Horaud, Radu, Alameda-Pineda, Xavier, Donley, Jacob, Kumar, Anurag
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 02.02.2022
Subjects
Coders
Head movement
Performance degradation
Performance enhancement
Speech processing
Online AccessGet full text

Cover

More Information
Summary:This paper investigates the impact of head movements on audio-visual speech enhancement (AVSE). Although being a common conversational feature, head movements have been ignored by past and recent studies: they challenge today's learning-based methods as they often degrade the performance of models that are trained on clean, frontal, and steady face images. To alleviate this problem, we propose to use robust face frontalization (RFF) in combination with an AVSE method based on a variational auto-encoder (VAE) model. We briefly describe the basic ingredients of the proposed pipeline and we perform experiments with a recently released audio-visual dataset. In the light of these experiments, and based on three standard metrics, namely STOI, PESQ and SI-SDR, we conclude that RFF improves the performance of AVSE by a considerable margin.
ISSN:2331-8422