Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound

Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically. Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges. Systems lacking explicit t...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Lee, Junwon, Im, Jaekwon, Dabin Kim, Nam, Juhan
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 21.08.2024
Subjects
Alignment
Annotations
Controllability
Multimedia
Self-supervised learning
Semantics
Sound generation
State-of-the-art reviews
Synchronism
User experience
Online AccessGet full text

Cover

More Information
Summary:Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically. Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges. Systems lacking explicit temporal features suffer from poor controllability and alignment, while timestamp-based models require costly and subjective human annotation. We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a temporal event condition with semantic timbre prompts (audio or text). RMS, a frame-level intensity envelope feature closely related to audio semantics, ensures high controllability and synchronization. The annotation-free self-supervised learning framework consists of two stages, Video2RMS and RMS2Sound, incorporating novel ideas including RMS discretization and RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation shows that Video-Foley achieves state-of-the-art performance in audio-visual alignment and controllability for sound timing, intensity, timbre, and nuance. Code, model weights, and demonstrations are available on the accompanying website. (https://jnwnlee.github.io/video-foley-demo)
ISSN:2331-8422