GACELA: A Generative Adversarial Context Encoder for Long Audio Inpainting of Music

• Published in: IEEE journal of selected topics in signal processing Vol. 15; no. 1; pp. 120 - 131
• Main Authors: Marafioti, Andres, Majdak, Piotr, Holighaus, Nicki, Perraudin, Nathanael
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Acoustic signal processing; Audio data; Coders; computer generated music; Context; Copying; Discriminators; machine learning; Multiple signal classification; Music; Neural networks; Spectrogram; Synthesizers; Task analysis; Time-domain analysis
• ISSN: 1932-4553; 1941-0484
• DOI: 10.1109/JSTSP.2020.3037506

In this article, we introduce GACELA, a conditional generative adversarial network (cGAN) designed to restore missing audio data with durations ranging between hundreds of milliseconds and a few seconds, i.e., to perform long-gap audio inpainting. While previous work either addressed shorter gaps or relied on exemplars by copying available information from other signal parts, GACELA addresses the inpainting of long gaps in two aspects. First, it considers various time scales of audio information by relying on five parallel discriminators with increasing resolution of receptive fields. Second, it is conditioned not only on the available information surrounding the gap, i.e., the context, but also on the latent variable of the cGAN. This addresses the inherent multi-modality of audio inpainting for such long gaps while providing the user with different inpainting options. GACELA was evaluated in listening tests on music signals of varying complexity and varying gap durations from 375 to 1500 ms. Under laboratory conditions, our subjects were often able to detect the inpainting. However, the severity of the inpainted artifacts was rated between not disturbing and mildly disturbing. GACELA represents a framework capable of integrating future improvements such as processing of more auditory-related features or explicit musical features. Our software and trained models, complemented by instructive examples, are available online.