Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

• Published in: Proceedings (IEEE International Conference on Multimedia and Expo) pp. 1 - 6
• Main Authors: Bai, Ye, Li, Chenxing, Li, Hao, Zhao, Yuanyuan, Wang, Xiaorui
• Format: Conference Proceeding
• Language: English
• Published: IEEE 15.07.2024
• Subjects: Accuracy; Benchmark testing; Complexity theory; lyrics recognition; multi-task audio source separation; Multitasking; Robustness; Source separation; Speech recognition
• ISSN: 1945-788X
• DOI: 10.1109/ICME57554.2024.10687477

In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.