Joint Detection and Classification of Singing Voice Melody Using Convolutional Recurrent Neural Networks

• Published in: Applied sciences Vol. 9; no. 7; p. 1324
• Main Authors: Kum, Sangeun, Nam, Juhan
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.04.2019
• Subjects: Acoustics; Classification; Convolution; convolutional recurrent neural network; Frequency; Information retrieval; International conferences; joint detection and classification; Melody; melody extraction; Methods; Musical instruments; Musical performances; Neural networks; Recurrent neural networks; Signal processing; Singing; singing voice detection; Test sets; Voice recognition; Queensland Australia; France; Australia; United States > US; Taiwan
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app9071324

Singing melody extraction essentially involves two tasks: one is detecting the activity of a singing voice in polyphonic music, and the other is estimating the pitch of a singing voice in the detected voiced segments. In this paper, we present a joint detection and classification (JDC) network that conducts the singing voice detection and the pitch estimation simultaneously. The JDC network is composed of the main network that predicts the pitch contours of the singing melody and an auxiliary network that facilitates the detection of the singing voice. The main network is built with a convolutional recurrent neural network with residual connections and predicts pitch labels that cover the vocal range with a high resolution, as well as non-voice status. The auxiliary network is trained to detect the singing voice using multi-level features shared from the main network. The two optimization processes are tied with a joint melody loss function. We evaluate the proposed model on multiple melody extraction and vocal detection datasets, including cross-dataset evaluation. The experiments demonstrate how the auxiliary network and the joint melody loss function improve the melody extraction performance. Furthermore, the results show that our method outperforms state-of-the-art algorithms on the datasets.