A Query-by-Singing System for Retrieving Karaoke Music

• Published in: IEEE transactions on multimedia Vol. 10; no. 8; pp. 1626 - 1637
• Main Authors: YU, Hung-Ming, TSAI, Wei-Ho, WANG, Hsin-Min
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.12.2008; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Acoustics; Applied sciences; Bayesian information criterion; Bayesian methods; Computer science; control theory; systems; Councils; Data mining; dynamic time warping; Error correction; Exact sciences and technology; Fundamental areas of phenomenology (including applications); Humans; Information systems. Data bases; Internet; Karaoke; Memory organisation. Data processing; Multiple signal classification; Music and musical intruments; Music information retrieval; Musical recordings; Physics; query-by-singing; Signal processing; Software; Bayes estimation; Error estimation; Database query; Similarity; Distributed database; Dynamic time warping; Stereophony; Very large databases; Information retrieval; Multidatabase retrieval; Similarity solution; Abstraction; Experimental study; karaoke; Song; query-by-singing; Music; Musical acoustics; Melody; Database; Audio acoustics; Feasibility; Bayesian information criterion; music information retrieval
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2008.2007345

This paper investigates the problem of retrieving karaoke music using query-by-singing techniques. Unlike regular CD music, where the stereo sound involves two audio channels that usually sound the same, karaoke music encompasses two distinct channels in each track: one is a mixture of the lead vocals and background accompaniment, and the other consists of accompaniment only. Although the two audio channels are distinct, the accompaniments in the two channels often resemble each other. We exploit this characteristic to: i) infer the background accompaniment for the lead vocals from the accompaniment-only channel, so that the main melody underlying the lead vocals can be extracted more effectively, and ii) detect phrase onsets based on the Bayesian information criterion (BIC) to predict the onset points of a song where a user's sung query may begin, so that the similarity between the melodies of the query and the song can be examined more efficiently. To further refine extraction of the main melody, we propose correcting potential errors in the estimated sung notes by exploiting a composition characteristic of popular songs whereby the sung notes within a verse or chorus section usually vary no more than two octaves. In addition, to facilitate an efficient and accurate search of a large music database, we employ multiple-pass dynamic time warping (DTW) combined with multiple-level data abstraction (MLDA) to compare the similarities of melodies. The results of experiments conducted on a karaoke database comprised of 1071 popular songs demonstrate the feasibility of query-by-singing retrieval for karaoke music.