Melody Extraction and Musical Onset Detection via Probabilistic Models of Framewise STFT Peak Data

• Published in: IEEE transactions on audio, speech, and language processing Vol. 15; no. 4; pp. 1257 - 1272
• Main Authors: Thornburg, H., Leistikow, R.J., Berger, J.
• Format: Journal Article
• Language: English
• Published: Piscataway, NJ IEEE 01.05.2007; Institute of Electrical and Electronics Engineers
• Subjects: Acoustics; Applied sciences; Audio signals; Bayesian methods; Cities and towns; Context modeling; Data mining; Detection, estimation, filtering, equalization, prediction; Dynamic Bayesian networks; Exact sciences and technology; Extraction; Fourier transforms; Inference; Information, signal and communications theory; Mathematical models; Music; music transcription; onset detection; Pianos; pitch identification; Probabilistic methods; Reverberation; Robustness; Segmentation; Signal and communications theory; Signal, noise; Spectra; Steady-state; Telecommunications and information theory; Fourier transformation; Probabilistic approach; Segmentation; Acoustic signal; Change detection; Signal estimation; Unsteady state; Violin; pitch identification; Modeling; Steady state; Dynamic Bayesian networks; onset detection; Audio signal; music transcription; Bayes network; Localization; Musical sound; Pitch(acoustics); Musical score; Musical instrument
• ISSN: 1558-7916; 1558-7924
• DOI: 10.1109/TASL.2006.889801

We propose a probabilistic method for the joint segmentation and melody extraction for musical audio signals which arise from a monophonic score. The method operates on framewise short-time Fourier transform (STFT) peaks, enabling a computationally efficient inference of note onset, duration, and pitch attributes while retaining sufficient information for pitch determination and spectral change detection. The system explicitly models note events in terms of transient and steady-state regions as well as possible gaps between note events. In this way, the system readily distinguishes abrupt spectral changes associated with musical onsets from other abrupt change events. Additionally, the method may incorporate melodic context by modeling note-to-note dependences. The method is successfully applied to a variety of piano and violin recordings containing reverberation, effective polyphony due to legato playing style, expressive pitch variations, and background voices. While the method does not provide a sample-accurate segmentation, it facilitates the latter in subsequent processing by isolating musical onsets to frame neighborhoods and identifying possible pitch content before and after the true onset sample location