Hierarchical Classification for Instrument Activity Detection in Orchestral Music Recordings

• Published in: IEEE/ACM transactions on audio, speech, and language processing Vol. 31; pp. 2567 - 2578
• Main Authors: Krause, Michael, Muller, Meinard
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Analytical models; Annotations; Classification; Hierarchical classification; Information retrieval; instrument activity detection; Instruments; Music; music information retrieval; music processing; orchestral music; Popular music; Recording; Speech processing; Task analysis; Taxonomy
• ISSN: 2329-9290; 2329-9304
• DOI: 10.1109/TASLP.2023.3291506

Instrument activity detection is a fundamental task in music information retrieval, serving as a basis for many applications, such as music recommendation, music tagging, or remixing. Most published works on this task cover popular music and music for smaller ensembles. In this article, we embrace orchestral and opera music recordings as a rarely considered scenario for automated instrument activity detection. Orchestral music is particularly challenging since it consists of intricate polyphonic and polytimbral sound mixtures where multiple instruments are playing simultaneously. Orchestral instruments can naturally be arranged in hierarchical taxonomies, according to instrument families. As the main contribution of this article, we show that a hierarchical classification approach can be used to detect instrument activity in our scenario, even if only few fine-grained, instrument-level annotations are available. We further consider additional loss terms for improving the hierarchical consistency of predictions. For our experiments, we collect a dataset containing 14 hours of orchestral music recordings with aligned instrument activity annotations. Finally, we perform an analysis of the behavior of our proposed approach with regard to potential confounding errors.