Detecting False Alarms and Misses in Audio Captions

Metrics to evaluate audio captions simply provide a score without much explanation regarding what may be wrong in case the score is low. Manual human intervention is needed to find any shortcomings of the caption. In this work, we introduce a metric which automatically identifies the shortcomings of...

Full description

Saved in:
Bibliographic Details
Main Authors Mahfuz, Rehana, Guo, Yinyi, Sridhar, Arvind Krishna, Visser, Erik
Format Journal Article
LanguageEnglish
Published 06.09.2023
Subjects
Computer Science - Multimedia
Online AccessGet full text

Cover

More Information
Summary:Metrics to evaluate audio captions simply provide a score without much explanation regarding what may be wrong in case the score is low. Manual human intervention is needed to find any shortcomings of the caption. In this work, we introduce a metric which automatically identifies the shortcomings of an audio caption by detecting the misses and false alarms in a candidate caption with respect to a reference caption, and reports the recall, precision and F-score. Such a metric is very useful in profiling the deficiencies of an audio captioning model, which is a milestone towards improving the quality of audio captions.
DOI:10.48550/arxiv.2309.03326