A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

• Published in: Transactions of the Association for Computational Linguistics Vol. 9; pp. 1132 - 1146
• Main Authors: Deutsch, Daniel, Dror, Rotem, Roth, Dan
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 27.10.2021; MIT Press Journals, The; The MIT Press
• Subjects: Annotations; Automatic summarization; Bootstrapping; Confidence intervals; Correlation; Mathematical analysis; Permutations; Resampling; Simulation; Statistical analysis

The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics’ correlations r...