SummEval: Re-evaluating Summarization Evaluation

• Published in: Transactions of the Association for Computational Linguistics Vol. 9; pp. 391 - 409
• Main Authors: Fabbri, Alexander R., Kryściński, Wojciech, McCann, Bryan, Xiong, Caiming, Socher, Richard, Radev, Dragomir
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.01.2021; MIT Press Journals, The; The MIT Press
• Subjects: Annotations; Automatic summarization; Crowdsourcing; Datasets; Linguistics; Summaries

The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 au...