SummaC : Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

• Published in: Transactions of the Association for Computational Linguistics Vol. 10; pp. 163 - 177
• Main Authors: Laban, Philippe, Schnabel, Tobias, Bennett, Paul N., Hearst, Marti A.
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 09.02.2022; MIT Press Journals, The; The MIT Press
• Subjects: Accuracy; Datasets; Documents; Hypotheses; Inference; Linguistics; Natural language; Natural language processing; Sentences; Summarization; Weight reduction
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00453

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaC that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called ( ry onsistency) which consists of six large inconsistency detection datasets. On this dataset, SummaC obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.