Automatic evaluation of summary on fidelity, conciseness and coherence for text summarization based on semantic link network

• Published in: Expert systems with applications Vol. 206; p. 117777
• Main Authors: Cao, Mengyun, Zhuge, Hai
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 15.11.2022
• Subjects: Evaluation method; Nature language processing; Semantic link network; Text modelling; Text Summarization; Text Summarization; Evaluation method; Text modelling; Nature language processing; Semantic link network
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2022.117777

•Automatic and accurate evaluation of summary is a key problem of text summarization research.•A framework for evaluating summary from fidelity, conciseness, and coherence.•An evaluation method based on the semantic link network representation of text.•The semantic link network plays an important role in representing and analyzing texts. Automatic evaluation of summary is one of the basic problems of automatic text summarization research. The commonly used approaches evaluate the informativeness, conciseness and coherence of summary separately based on different representations with weak semantics such as n-gram and vector of words. By transforming text into semantic link network, this paper proposes a general framework for automatically evaluating summary from the following aspects: (1) fidelity, the extent of a summary that conveys the core of its source text; (2) conciseness, the extent of non-redundancy within summary and, (3) coherence, the extent of relatedness between representations within summary, which are based on a uniform semantic representation of text. Experiments on the evaluation of the summarization of scientific papers and news show that the proposed framework achieves comparable or higher correlations with human judgments than the popular evaluation models. This research also verifies the role of the semantic link network in representing and analysing texts.