Evaluating Semantic Similarity Methods to Build Semantic Predictability Norms of Reading Data

• Published in: Text, Speech, and Dialogue Vol. 12848; pp. 35 - 47
• Main Authors: Leal, Sidney, Casanova, Edresson, Paetzold, Gustavo, Aluísio, Sandra
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Cloze test; Language models; Semantic predictability
• ISBN: 303083526X; 9783030835262
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-83527-9_3

Predictability corpora built via Cloze task generally accompany eye-tracking data for the study of processing costs of linguistic structures in tasks of reading for comprehension. Two semantic measures are commonly calculated to evaluate expectations about forthcoming words: (i) the semantic fit of the target word with the previous context of a sentence, and (ii) semantic similarity scores that represent the semantic similarity between the target word and Cloze task responses for it. For Brazilian Portuguese (BP), there was no large eye-tracking corpora with predictability norms. The goal of this paper is to present a method to calculate the two semantic measures used in the first BP corpus of eye movements during silent reading of short paragraphs by undergraduate students. The method was informed by a large evaluation of both static and contextualized word embeddings, trained on large corpora of texts. Here, we make publicly available: (i) a BP corpus for a sentence-completion task to evaluate semantic similarity, (ii) a new methodology to build this corpus based on the scores of Cloze data taken from our project, and (iii) a hybrid method to compute the two semantic measures in order to build predictability corpora in BP.