EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models

Events serve as fundamental units of occurrence within various contexts. The processing of event semantics in textual information forms the basis of numerous natural language processing (NLP) applications. Recent studies have begun leveraging large language models (LLMs) to address event semantic pr...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Tao, Zhengwei, Jin, Zhi, Bai, Xiaoying, Zhao, Haiyan, Feng, Yanlin, Li, Jia, Hu, Wenpeng
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 24.05.2023
Subjects
Large language models
Natural language processing
Semantics
Online AccessGet full text

Cover

More Information
Summary:Events serve as fundamental units of occurrence within various contexts. The processing of event semantics in textual information forms the basis of numerous natural language processing (NLP) applications. Recent studies have begun leveraging large language models (LLMs) to address event semantic processing. However, the extent that LLMs can effectively tackle these challenges remains uncertain. Furthermore, the lack of a comprehensive evaluation framework for event semantic processing poses a significant challenge in evaluating these capabilities. In this paper, we propose an overarching framework for event semantic processing, encompassing understanding, reasoning, and prediction, along with their fine-grained aspects. To comprehensively evaluate the event semantic processing abilities of models, we introduce a novel benchmark called EVEVAL. We collect 8 datasets that cover all aspects of event semantic processing. Extensive experiments are conducted on EVEVAL, leading to several noteworthy findings based on the obtained results.
ISSN:2331-8422