Identifying the Causer of the Event: Evaluating the Performance of ChatGPT on Thematic Roles Recognition

• Published in: Journal of Umm Al-Qura University for Language Sciences and Literature no. 34; pp. 90 - 99
• Main Author: Aseri, Yahya
• Format: Journal Article
• Language: English
• Published: 30.12.2024
• ISSN: 1658-8126; 1658-8126
• DOI: 10.54940/ll90229146

This study aims to test the capability of large language models (LLMS), specifically the ChatGPT model, to determine the causer of an event without relying on morphological and syntactic algorithms and lexical resources. The model's ability was evaluated on roughly 200 sentences containing 43 verbs/derivatives of verbs that lexically require a causer in the event structure. The performance of the model was then compared to a gold standard model annotated by two linguists. The results of this study revealed the limited ability of this artificial model to determine the causer of an event. The model achieved an average F1-Score of 0.53 in identifying the event-relevant word and an average F1-Score of 0.63 in determining the causer of the event, compared to human performance. This performance indicates that the ChatGPT model, as one of the large language models, while demonstrating text generation capabilities approaching human-level performance, still lacks the ability to understand and comprehend semantic relationships within a sentence to the same extent as humans do. Nevertheless, the study suggests utilizing this model in semantic role labelling with human intervention by presenting the results to a specialist for review and correction, which may aid in data annotation for machine learning.