Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph

• Published in: Information (Basel) Vol. 15; no. 6; p. 328
• Main Authors: Nechakhin, Vladyslav, D’Souza, Jennifer, Eger, Steffen
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.06.2024
• Subjects: Alignment; Analysis; Annotations; Computational linguistics; Datasets; Knowledge representation; Language; Language processing; Large language models; Mimicry; Natural language; Natural language interfaces; Open Research Knowledge Graph; Performance evaluation; Recommender systems; Semantics; State-of-the-art reviews; structured summarization; Subject specialists; Germany
• ISSN: 2078-2489; 2078-2489
• DOI: 10.3390/info15060328

Structured science summaries or research contributions using properties or dimensions beyond traditional keywords enhance science findability. Current methods, such as those used by the Open Research Knowledge Graph (ORKG), involve manually curating properties to describe research papers’ contributions in a structured manner, but this is labor-intensive and inconsistent among human domain-expert curators. We propose using Large Language Models (LLMs) to automatically suggest these properties. However, it is essential to assess the readiness of LLMs like GPT-3.5, Llama 2, and Mistral for this task before their application. Our study performs a comprehensive comparative analysis between the ORKG’s manually curated properties and those generated by the aforementioned state-of-the-art LLMs. We evaluate LLM performance from four unique perspectives: semantic alignment with and deviation from ORKG properties, fine-grained property mapping accuracy, SciNCL embedding-based cosine similarity, and expert surveys comparing manual annotations with LLM outputs. These evaluations occur within a multidisciplinary science setting. Overall, LLMs show potential as recommendation systems for structuring science, but further fine-tuning is recommended to improve their alignment with scientific tasks and mimicry of human expertise.