Cross-Linguistic Analysis of LLM Performance in Academic Title Generation

• Published in: Litera no. 5; pp. 297 - 319
• Main Author: Timokhov, Alexey Dmitrievich
• Format: Journal Article
• Language: English
• Published: 01.05.2025
• DOI: 10.25136/2409-8698.2025.5.74592
• ISSN: 2409-8698

This study evaluates the performance and nuances of several large language models — ChatGPT, Gemini, Mistral, and Llama — focusing on their capacity to generate academic article titles in both Russian and English. The analysis explores how these models perform in terms of linguistic quality and cross-linguistic adaptation, as well as their adherence to established conventions of different academic traditions. Drawing on a diverse corpus of 100 academic articles published between 2018 and 2023 across humanities and technical fields in both languages, the research examines the ability of these models to handle a wide spectrum of subject matter and genre-specific demands. Special attention is given to identifying differences between models, both in terms of stylistic and structural preferences and in the context of cross-linguistic adaptation when generating titles in Russian and English. Employing unified zero-shot prompts based on concise summaries of the original articles, the models generated alternative titles, which were subsequently analysed according to their level of detail, terminological accuracy, and stylistic conformity to academic conventions. The findings indicate that all tested models are generally capable of producing relevant and genre-appropriate titles; however, they exhibit clear differences in informativeness, granularity, and stylistic nuance, each demonstrating its own generation strategy. This paper offers the first comparative multilingual analysis of several large language models within the context of academic discourse, introducing the linguistic community and academia to an emerging type of research material — AI-generated texts, as opposed to conventionally authored texts produced directly by humans. Despite demonstrating considerable potential as preliminary aids in generating academic titles, variations in informativeness and style among models highlight the necessity for careful editorial oversight. AI-generated titles should thus be viewed as initial drafts that require refinement to ensure full compliance with academic standards.