Data-driven artificial intelligence to automate researcher assessment

• Published in: Scientometrics Vol. 126; no. 4; pp. 3265 - 3281
• Main Authors: Weber, Rosina O., Duarte, Kedma B.
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01.04.2021; Springer Nature B.V
• Subjects: Artificial intelligence; Automation; Careers; Classification; Cognition & reasoning; Computer Science; Curricula; Information Storage and Retrieval; Job descriptions; Learning algorithms; Library Science; Machine learning; Methodology; Placement; Reasoning; Case-based reasoning; C45 Neural Networks and Related Topics; 03–04 Explicit machine computation and programs; Simulation Methods; C53 Forecasting and Prediction Methods; 68T05 Learning and adaptive systems; C52 Model Evaluation, Validation, and Selection; Career trajectory; Researcher assessment; Machine learning; Leiden manifesto; Profiling systems; 01–08 Computational methods; 91E40 Memory and learning
• ISSN: 0138-9130; 1588-2861
• DOI: 10.1007/s11192-020-03859-x

This article describes how to utilize data-driven artificial intelligence (AI) to automate researcher assessment using data from profiling systems. We consider that a researcher assessment is done for a purpose and not divorced from a specific target placement. We formulate researcher assessment as a binary classification task, that is, a candidate researcher is classified as either fit or unfit for a given placement. For classifying researchers, we adopt case-based reasoning, a transparent artificial intelligence methodology that implements analogical reasoning, allows adaptation, machine learning, and explainability. This work addresses a human limitation through AI. Given a small number of candidates for a job or award and a clear job description, even if capable of selecting the best fit candidate, human decisions may be neither transparent nor reproducible. The approach in this article describes how to use AI methods to, from a job description, select the best fit candidate while considering career trajectories, providing explanations, and being reproducible. We describe the implementation of the methodology for a hypothetical placement in a real research institute from real but anonymized curriculum vitae from the Brazilian Lattes Database. We describe an experiment demonstrating that the purpose-oriented approach is more accurate than purpose-independent classifiers. The proposed methodology meets various principles from the Leiden Manifesto.