An ablation study on the use of publication venue quality to rank computer science departments

• Published in: Scientometrics Vol. 128; no. 8; pp. 4197 - 4218
• Main Authors: Maiti Aniruddha, Shi, Sai, Vucetic Slobodan
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Nature B.V 01.01.2023
• Subjects: Ablation; Clustering; Computer science; Correlation coefficient; Correlation coefficients; Decisions; Evaluation; Ranking; United States > US
• ISSN: 0138-9130; 1588-2861
• DOI: 10.1007/s11192-023-04733-2

This paper focuses on ranking computer science departments based on the quality of publications by the faculty in those departments. There are multiple strategies to convert publication lists into ranking scores for the departments. Important open questions include handling multi-author publications, inclusion criteria for publications and publication venues, accounting for the quality of publication venues, and accounting for the sub-areas of computer science. An ablation study is performed to evaluate the importance of different decisions for department ranking. The correlation between the resulting rankings and the peer assessment of computer science departments provided by the U.S. News was measured to evaluate the importance of different decisions. The results show that the selection of publication venues has the highest impact on the ranking. In contrast, decisions related to publication recency, multi-author publications, and clustering publications into subareas have less impact. Overall, Pearson’s correlation coefficient between the publication-based scores and the U.S. News ranking is above 0.90 for a large range of decisions, indicating a strong agreement between the objective measure and the subjective opinion of peers.