Are Ranked Journal Lists Effective Determiners of Article Quality?

This study examines the effectiveness of the method of using the traditional method of counting articles in ranked venues (CARV) to evaluate scholarly output. CARV has been criticized for its lack of theoretical basis and performative effects, but it has never been empirically studied to determine i...

Full description

Saved in:
Bibliographic Details
Published inThe Journal of computer information systems Vol. 64; no. 5; pp. 622 - 635
Main Authors Cuellar, Michael J., Takeda, H., Truex, D. T.
Format Journal Article
LanguageEnglish
Published Stillwater Taylor & Francis Ltd 02.09.2024
Subjects
Classification
Effectiveness
Evaluation
Ratings & rankings
Scholarly publishing
Online AccessGet full text

Cover

More Information
Summary:This study examines the effectiveness of the method of using the traditional method of counting articles in ranked venues (CARV) to evaluate scholarly output. CARV has been criticized for its lack of theoretical basis and performative effects, but it has never been empirically studied to determine its effectiveness in correctly classifying scholarly output as to its quality. This study fills that gap by testing a set of six published journal lists to examine their ability to discern the quality of papers. We examine the consistency of quality across journals in each stratum, the ability of the lists to discriminate levels of quality in its strata and the ability of the method to correctly classify papers in strata based on quality. We find the journal lists substantially misclassify articles as to quality and are therefore problematic as evaluative mechanisms for scholarly ability.
ISSN:0887-4417
2380-2057
DOI:10.1080/08874417.2023.2240725