Ranking versus rating in peer review of research grant applications

The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approac...

Full description

Saved in:
Bibliographic Details
Published inPloS one Vol. 18; no. 10; p. e0292306
Main Authors Tamblyn, Robyn, Girard, Nadyne, Hanley, James, Habib, Bettina, Mota, Adrian, Khan, Karim M, Ardern, Clare L
Format Journal Article
LanguageEnglish
Published San Francisco Public Library of Science 05.10.2023
Public Library of Science (PLoS)
Subjects
Bias
Careers
Comparative analysis
Decision making
Evaluation
Funding
Grants-in-aid
Peer review
People and Places
Productivity
Quality assessment
Ranking
Ratings & rankings
Reliability
Research and Analysis Methods
Research grants
Research institutes
Reviews
Science Policy
Sex discrimination
Social Sciences
Canada
Online AccessGet full text

Cover

More Information
Summary:The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the Canadian Institutes of Health Research. Ranking was more reliable than rating and less susceptible to the characteristics of the review panel, such as level of expertise and experience, for both reliability and potential sources of bias. However, both rating and ranking penalized early career investigators and favoured older applicants. Sex bias was only evident for rating and only when the applicant's H-index was at the lower end of the H-index distribution. We conclude that when compared to rating, ranking provides a more reliable assessment of the quality of research applications, is not as influenced by reviewer expertise or experience, and is associated with fewer sources of bias. Research funding agencies should consider adopting ranking methods to improve the quality of funding decisions in health research.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
Competing Interests: The authors have declared that no competing interests exist.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0292306