When is the probability ranking principle suboptimal?

• Published in: Journal of the American Society for Information Science Vol. 43; no. 1; pp. 1 - 14
• Main Authors: Gordon, Michael D., Lenk, Peter
• Format: Journal Article
• Language: English
• Published: Washington, D.C Wiley Subscription Services, Inc., A Wiley Company 01.01.1992; John Wiley & Sons; American Documentation Institute; Wiley Periodicals Inc
• Subjects: Calibration; Confidence Intervals (Statistics); Decision theory; Exact sciences and technology; Expected utility; Information and communication sciences; Information processing and retrieval; Information Retrieval; Information retrieval systems. Information and document management system; Information science; Information science. Documentation; Information storage; Information storage and retrieval; Information work; Mathematical Formulas; Parameter estimation; Probabilistic indexing; Probabilistic Models; Probability; Ranking; Relevance; Relevance (Information Retrieval); Science; Sciences and techniques of general use; Searching; Signal detection; Subject indexing; Technical services; User Satisfaction (Information); New York; United States > US; California; Condition; Relevance; Document retrieval; Document retrieval system; Decision theory; Prediction; Probabilistic model; Statistical decision; Optimization
• ISSN: 0002-8231; 1097-4571
• DOI: 10.1002/(SICI)1097-4571(199201)43:1<1::AID-ASI1>3.0.CO;2-5

The probability ranking principle retrieves documents in decreasing order of their predictive probabilities of relevance. Gordon and Lenk (1991) demonstrated that this principal is optimal within a signal detection—decision theory framework, and it maximizes the inquirer's expected utility for relevant documents. These results hold under three conditions: calibration, independent assessment of relevance by the inquirer, and certainty about the computed probabilities of relevance. We demonstrate that the probability ranking principle can be suboptimal with respect to expected utility when one of these conditions fails to hold. © 1992 John Wiley & Sons, Inc.