How experts’ own inconsistency relates to their confidence and between-expert disagreement

• Published in: Scientific reports Vol. 12; no. 1; p. 9273
• Main Authors: Litvinova, Aleksandra, Kurvers, Ralf H. J. M., Hertwig, Ralph, Herzog, Stefan M.
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 03.06.2022; Nature Publishing Group; Nature Portfolio
• Subjects: 631/114/2397; 631/477; Agreements; Cancer; Confidence; Datasets; Decision making; Humanities and Social Sciences; Investigations; multidisciplinary; Science; Science (multidisciplinary); Spine
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-022-12847-5

People routinely rely on experts’ advice to guide their decisions. However, experts are known to make inconsistent judgments when judging the same case twice. Previous research on expert inconsistency has largely focused on individual or situational factors; here we focus directly on the cases themselves. First, using a theoretical model, we study how within-expert inconsistency and confidence are related to how strongly experts agree on a case. Second, we empirically test the model’s predictions in two real-world datasets with a diagnostic ground truth from follow-up research: diagnosticians rating the same mammograms or images of the lower spine twice. Our modeling and empirical analyses converge on the same novel results: The more experts disagree in their initial decisions about a case (i.e., as consensus decreases), the less confident individual experts are in their initial decision—despite not knowing the level of consensus—and the more likely they are to judge that same case differently when facing it again months later, regardless of whether the expert consensus is correct. Our results suggest the following advice when faced with two conflicting decisions from a single expert: In the absence of more predictive cues, choose the more confident decision.