Illusory inconsistencies in judgment: Stimulus-evoked reference sets and between-subjects designs

• Published in: Psychonomic bulletin & review Vol. 26; no. 2; pp. 647 - 653
• Main Authors: Leong, Lim M., McKenzie, Craig R. M., Sher, Shlomi, Müller-Trede, Johannes
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2019; Springer Nature B.V
• Subjects: Adult; Behavioral Science and Psychology; Between-subjects design; Brief Report; Cognitive Psychology; Data Interpretation, Statistical; Experiments; Female; Humans; Illusions - physiology; Judgment - physiology; Male; Mathematical Concepts; Neuropsychological Tests - standards; Psychology; Ratings & rankings; Evoked reference set; Replication; Between-subjects design; Context effect
• ISSN: 1069-9384; 1531-5320; 1531-5320
• DOI: 10.3758/s13423-019-01585-x

Asked to judge the subjective size of numbers in a between-subjects design, participants rated 9 as larger than 221 (Birnbaum, 1999). The 9 > 221 effect seems to indicate that different stimuli evoke different contexts for comparison, and sounds a warning for the interpretation of between-subjects comparisons. We show that, contrary to appearances, the effect is not a result of stimulus-evoked reference sets. Instead, it is an artifact of the original 1–10 response scale and task instructions, which encourage a conflation of the response scale and the reference set. When ratings are expressed on a 1–1000 scale, or on a non-numerical slider scale, the effect reverses. However, we also show that stimuli can evoke their own comparative contexts, generating illusions of inconsistency in between-subjects designs. We report two novel findings – a 9 > 009 effect and a -2 > 2 effect – which are best explained by stimulus-evoked reference sets. Thus, while revealing that the 9 > 221 effect is an artifact of the original response scale, our study ultimately affirms Birnbaum’s warning about the comparison of between-subjects ratings.