A limitation of the Cognitive Reflection Test: familiarity

• Published in: PeerJ (San Francisco, CA) Vol. 4; p. e2395
• Main Authors: Stieger, Stefan, Reips, Ulf-Dietrich
• Format: Journal Article
• Language: English
• Published: United States PeerJ. Ltd 06.09.2016; PeerJ, Inc; PeerJ Inc
• Subjects: Age; Age-stratified sample; Behavior; Children; Cognition & reasoning; Cognitive ability; Cognitive Reflection Test; Cognitive vs. intuitive reflection; Decision making; Demography; Dominance analysis; Familiarity; Neuroscience; Personality; Pharmacology; Psychiatry and Psychology; Research methodology; Researchers; Science; Siblings; Studies; Test experience; Validity; Age-stratified sample; Validity; Cognitive vs. intuitive reflection; Cognitive Reflection Test; Test experience; Dominance analysis
• ISSN: 2167-8359; 2167-8359
• DOI: 10.7717/peerj.2395

The Cognitive Reflection Test (CRT; Frederick, 2005) is a frequently used measure of cognitive vs. intuitive reflection. It is also a frequently found entertaining 'test' on the Internet. In a large age-stratified community-based sample (N = 2,272), we analyzed the impact of having already performed the CRT or any similar task in the past. Indeed, we found that 44% of participants had experiences with these tasks, which was reflected in higher CRT scores (Cohen's d = 0.41). Furthermore, experienced participants were different from naïve participants in regard to their socio-demographics (younger, higher educated, fewer siblings, more likely single or in a relationship than married, having no children). The best predictors of a high CRT score were the highest educational qualification (4.62% explained variance) followed by the experience with the task (3.06%). Therefore, we suggest using more recent multi-item CRTs with newer items and a more elaborated test construction.