Underpowered samples, false negatives, and unconscious learning

• Published in: Psychonomic bulletin & review Vol. 23; no. 1; pp. 87 - 102
• Main Authors: Vadillo, Miguel A., Konstantinidis, Emmanouil, Shanks, David R.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2016; Springer Nature B.V
• Subjects: Adult; Awareness - physiology; Behavior; Behavioral Science and Psychology; Cognitive Psychology; Confidence intervals; Cues; Data Interpretation, Statistical; Economic models; Experiments; Female; Humans; Hypotheses; Laboratories; Learning - physiology; Male; Meta-analysis; Meta-Analysis as Topic; Psychology; Researchers; Studies; Systematic review; Theoretical Review; Unconscious (Psychology); Contextual cuing; Implicit learning; Null hypothesis Significance testing· Statistical power; False negatives
• ISSN: 1069-9384; 1531-5320
• DOI: 10.3758/s13423-015-0892-6

The scientific community has witnessed growing concern about the high rate of false positives and unreliable results within the psychological literature, but the harmful impact of false negatives has been largely ignored. False negatives are particularly concerning in research areas where demonstrating the absence of an effect is crucial, such as studies of unconscious or implicit processing. Research on implicit processes seeks evidence of above-chance performance on some implicit behavioral measure at the same time as chance-level performance (that is, a null result) on an explicit measure of awareness. A systematic review of 73 studies of contextual cuing, a popular implicit learning paradigm, involving 181 statistical analyses of awareness tests, reveals how underpowered studies can lead to failure to reject a false null hypothesis. Among the studies that reported sufficient information, the meta-analytic effect size across awareness tests was d z = 0.31 (95 % CI 0.24–0.37), showing that participants’ learning in these experiments was conscious. The unusually large number of positive results in this literature cannot be explained by selective publication. Instead, our analyses demonstrate that these tests are typically insensitive and underpowered to detect medium to small, but true, effects in awareness tests. These findings challenge a widespread and theoretically important claim about the extent of unconscious human cognition.