What meta-analyses reveal about the replicability of psychological research
Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power, bias or heterogeneity? A large survey of 12,065 estimated effect sizes from 200 meta-analyses and nearly 8,000 papers is used to assess these key dimensions of replicability. First, our s...
Saved in:
Published in | Psychological bulletin Vol. 144; no. 12; p. 1325 |
---|---|
Main Authors | , , |
Format | Journal Article |
Language | English |
Published |
United States
01.12.2018
|
Online Access | Get more information |
Cover
Loading…
Abstract | Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power, bias or heterogeneity? A large survey of 12,065 estimated effect sizes from 200 meta-analyses and nearly 8,000 papers is used to assess these key dimensions of replicability. First, our survey finds that psychological research is, on average, afflicted with low statistical power. The median of median power across these 200 areas of research is about 36%, and only about 8% of studies have adequate power (using Cohen's 80% convention). Second, the median proportion of the observed variation among reported effect sizes attributed to heterogeneity is 74% (I2). Heterogeneity of this magnitude makes it unlikely that the typical psychological study can be closely replicated when replication is defined as study-level null hypothesis significance testing. Third, the good news is that we find only a small amount of average residual reporting bias, allaying some of the often-expressed concerns about the reach of publication bias and questionable research practices. Nonetheless, the low power and high heterogeneity that our survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology. (PsycINFO Database Record (c) 2018 APA, all rights reserved). |
---|---|
AbstractList | Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power, bias or heterogeneity? A large survey of 12,065 estimated effect sizes from 200 meta-analyses and nearly 8,000 papers is used to assess these key dimensions of replicability. First, our survey finds that psychological research is, on average, afflicted with low statistical power. The median of median power across these 200 areas of research is about 36%, and only about 8% of studies have adequate power (using Cohen's 80% convention). Second, the median proportion of the observed variation among reported effect sizes attributed to heterogeneity is 74% (I2). Heterogeneity of this magnitude makes it unlikely that the typical psychological study can be closely replicated when replication is defined as study-level null hypothesis significance testing. Third, the good news is that we find only a small amount of average residual reporting bias, allaying some of the often-expressed concerns about the reach of publication bias and questionable research practices. Nonetheless, the low power and high heterogeneity that our survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology. (PsycINFO Database Record (c) 2018 APA, all rights reserved). |
Author | Carter, Evan C Doucouliagos, Hristos Stanley, T D |
Author_xml | – sequence: 1 givenname: T D surname: Stanley fullname: Stanley, T D organization: Deakin Lab for the Meta-Analysis of Research (DeLMAR), School of Business, Deakin University – sequence: 2 givenname: Evan C surname: Carter fullname: Carter, Evan C organization: Human Research and Engineering Directorate, U.S. Army Research Laboratory, Aberdeen Proving Ground – sequence: 3 givenname: Hristos surname: Doucouliagos fullname: Doucouliagos, Hristos organization: Department of Economics, DeLMAR, and Alfred Deakin Institute for Citizenship and Globalisation, Deakin University |
BackLink | https://www.ncbi.nlm.nih.gov/pubmed/30321017$$D View this record in MEDLINE/PubMed |
BookMark | eNo1j81KxDAUhYMozo9ufADpC1STJjdpljL4hwNuFJfDTXNrK-m0NKnQt7egns2B74MDZ8NOj_2RGLsS_EZwaW7dFPgSoe0JWwsrbS4UwIptYvxauAEtz9lKclkILsyavXw0mLKOEuZ4xDBHitlI34QhQ9dPKUsNLWAIbYWuDW2as77OhjhXTR_6z4WGRUfCsWou2FmNIdLlX2_Z-8P92-4p378-Pu_u9jnKskg5gSxVXXpFZAXXXBswnhMI0qSNqT1JybUAKG3pHa-Be2cBFBIobRUvtuz6d3eYXEf-MIxth-N8-H9V_ABI-E3X |
CitedBy_id | crossref_primary_10_3389_fpsyg_2020_00947 crossref_primary_10_3390_brainsci11060814 crossref_primary_10_1177_2515245919884304 crossref_primary_10_1093_jeea_jvac053 crossref_primary_10_1146_annurev_psych_020821_094927 crossref_primary_10_1016_j_cognition_2023_105636 crossref_primary_10_1016_j_jbusres_2023_114189 crossref_primary_10_1073_pnas_2306281121 crossref_primary_10_3389_fnrgo_2021_643969 crossref_primary_10_1002_pld3_411 crossref_primary_10_1007_s13194_020_00317_6 crossref_primary_10_1080_00031305_2018_1505655 crossref_primary_10_1080_09297049_2023_2172150 crossref_primary_10_1002_jrsm_1704 crossref_primary_10_1038_s41562_021_01143_3 crossref_primary_10_1073_pnas_2120377119 crossref_primary_10_1111_bjir_12746 crossref_primary_10_1111_gcb_15972 crossref_primary_10_15626_MP_2023_3987 crossref_primary_10_2139_ssrn_3140006 crossref_primary_10_2139_ssrn_3480708 crossref_primary_10_1177_0886260521997945 crossref_primary_10_3389_fpsyg_2023_1253452 crossref_primary_10_1098_rsos_211028 crossref_primary_10_2139_ssrn_4706575 crossref_primary_10_1177_1745691619895036 crossref_primary_10_1111_joes_12598 crossref_primary_10_1177_2515245919858428 crossref_primary_10_1016_j_appet_2023_106586 crossref_primary_10_3389_fnhum_2022_933718 crossref_primary_10_1002_ejsp_3056 crossref_primary_10_3758_s13428_021_01644_z crossref_primary_10_1007_s13164_021_00531_y crossref_primary_10_1177_17470218211056772 crossref_primary_10_3389_fpsyg_2021_614236 crossref_primary_10_1080_02640414_2023_2269357 crossref_primary_10_3390_languages7040293 crossref_primary_10_1027_2151_2604_a000431 crossref_primary_10_1177_25152459221120427 crossref_primary_10_1371_journal_pone_0290084 crossref_primary_10_1080_1750984X_2021_1946835 crossref_primary_10_1002_aepp_13323 crossref_primary_10_1111_joes_12580 crossref_primary_10_1002_ejsp_2858 crossref_primary_10_1073_pnas_2202657119 crossref_primary_10_1016_j_ijpsycho_2021_01_018 crossref_primary_10_1016_j_jcrimjus_2021_101878 crossref_primary_10_1016_j_psfr_2020_09_001 crossref_primary_10_1002_jrsm_1332 crossref_primary_10_1080_1359432X_2023_2206571 crossref_primary_10_1016_j_cobeha_2020_09_005 crossref_primary_10_3390_jintelligence8040036 crossref_primary_10_1017_S0140525X18000882 crossref_primary_10_1080_00031305_2019_1678521 crossref_primary_10_1016_j_jclinepi_2019_06_014 crossref_primary_10_1016_j_jenvp_2022_101809 crossref_primary_10_3389_fpsyg_2022_974683 crossref_primary_10_1007_s11336_024_09980_7 crossref_primary_10_1016_j_ssci_2019_104564 crossref_primary_10_1098_rsos_220946 crossref_primary_10_3758_s13428_022_01817_4 crossref_primary_10_3389_fpsyg_2019_01970 crossref_primary_10_1002_ejsp_2748 crossref_primary_10_1177_1089268019880891 crossref_primary_10_1371_journal_pone_0236079 crossref_primary_10_1093_psyrad_kkad010 crossref_primary_10_2147_NSS_S286701 crossref_primary_10_1123_jpah_2021_0243 crossref_primary_10_1016_j_jpubeco_2021_104482 crossref_primary_10_1111_jomf_12693 crossref_primary_10_1080_1047840X_2023_2274429 crossref_primary_10_1080_21642850_2021_2012474 crossref_primary_10_1080_00273171_2021_1997701 crossref_primary_10_1214_23_AOAS1806 crossref_primary_10_1016_j_jml_2019_104047 crossref_primary_10_1186_s41235_021_00285_2 crossref_primary_10_1177_00332941241238208 crossref_primary_10_3758_s13428_020_01365_9 crossref_primary_10_1007_s10648_023_09716_6 crossref_primary_10_1016_j_jocrd_2023_100805 crossref_primary_10_1002_jrsm_1512 crossref_primary_10_36850_mr4 crossref_primary_10_3389_fpsyg_2019_02474 crossref_primary_10_1017_S2045796021000664 crossref_primary_10_1177_21677026211049366 crossref_primary_10_1016_j_shpsc_2019_101177 crossref_primary_10_1080_14459795_2019_1672769 crossref_primary_10_1016_j_paid_2019_109765 crossref_primary_10_1590_1983_3652_2023_46636 crossref_primary_10_3389_fpsyg_2019_00337 crossref_primary_10_1371_journal_pbio_3002423 crossref_primary_10_2139_ssrn_4132196 crossref_primary_10_5334_joc_72 crossref_primary_10_1016_j_jesp_2020_104060 crossref_primary_10_1038_s41559_023_02144_3 crossref_primary_10_1177_1745691619850561 crossref_primary_10_54359_ps_v13i70_202 crossref_primary_10_1007_s11229_022_03685_z crossref_primary_10_1016_j_ejpoleco_2021_102056 crossref_primary_10_1111_obr_13141 crossref_primary_10_1177_10892680211015635 crossref_primary_10_1027_2151_2604_a000515 crossref_primary_10_1016_j_newideapsych_2023_101014 crossref_primary_10_1186_s12915_022_01485_y crossref_primary_10_1002_icd_2348 crossref_primary_10_1017_S1366728920000747 crossref_primary_10_1186_s41235_024_00540_2 crossref_primary_10_1002_acp_3771 crossref_primary_10_1007_s10899_022_10118_6 crossref_primary_10_1177_19485506211061017 crossref_primary_10_1080_1750984X_2021_1966824 crossref_primary_10_1007_s10964_020_01294_y crossref_primary_10_1080_2372966X_2024_2346781 crossref_primary_10_1016_j_fufo_2021_100111 crossref_primary_10_5093_ejpalc2021a7 crossref_primary_10_1007_s11229_019_02433_0 crossref_primary_10_1038_s41562_023_01555_3 crossref_primary_10_1186_s40359_021_00548_z crossref_primary_10_1007_s10940_020_09459_5 crossref_primary_10_1177_0963721420904967 crossref_primary_10_3389_fnhum_2022_1009219 crossref_primary_10_1080_16066359_2020_1751130 crossref_primary_10_1007_s11266_022_00499_y crossref_primary_10_1027_1015_5759_a000705 crossref_primary_10_1027_1618_3169_a000517 crossref_primary_10_1177_02655322211018139 crossref_primary_10_1093_jcr_ucad062 crossref_primary_10_1017_S1368980024000697 crossref_primary_10_1080_03080188_2023_2234216 crossref_primary_10_1177_2515245920902370 crossref_primary_10_1371_journal_pone_0262809 crossref_primary_10_1002_jrsm_1529 crossref_primary_10_3389_fpsyg_2020_580111 crossref_primary_10_1027_2151_2604_a000386 crossref_primary_10_1177_25152459221109259 crossref_primary_10_17759_sps_2021120402 crossref_primary_10_1177_2515245919847196 crossref_primary_10_1136_bmjopen_2019_036349 crossref_primary_10_1002_jrsm_1424 crossref_primary_10_1177_2515245918815849 crossref_primary_10_1016_j_rmal_2023_100059 crossref_primary_10_1016_j_jsp_2024_101294 crossref_primary_10_1177_02654075221114029 crossref_primary_10_7554_eLife_58237 crossref_primary_10_1002_jrsm_1703 crossref_primary_10_1080_1750984X_2021_1977974 crossref_primary_10_1016_j_emj_2021_01_002 crossref_primary_10_1038_s41562_019_0787_z crossref_primary_10_1177_17456916221079611 crossref_primary_10_1177_17456916221091830 crossref_primary_10_3389_fpsyg_2022_967318 crossref_primary_10_1007_s40881_020_00084_3 crossref_primary_10_20982_tqmp_17_1_p024 crossref_primary_10_1080_14616734_2021_1918453 crossref_primary_10_1111_jasp_12913 crossref_primary_10_1177_1088868319887599 crossref_primary_10_1177_17456916221141357 crossref_primary_10_1177_1745691620964193 crossref_primary_10_1007_s10869_023_09890_y crossref_primary_10_1098_rsos_210544 crossref_primary_10_3758_s13423_019_01677_8 |
ContentType | Journal Article |
DBID | NPM |
DOI | 10.1037/bul0000169 |
DatabaseName | PubMed |
DatabaseTitle | PubMed |
DatabaseTitleList | PubMed |
Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database |
DeliveryMethod | no_fulltext_linktorsrc |
Discipline | Psychology |
EISSN | 1939-1455 |
ExternalDocumentID | 30321017 |
Genre | Journal Article |
GroupedDBID | --- --Z -DZ -ET -~X .-4 .GJ 07C 0R~ 123 186 29P 354 3EH 3O- 53G 5RE 5VS 6TJ 6TS 7RZ 85S 9M8 AAAHA AABCJ AAIKC AAMNW AAYOK ABCQX ABIVO ABNCP ABPPZ ABTAH ACGFO ACHQT ACNCT ACPQG ACTDY ADMHC ADMHG AEHFB AENEX AFFNX AGNAY AIDAL AJUXI ALEEW ALMA_UNASSIGNED_HOLDINGS ASUFR AWKKM AZXWR B-7 BKOMP CGNQK CS3 DU5 EPA F20 F5P FTD HVGLF HZ~ H~9 ISO L7B LPU LW5 MS~ MVM NHB NPM O9- OHT OPA OVD P-O P2P PQQKQ ROL RXW SES SPA TAE TEORI TN5 TWZ U5U UAO UBC UHB UHS ULE URZ VQA WH7 XIH XJT XKC XOL XZL YHZ YNT YXB YYP YYQ YZZ ZCA ZCG ZGI ZHY ZKG ZPI ZY4 |
ID | FETCH-LOGICAL-a382t-e5384f8d4ee910606757d0e51e6e677fde3306155898db0f50db9554ae5469402 |
IngestDate | Wed Oct 16 00:50:16 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 12 |
Language | English |
LinkModel | OpenURL |
MergedId | FETCHMERGED-LOGICAL-a382t-e5384f8d4ee910606757d0e51e6e677fde3306155898db0f50db9554ae5469402 |
PMID | 30321017 |
ParticipantIDs | pubmed_primary_30321017 |
PublicationCentury | 2000 |
PublicationDate | 2018-12-00 |
PublicationDateYYYYMMDD | 2018-12-01 |
PublicationDate_xml | – month: 12 year: 2018 text: 2018-12-00 |
PublicationDecade | 2010 |
PublicationPlace | United States |
PublicationPlace_xml | – name: United States |
PublicationTitle | Psychological bulletin |
PublicationTitleAlternate | Psychol Bull |
PublicationYear | 2018 |
SSID | ssj0007563 |
Score | 2.6805305 |
Snippet | Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power, bias or heterogeneity? A large survey of... |
SourceID | pubmed |
SourceType | Index Database |
StartPage | 1325 |
Title | What meta-analyses reveal about the replicability of psychological research |
URI | https://www.ncbi.nlm.nih.gov/pubmed/30321017 |
Volume | 144 |
hasFullText | |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnZ07T8MwEMctChLqgni_kQfWQJrErxFVoApEp1bqVtmNAwtth4JUPj13dp5VEY8liuK0SvJLzmf77n-EXBuecpaGKhBKhLjMqAMNbm6Asw1ZLHVoFOYOP_d5b5g8jtioCmt22SULczP5XJtX8h-qcAy4YpbsH8iWfwoHYB_4whYIw_ZXjFF3G2tAw507bRGLawAfKBXs441d-KB1K9QuBnZZxjeXJi8X-3mtO6lNo2jqAt153eF8ontQxQt3XWyos6yYEVWrgPw-waLr-sXH8_WclkFjsqEja4Eb1htIFasAxc0bFtRLOBavSlQziDDYZWsttc_1h1tw6wu-XEsN2fzNMYMONkKj8XPrimp20dQiLSGxpEcfZ3HyHlowHhdStbG4rS6iTbaLH64MM5y7MdglO_k4gd556Htkw073SbskszwgT0ifNuhTT586-hTo0wZ9Ostogz4t6B-S4cP9oNsL8tIYgY5ltAgs9FNJJtPEWvD3OA77RBpa1rHcciGy1MYxOqtMKpmaMGNhahR4jtqyhKskjI7I5nQ2tSeEdsBN0naSSTgtSViijHMyOTdGWM3iU3LsH8R47vVPxsUjOvu25Zy0q7fngmxl8MHZS_DeFubKofgCzf1GKQ |
link.rule.ids | 780 |
linkProvider | National Library of Medicine |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+meta-analyses+reveal+about+the+replicability+of+psychological+research&rft.jtitle=Psychological+bulletin&rft.au=Stanley%2C+T+D&rft.au=Carter%2C+Evan+C&rft.au=Doucouliagos%2C+Hristos&rft.date=2018-12-01&rft.eissn=1939-1455&rft.volume=144&rft.issue=12&rft.spage=1325&rft_id=info:doi/10.1037%2Fbul0000169&rft_id=info%3Apmid%2F30321017&rft_id=info%3Apmid%2F30321017&rft.externalDocID=30321017 |