How Consistent Are Meanings of “Evidence-Based”? A Comparative Review of 12 Clearinghouses that Rate the Effectiveness of Educational Programs
Clearinghouses set standards of scientific quality to vet existing research to determine how “evidence-based” an intervention is. This paper examines 12 educational clearinghouses to describe their effectiveness criteria, to estimate how consistently they rate the same program, and to probe why thei...
Saved in:
Published in | Review of educational research Vol. 94; no. 1; pp. 3 - 32 |
---|---|
Main Authors | , , |
Format | Journal Article |
Language | English |
Published |
Los Angeles, CA
SAGE Publications
01.02.2024
American Educational Research Association |
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Clearinghouses set standards of scientific quality to vet existing research to determine how “evidence-based” an intervention is. This paper examines 12 educational clearinghouses to describe their effectiveness criteria, to estimate how consistently they rate the same program, and to probe why their judgments differ. All the clearinghouses value random assignment, but they differ in how they treat its implementation, how they weight quasi-experiments, and how they value ancillary causal factors like independent replication and persisting effects. A total of 1359 programs were analyzed over 10 clearinghouses; 83% of them were assessed by a single clearinghouse and, of those rated by more than one, similar ratings were achieved for only about 30% of the programs. This high level of inconsistency seems to be mostly due to clearinghouses disagreeing about whether a high program rating requires effects that are replicated and/or temporally persisting. Clearinghouses exist to identify “evidence-based” programs, but the inconsistency in their recommendations of the same program suggests that identifying “evidence-based” interventions is still more of a policy aspiration than a reliable research practice. |
---|---|
AbstractList | Clearinghouses set standards of scientific quality to vet existing research to determine how “evidence-based” an intervention is. This paper examines 12 educational clearinghouses to describe their effectiveness criteria, to estimate how consistently they rate the same program, and to probe why their judgments differ. All the clearinghouses value random assignment, but they differ in how they treat its implementation, how they weight quasi-experiments, and how they value ancillary causal factors like independent replication and persisting effects. A total of 1359 programs were analyzed over 10 clearinghouses; 83% of them were assessed by a single clearinghouse and, of those rated by more than one, similar ratings were achieved for only about 30% of the programs. This high level of inconsistency seems to be mostly due to clearinghouses disagreeing about whether a high program rating requires effects that are replicated and/or temporally persisting. Clearinghouses exist to identify “evidence-based” programs, but the inconsistency in their recommendations of the same program suggests that identifying “evidence-based” interventions is still more of a policy aspiration than a reliable research practice. |
Author | Wadhwa, Mansi Zheng, Jingwen Cook, Thomas D. |
Author_xml | – sequence: 1 givenname: Mansi surname: Wadhwa fullname: Wadhwa, Mansi – sequence: 2 givenname: Jingwen surname: Zheng fullname: Zheng, Jingwen organization: George Washington University – sequence: 3 givenname: Thomas D. surname: Cook fullname: Cook, Thomas D. email: t-cook@northwestern.edu organization: Northwestern University |
BookMark | eNp1kMtKAzEUhoNUsK0-gLuA69Fc57KSWqoVKkrR9ZBmTtopbVKTaYu7PoNrfbk-iTNWcCGuzoHzfT-Hv4Na1llA6JySS04JuyKEi1gKzjilkrGYHaE2zbiMKBFxC7Wbe9QAJ6gTwpwQyuKUtdH70G1x39lQhgpshXse8AMoW9ppwM7g_e5jsCkLsBqiGxWg2O8-r3GvVpYr5VVVbgCPYVPCtqEpw_0FKF_bM7cOEHA1UxUeqwrqDfDAGNCNYyF8xw-Kta5DnFUL_OTd1KtlOEXHRi0CnP3MLnq5HTz3h9Ho8e6-3xtFmlNWRUnCYjLJhOFaJNooSdJkIhOTCko4m2RaFsAToWkmU20opFIZLZRRk0JwYgjvootD7sq71zWEKp-7ta8_CTnLKMk4lzGrKXqgtHcheDD5ypdL5d9ySvKm-vxP9bVzeXCCmsJv6v_CF2Glh3E |
CitedBy_id | crossref_primary_10_3390_educsci13080839 crossref_primary_10_1002_rev3_3448 crossref_primary_10_1038_s41539_023_00186_7 |
Cites_doi | 10.1177/001440290507100302 10.1086/505437 10.1057/s41599-019-0253-6 10.1007/s11121-022-01407-y 10.1007/s11121-005-5553-y 10.1080/02796015.2002.12086175 10.1332/174426421X16353477842207 10.3102/0002831207306743 10.3102/0034654309341374 10.59656/YD-G1574.001 10.3102/0162373708326283 10.1037/h0046016 10.3102/0034654316631117 10.3102/00346543063003345 |
ContentType | Journal Article |
Copyright | 2023 AERA |
Copyright_xml | – notice: 2023 AERA |
DBID | AAYXX CITATION |
DOI | 10.3102/00346543231152262 |
DatabaseName | CrossRef |
DatabaseTitle | CrossRef |
DatabaseTitleList | CrossRef |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Education |
EISSN | 1935-1046 |
EndPage | 32 |
ExternalDocumentID | 10_3102_00346543231152262 10.3102_00346543231152262 |
GrantInformation_xml | – fundername: National Science Foundation grantid: 100000001 funderid: https://doi.org/10.13039/100000001 |
GroupedDBID | -DZ -MK -TM -~X .2G .2L .GJ 0-V 01A 07C 09Z 0R~ 123 18M 1OL 1~K 29P 2KS 31S 31V 31W 31X 3EH 3V. 4.4 53G 56W 5VS 5WV 85S 8A4 8FI 8FJ 8FW 8G5 8R4 8R5 AAAHA AABOD AACKU AADIR AADUE AAGGD AAHSB AAJPV AAKTJ AAMFR AAMGE AANSI AAPEO AAQDB AARIX AATAA AAWLO AAYOK ABAWP ABBHK ABCCA ABEIX ABFXH ABHQH ABIVO ABJNI ABKRH ABLWH ABOPQ ABPNF ABPPZ ABQKF ABQPY ABQXT ABRHV ABRJW ABTAH ABUWG ABXSQ ABYTW ACAEP ACDXX ACFUR ACFZE ACGFO ACGFS ACHQT ACLZU ACNCT ACOFE ACOXC ACROE ACSIQ ACTDY ACTQU ACUFS ACUIR ACZOB ADACV ADBBV ADEIA ADEPB ADMHC ADNFJ ADPEE ADRRZ ADTOS ADUKL ADULT AEOBU AESMA AESZF AEUHG AEUIJ AEUPB AEVPJ AEWDL AEWHI AEXNY AFEET AFFNX AFKBI AFKRA AFKRG AFMOU AFQAA AFRQD AFUIA AGDVU AGHSJ AGKLV AGNAY AGNHF AGNWV AGWVZ AHHFK AHWHD AIDAL AIKWM AIMQZ AIOMO AJPNJ AJUXI AJUZI ALFTD ALIPV ALJHS ALMA_UNASSIGNED_HOLDINGS ALSLI ANDLU AQSKT ARALO ARTOV ARYUH ASPBG AS~ ATKJL AUTPY AUVAJ AVWKF AYPQM AZFZN AZQEC B-7 B8O B8S B8T B8Z BDZRT BENPR BKOMP BMVBW BPACV BPHCQ BVXVI BYIEH CAG CBRKF CCGJY CCPQU CEADM CJNVE COF CS3 DD0 DD~ DG~ DOPDO DU5 DV7 DV8 DWQXO EBS EJD F20 F5P FEDTE FHBDP FJW FYUFA G8K GENNL GNUQQ GROUPED_SAGE_PREMIER_JOURNAL_COLLECTION GUQSH HF~ HGD HVGLF HW5 HZ~ H~9 IPSME J8X JAAYA JAV JBMMH JBZCM JCYGO JENOY JHFFW JKQEH JLEZI JLXEF JPL JSODD JST L7B LIQON LWO M0P M2M M2O M4V MVM N9A NHB O9- OMK P-O P.B P2P PADUT PQEDU PQQKQ PRG PROAC PSYQQ Q1R Q2X Q7O Q7P Q7X ROL RWL RXW S01 S0X SA0 SCNPE SFB SFC SFF SFK SFN SFT SFX SGP SGU SGV SHB SHF SHM SQCSI SSDHQ TAE TN5 UBH UCJ UKHRP UKR VQA WH7 XKC XOL XXG XZL YCJ YYP YZZ ZCA ZCG ZPLXX ZPPRI ZY4 ~32 ~34 ~A~ AAAZS AAYXX ACJER ADMHG AEDXQ AGWNL CITATION H13 |
ID | FETCH-LOGICAL-c312t-77260b94f3c47cfa5087b57f841032b9c5de374c1958cf1e85afc4afabd430f03 |
ISSN | 0034-6543 |
IngestDate | Thu Oct 10 19:32:42 EDT 2024 Wed Oct 16 15:14:57 EDT 2024 Tue Jul 16 20:39:05 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 1 |
Keywords | mixed methods case studies evaluation research utilization descriptive analysis program evaluation experimental research clearinghouse causal identification what works policy analysis validity/reliability research methodology evidence-based policy |
Language | English |
LinkModel | OpenURL |
MergedId | FETCHMERGED-LOGICAL-c312t-77260b94f3c47cfa5087b57f841032b9c5de374c1958cf1e85afc4afabd430f03 |
PQID | 2910933562 |
PQPubID | 42090 |
PageCount | 30 |
ParticipantIDs | proquest_journals_2910933562 crossref_primary_10_3102_00346543231152262 sage_journals_10_3102_00346543231152262 |
PublicationCentury | 2000 |
PublicationDate | 20240200 2024-02-00 20240201 |
PublicationDateYYYYMMDD | 2024-02-01 |
PublicationDate_xml | – month: 2 year: 2024 text: 20240200 |
PublicationDecade | 2020 |
PublicationPlace | Los Angeles, CA |
PublicationPlace_xml | – name: Los Angeles, CA – name: Washington |
PublicationTitle | Review of educational research |
PublicationYear | 2024 |
Publisher | SAGE Publications American Educational Research Association |
Publisher_xml | – name: SAGE Publications – name: American Educational Research Association |
References | Polanin, Maynard, Dell 2017; 87 Borman, Slavin, Cheung, Chamberlain, Madden, Chambers 2007; 44 Flay, Biglan, Boruch, Castro, Gottfredson, Kellam, Ji 2005; 6 Parise, Corrin, Granito, Haider, Somers, Cerna 2017 Skindrud, Gersten 2006; 106 White 2019; 5 Sáenz, Fuchs, Fuchs 2005; 71 Somers, Haider 2017 Zheng, Wadhwa, Cook 2022; 23 Hedges, Waddington 1993; 63 Neuhoff A. (bibr23-00346543231152262) 2015 bibr34-00346543231152262 bibr21-00346543231152262 bibr1-00346543231152262 bibr11-00346543231152262 What Works Clearinghouse (bibr38-00346543231152262) 2017 Parise L. (bibr24-00346543231152262) 2017 bibr19-00346543231152262 bibr3-00346543231152262 bibr9-00346543231152262 bibr6-00346543231152262 bibr16-00346543231152262 bibr32-00346543231152262 bibr39-00346543231152262 bibr22-00346543231152262 bibr29-00346543231152262 bibr35-00346543231152262 bibr25-00346543231152262 bibr12-00346543231152262 Curtis A. (bibr14-00346543231152262) 2016 bibr15-00346543231152262 bibr2-00346543231152262 bibr5-00346543231152262 bibr40-00346543231152262 bibr8-00346543231152262 bibr18-00346543231152262 bibr28-00346543231152262 bibr10-00346543231152262 Curtis A. (bibr13-00346543231152262) 2015 bibr20-00346543231152262 bibr26-00346543231152262 bibr30-00346543231152262 bibr36-00346543231152262 bibr33-00346543231152262 bibr7-00346543231152262 bibr4-00346543231152262 bibr31-00346543231152262 bibr17-00346543231152262 bibr27-00346543231152262 What Works Clearinghouse (bibr37-00346543231152262) 2017 |
References_xml | – volume: 6 start-page: 151 issue: 3 year: 2005 end-page: 175 article-title: Standards of evidence: Criteria for efficacy, effectiveness and dissemination publication-title: Prevention Science contributor: fullname: Ji – volume: 87 start-page: 172 issue: 1 year: 2017 end-page: 203 article-title: Overviews in education research: A systematic review and analysis publication-title: Review of Educational Research contributor: fullname: Dell – volume: 106 start-page: 389 issue: 5 year: 2006 end-page: 408 article-title: An evaluation of two contrasting approaches for improving reading achievement in a large urban district publication-title: The Elementary School Journal contributor: fullname: Gersten – volume: 5 start-page: 47 issue: 1 year: 2019 article-title: The twenty-first century experimenting society: The four waves of the evidence revolution publication-title: Palgrave Communications contributor: fullname: White – volume: 71 start-page: 231 issue: 3 year: 2005 end-page: 247 article-title: Peer-assisted learning strategies for English language learners with learning disabilities publication-title: Exceptional Children contributor: fullname: Fuchs – volume: 23 start-page: 1343 issue: 8 year: 2022 end-page: 1358 article-title: How consistently do 13 clearinghouses identify social and behavioral development programs as “evidence-based”? publication-title: Prevention Science contributor: fullname: Cook – year: 2017 article-title: Two years of case management: Final findings from the Communities in Schools random assignment evaluation. MDRC publication-title: Project: Communities in Schools contributor: fullname: Cerna – year: 2017 article-title: Using integrated student supports to keep kids in school: A quasi-experimental evaluation of communities in schools. MDRC contributor: fullname: Haider – volume: 44 start-page: 701 issue: 3 year: 2007 end-page: 731 article-title: Final reading outcomes of the national randomized field trial of Success for All publication-title: American Education Research Journal contributor: fullname: Chambers – volume: 63 start-page: 345 issue: 3 year: 1993 end-page: 352 article-title: From evidence to knowledge to policy: Research synthesis for policy formation publication-title: Review of Educational Research contributor: fullname: Waddington – ident: bibr21-00346543231152262 – ident: bibr6-00346543231152262 – ident: bibr1-00346543231152262 – ident: bibr19-00346543231152262 – ident: bibr29-00346543231152262 doi: 10.1177/001440290507100302 – ident: bibr31-00346543231152262 doi: 10.1086/505437 – volume-title: The Quantum Opportunities Program: A randomized control evaluation year: 2015 ident: bibr13-00346543231152262 contributor: fullname: Curtis A. – ident: bibr30-00346543231152262 – ident: bibr8-00346543231152262 – ident: bibr9-00346543231152262 – ident: bibr36-00346543231152262 – ident: bibr39-00346543231152262 doi: 10.1057/s41599-019-0253-6 – volume-title: Procedures handbook year: 2017 ident: bibr38-00346543231152262 contributor: fullname: What Works Clearinghouse – ident: bibr40-00346543231152262 doi: 10.1007/s11121-022-01407-y – ident: bibr12-00346543231152262 – ident: bibr28-00346543231152262 – volume-title: The what works marketplace: Helping leaders use evidence to make smarter choices year: 2015 ident: bibr23-00346543231152262 contributor: fullname: Neuhoff A. – ident: bibr15-00346543231152262 doi: 10.1007/s11121-005-5553-y – ident: bibr16-00346543231152262 doi: 10.1080/02796015.2002.12086175 – ident: bibr17-00346543231152262 doi: 10.1332/174426421X16353477842207 – ident: bibr34-00346543231152262 – ident: bibr25-00346543231152262 – ident: bibr10-00346543231152262 – ident: bibr4-00346543231152262 doi: 10.3102/0002831207306743 – ident: bibr22-00346543231152262 – volume-title: The Quantum Opportunities Program: A randomized control evaluation year: 2016 ident: bibr14-00346543231152262 contributor: fullname: Curtis A. – ident: bibr2-00346543231152262 – ident: bibr33-00346543231152262 doi: 10.3102/0034654309341374 – ident: bibr35-00346543231152262 doi: 10.59656/YD-G1574.001 – ident: bibr3-00346543231152262 doi: 10.3102/0162373708326283 – ident: bibr7-00346543231152262 doi: 10.1037/h0046016 – volume-title: Standards handbook year: 2017 ident: bibr37-00346543231152262 contributor: fullname: What Works Clearinghouse – ident: bibr11-00346543231152262 – ident: bibr26-00346543231152262 doi: 10.3102/0034654316631117 – ident: bibr5-00346543231152262 – ident: bibr27-00346543231152262 – ident: bibr20-00346543231152262 – ident: bibr32-00346543231152262 – year: 2017 ident: bibr24-00346543231152262 publication-title: Project: Communities in Schools contributor: fullname: Parise L. – ident: bibr18-00346543231152262 doi: 10.3102/00346543063003345 |
SSID | ssj0012682 |
Score | 2.483673 |
Snippet | Clearinghouses set standards of scientific quality to vet existing research to determine how “evidence-based” an intervention is. This paper examines 12... |
SourceID | proquest crossref sage |
SourceType | Aggregation Database Publisher |
StartPage | 3 |
SubjectTerms | Aspiration Clearinghouses Education Evidence |
Title | How Consistent Are Meanings of “Evidence-Based”? A Comparative Review of 12 Clearinghouses that Rate the Effectiveness of Educational Programs |
URI | https://journals.sagepub.com/doi/full/10.3102/00346543231152262 https://www.proquest.com/docview/2910933562 |
Volume | 94 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3NbtQwELaW7YUL4le0FOQDCIlVStZ2_k5o2VJVSEWlakU5RbZja09ZxGZViVOfgTM8DK_SJ2Emdv7oIhUuUZTMWsnOF3tmPPMNIc-FLpLYcBuAbRsGIlNxkClQSBpFnKlCCVnXVxx9iA_PxPvz6Hw0-tXLWlpXak9_21hX8j9ahWugV6yS_QfNtoPCBTgH_cIRNAzHG-kY-8HVHTdBVWU1mSFDv5Fl3YkTjMAmkWHetA4N3sKaVTSX91_wg7oqveP_PmkLWaZsMseOEjDWYrleIRHEQlaTEzBNa1vVsR43U6VPT2wCi8cu6WvVt3y7oU1P0nMNtTHpT7JYXEhXRATv1YW1jc8chue56KrX5uAidGlOk_29fhCDiSbvebB7tjlOGXIRYAGsW7DcLJ1xpE_1sUs_jbteyQO4ujmZ9xZ3F0v9c9kAE5fVeZYc2eU4QwYiMEtZt0a2mYteOr8me4tssSSLojHZmn0-_njUbmWxOHWU9f493NY6DvL62iBD46jzeHpJhrXdc3qX3PEOC5059N0jI1Pex17fXocPyHdAIe1QCJKGNiikS0uvLn8M8Xd1-fMNndEe7qgDB0pPGR3ijiLuKOIOzgwd4A5_0MMdbXD3kJwdvDudHwa-00eg-ZRV6OLFocqE5Vok2krwGhIVJTYVyPeoMh0VhidCIzOStlOTRtJqIa1UheChDfkjMi6XpXlMaBgbY2MZKSm5MGGhQlUkyqYm1EZonW2TV81fnH9xhC45OMKblbpNdhsl5P67X-UsQwo2HuHtl6iY7tZfB9q5seQTcrv7QHbJuPq6Nk_B7q3UM4-s3w90p1g |
link.rule.ids | 315,786,790,27955,27956 |
linkProvider | SAGE Publications |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Consistent+Are+Meanings+of+%E2%80%9CEvidence-Based%E2%80%9D%3F+A+Comparative+Review+of+12+Clearinghouses+that+Rate+the+Effectiveness+of+Educational+Programs&rft.jtitle=Review+of+educational+research&rft.au=Wadhwa%2C+Mansi&rft.au=Zheng%2C+Jingwen&rft.au=Cook%2C+Thomas+D.&rft.date=2024-02-01&rft.pub=SAGE+Publications&rft.issn=0034-6543&rft.eissn=1935-1046&rft.volume=94&rft.issue=1&rft.spage=3&rft.epage=32&rft_id=info:doi/10.3102%2F00346543231152262&rft.externalDocID=10.3102_00346543231152262 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0034-6543&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0034-6543&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0034-6543&client=summon |