Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review
To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclu...
Saved in:
Published in | Journal of clinical epidemiology Vol. 164; pp. 1 - 8 |
---|---|
Main Authors | , , , , , , , , , , , , |
Format | Journal Article |
Language | English |
Published |
United States
Elsevier Limited
01.12.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR).
Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.
From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).
A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive. |
---|---|
AbstractList | To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR).
Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.
From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).
A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive. OBJECTIVESTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR).METHODSRecords from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.RESULTSFrom 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).CONCLUSIONA screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive. |
Author | Wells, George Glennon, Vanessa Tugwell, Peter Tomlinson, George Hazlewood, Glen S Kelly, Shannon E Buchbinder, Rachelle Deardon, Rob Elliott, Jesse Pardo, Jordi Pardo Kamso, Mohammed Mujaab Sajobi, Tolulope Whittle, Samuel L |
Author_xml | – sequence: 1 givenname: Mohammed Mujaab surname: Kamso fullname: Kamso, Mohammed Mujaab email: mohammedmujaab.kamso@ucalgary.ca organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada. Electronic address: mohammedmujaab.kamso@ucalgary.ca – sequence: 2 givenname: Jordi Pardo surname: Pardo fullname: Pardo, Jordi Pardo organization: Centre for Practice-Changing Research, Ottawa Hospital Research Institute, The Ottawa Hospital - General Campus, Ottawa, Canada – sequence: 3 givenname: Samuel L surname: Whittle fullname: Whittle, Samuel L organization: Department Rheumatology, The Queen Elizabeth Hospital, Adelaide, Australia; School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia – sequence: 4 givenname: Rachelle surname: Buchbinder fullname: Buchbinder, Rachelle organization: School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia – sequence: 5 givenname: George surname: Wells fullname: Wells, George organization: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada – sequence: 6 givenname: Vanessa surname: Glennon fullname: Glennon, Vanessa organization: School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia – sequence: 7 givenname: Peter surname: Tugwell fullname: Tugwell, Peter organization: Department of Medicine, University of Ottawa, Ottawa, Canada; Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; Department of Medicine, University of Ottawa, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Bruyere Research institute, Ottawa, Canada – sequence: 8 givenname: Rob surname: Deardon fullname: Deardon, Rob organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada; Department of Mathematics & Statistics, Faculty of Science, University of Calgary, Calgary, Canada – sequence: 9 givenname: Tolulope surname: Sajobi fullname: Sajobi, Tolulope organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada – sequence: 10 givenname: George surname: Tomlinson fullname: Tomlinson, George organization: Department of Medicine, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada – sequence: 11 givenname: Jesse surname: Elliott fullname: Elliott, Jesse organization: Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada – sequence: 12 givenname: Shannon E surname: Kelly fullname: Kelly, Shannon E organization: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada – sequence: 13 givenname: Glen S surname: Hazlewood fullname: Hazlewood, Glen S organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada |
BackLink | https://www.ncbi.nlm.nih.gov/pubmed/37865299$$D View this record in MEDLINE/PubMed |
BookMark | eNpdkUFPHSEUhUljU5-2f8FM4qabeb0wwwBL89KqiYmbdk0YBlomDDxhRqPL_nIZn9qkK8I537mXcE7QUYjBIHSGYYsBd9_G7ai9C2bvtgRIU8QtAPuANpgzXlNB8BHaABe0bhvaHaOTnEcAzIDRT-i4YbyjRIgN-rtL8WGoc1ySduF3pcJQqWWOk5pdDJVV2nk3q9kM1fzHVG4wYXbW6YO90tqrnP9J0VapyHFyTyWjY5hT9H6NJ6d8rlxJVd7dr8uSuXfm4TP6aItjvryep-jXj-8_d1f1ze3l9e7iptaYc1GLzrC2o9D3TDNqregoxlYYoVqt6cBpW-6a2RcfLNF9Y4Z24KzpMbG4b07R18PcfYp3i8mznFzWxnsVTFyyJJwDCEaAFfT8P3QsHxTK6yQRBSIt76BQ3YHSKeacjJX75CaVHiUGubYkR_nWklxbWnV4GX_2On7pJzO8x95qaZ4B1naVNQ |
Cites_doi | 10.1186/s13643-015-0066-7 10.7326/0003-4819-147-4-200708210-00179 10.3899/jrheum.220209 10.1016/j.jclinepi.2022.03.019 10.1016/j.jclinepi.2020.11.003 10.1093/jamia/ocx053 10.1037/h0031619 10.1186/s12874-022-01507-x 10.1177/001316446002000104 10.1016/j.jclinepi.2021.12.005 10.1136/bmj.i1777 10.1186/s12916-016-0596-4 10.1136/bmj.f139 10.1002/art.38202 10.7326/M22-0514 |
ContentType | Journal Article |
Copyright | Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved. 2023. The Authors |
Copyright_xml | – notice: Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved. – notice: 2023. The Authors |
DBID | NPM AAYXX CITATION 3V. 7QL 7QP 7RV 7T2 7T7 7TK 7U7 7U9 7X7 7XB 88C 88E 8AO 8C1 8FD 8FI 8FJ 8FK 8G5 ABUWG AFKRA AZQEC BENPR C1K CCPQU DWQXO FR3 FYUFA GHDGH GNUQQ GUQSH H94 K9. KB0 M0S M0T M1P M2O M7N MBDVC NAPCQ P64 PQEST PQQKQ PQUKI Q9U 7X8 |
DOI | 10.1016/j.jclinepi.2023.10.007 |
DatabaseName | PubMed CrossRef ProQuest Central (Corporate) Bacteriology Abstracts (Microbiology B) Calcium & Calcified Tissue Abstracts Nursing & Allied Health Database (ProQuest) Health and Safety Science Abstracts (Full archive) Industrial and Applied Microbiology Abstracts (Microbiology A) Neurosciences Abstracts Toxicology Abstracts Virology and AIDS Abstracts Health & Medical Collection ProQuest Central (purchase pre-March 2016) Healthcare Administration Database (Alumni) Medical Database (Alumni Edition) ProQuest Pharma Collection Public Health Database Technology Research Database Hospital Premium Collection Hospital Premium Collection (Alumni Edition) ProQuest Central (Alumni) (purchase pre-March 2016) Research Library (Alumni Edition) ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials AUTh Library subscriptions: ProQuest Central Environmental Sciences and Pollution Management ProQuest One Community College ProQuest Central Korea Engineering Research Database Health Research Premium Collection Health Research Premium Collection (Alumni) ProQuest Central Student Research Library Prep AIDS and Cancer Research Abstracts ProQuest Health & Medical Complete (Alumni) Nursing & Allied Health Database (Alumni Edition) Health & Medical Collection (Alumni Edition) Health Management Database (Proquest) Medical Database Research Library Algology Mycology and Protozoology Abstracts (Microbiology C) Research Library (Corporate) Nursing & Allied Health Premium Biotechnology and BioEngineering Abstracts ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central Basic MEDLINE - Academic |
DatabaseTitle | PubMed CrossRef Research Library Prep ProQuest Central Student Technology Research Database ProQuest Central Essentials ProQuest Health & Medical Complete (Alumni) ProQuest Central (Alumni Edition) ProQuest One Community College Research Library (Alumni Edition) ProQuest Pharma Collection Environmental Sciences and Pollution Management ProQuest Central Health Research Premium Collection Health and Medicine Complete (Alumni Edition) ProQuest Central Korea Bacteriology Abstracts (Microbiology B) Algology Mycology and Protozoology Abstracts (Microbiology C) AIDS and Cancer Research Abstracts ProQuest Research Library Health & Safety Science Abstracts Industrial and Applied Microbiology Abstracts (Microbiology A) ProQuest Medical Library (Alumni) ProQuest Public Health Virology and AIDS Abstracts ProQuest Central Basic Toxicology Abstracts ProQuest One Academic Eastern Edition ProQuest Health Management ProQuest Nursing & Allied Health Source ProQuest Hospital Collection Health Research Premium Collection (Alumni) Neurosciences Abstracts ProQuest Hospital Collection (Alumni) Biotechnology and BioEngineering Abstracts Nursing & Allied Health Premium ProQuest Health & Medical Complete ProQuest Medical Library ProQuest One Academic UKI Edition ProQuest Health Management (Alumni Edition) ProQuest Nursing & Allied Health Source (Alumni) Engineering Research Database ProQuest One Academic Calcium & Calcified Tissue Abstracts ProQuest Central (Alumni) MEDLINE - Academic |
DatabaseTitleList | PubMed MEDLINE - Academic Research Library Prep |
Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: BENPR name: ProQuest Central url: https://www.proquest.com/central sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Medicine |
EISSN | 1878-5921 |
EndPage | 8 |
ExternalDocumentID | 10_1016_j_jclinepi_2023_10_007 37865299 |
Genre | Journal Article Review |
GroupedDBID | --- --K --M -~X .1- .55 .FO .GJ .~1 0R~ 1B1 1P~ 1RT 1~. 1~5 29K 3V. 4.4 457 4CK 4G. 53G 5GY 5RE 5VS 7-5 71M 7RV 7X7 88E 8AO 8C1 8FI 8FJ 8G5 8P~ 9JM 9JO AABNK AACTN AAEDT AAEDW AAFJI AAIKJ AAKOC AALRI AAOAW AAQFI AAQXK AAWTL AAXKI AAXUO AAYJJ ABBQC ABFNM ABIVO ABJNI ABLJU ABMAC ABMMH ABMZM ABOCM ABUWG ABXDB ACDAQ ACGFS ACIUM ACPRK ACRLP ADBBV ADEZE ADMUD AEBSH AEKER AENEX AEVXI AFCTW AFFNX AFJKZ AFKRA AFKWA AFRAH AFRHN AFTJW AFXIZ AGHFR AGUBO AGYEJ AHHHB AHMBA AIEXJ AIKHN AITUG AJOXV AJRQY AJUYK AKRWK ALIPV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ ANZVX AOMHK AQUVI ASPBG AVARZ AVWKF AXJTR AZFZN AZQEC BENPR BKEYQ BKOJK BLXMC BNPGV BPHCQ BVXVI CCPQU CS3 D-I DU5 DWQXO EBS EFJIC EJD EMOBN EO8 EO9 EP2 EP3 EX3 F3I F5P FDB FEDTE FGOYB FIRID FNPLU FYGXN FYUFA G-2 G-Q GBLVA GNUQQ GUQSH HEH HMCUK HMK HMO HVGLF HZ~ IHE J1W KOM L7B M0T M1P M29 M2O M3W M41 MO0 N9A NAPCQ NPM O-L O9- OAUVE OD~ OHT OO0 OZT P-8 P-9 P2P PC. PQQKQ PRBVW PROAC PSQYO Q38 R2- RIG ROL RPZ SAE SCC SDF SDG SDP SEL SES SEW SPCBC SSB SSH SSO SSZ SV3 T5K UAP UKHRP WOW WUQ X7M XPP YHZ Z5R ZGI ~G- AAYXX ACRPL ADNMO CITATION 7QL 7QP 7T2 7T7 7TK 7U7 7U9 7XB 8FD 8FK C1K FR3 H94 K9. M7N MBDVC P64 PQEST PQUKI Q9U 7X8 |
ID | FETCH-LOGICAL-c1889-96e74650bb7c75ff96511f9e9a4cc5d854511c7fbb7c70f2cb3ed4d873b12f1b3 |
IEDL.DBID | 8C1 |
ISSN | 0895-4356 |
IngestDate | Wed Dec 04 08:19:58 EST 2024 Thu Dec 05 17:25:13 EST 2024 Fri Dec 06 02:00:10 EST 2024 Sat Nov 02 12:25:19 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Keywords | Crowdsourcing Living systematic reviews Automation Rheumatoid arthritis Machine learning Systematic reviews Randomized controlled trials (RCTs) |
Language | English |
License | Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved. |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c1889-96e74650bb7c75ff96511f9e9a4cc5d854511c7fbb7c70f2cb3ed4d873b12f1b3 |
Notes | ObjectType-Article-2 SourceType-Scholarly Journals-1 ObjectType-Feature-3 content type line 23 ObjectType-Review-1 ObjectType-Undefined-4 |
ORCID | 0000-0001-7417-7691 0000-0001-7709-3741 0000-0001-5062-0556 0000-0002-3985-5335 0000-0002-0597-0933 0000-0003-0436-1850 0000-0003-2792-7706 |
PMID | 37865299 |
PQID | 2900924860 |
PQPubID | 105585 |
PageCount | 8 |
ParticipantIDs | proquest_miscellaneous_2880097207 proquest_journals_2900924860 crossref_primary_10_1016_j_jclinepi_2023_10_007 pubmed_primary_37865299 |
PublicationCentury | 2000 |
PublicationDate | 2023-Dec 2023-12-00 20231201 |
PublicationDateYYYYMMDD | 2023-12-01 |
PublicationDate_xml | – month: 12 year: 2023 text: 2023-Dec |
PublicationDecade | 2020 |
PublicationPlace | United States |
PublicationPlace_xml | – name: United States – name: Elmsford |
PublicationTitle | Journal of clinical epidemiology |
PublicationTitleAlternate | J Clin Epidemiol |
PublicationYear | 2023 |
Publisher | Elsevier Limited |
Publisher_xml | – sequence: 0 name: Elsevier Limited |
References | Lee (10.1016/j.jclinepi.2023.10.007_bib14) 2022; 147 Xin (10.1016/j.jclinepi.2023.10.007_bib2) 2022; 22 Ducreux (10.1016/j.jclinepi.2023.10.007_bib20) 2014; 66 Jonnalagadda (10.1016/j.jclinepi.2023.10.007_bib23) 2015; 4 Hazlewood (10.1016/j.jclinepi.2023.10.007_bib12) 2022; 49 Singh (10.1016/j.jclinepi.2023.10.007_bib17) 2017; 3 Conaghan (10.1016/j.jclinepi.2023.10.007_bib21) 2014; 66 Tsafnat (10.1016/j.jclinepi.2023.10.007_bib9) 2013; 346 Thomas (10.1016/j.jclinepi.2023.10.007_bib7) 2021; 133 Cohen (10.1016/j.jclinepi.2023.10.007_bib8) 2010 Vandvik (10.1016/j.jclinepi.2023.10.007_bib1) 2016; 14 Shojania (10.1016/j.jclinepi.2023.10.007_bib5) 2007; 147 Hazlewood (10.1016/j.jclinepi.2023.10.007_bib11) 2020; 2020 Khalil (10.1016/j.jclinepi.2023.10.007_bib6) 2022; 144 Fleiss (10.1016/j.jclinepi.2023.10.007_bib18) 1971; 76 Moher (10.1016/j.jclinepi.2023.10.007_bib4) 2008; 2008 Cohen (10.1016/j.jclinepi.2023.10.007_bib19) 1960; 20 Noel-Storr (10.1016/j.jclinepi.2023.10.007_bib22) 2014 Hazlewood (10.1016/j.jclinepi.2023.10.007_bib16) 2016; 353 El Mikati (10.1016/j.jclinepi.2023.10.007_bib3) 2022; 175 Wallace (10.1016/j.jclinepi.2023.10.007_bib10) 2017; 24 |
References_xml | – volume: 4 start-page: 1 year: 2015 ident: 10.1016/j.jclinepi.2023.10.007_bib23 article-title: Automating data extraction in systematic reviews: a systematic review publication-title: Syst Rev doi: 10.1186/s13643-015-0066-7 contributor: fullname: Jonnalagadda – volume: 2020 year: 2020 ident: 10.1016/j.jclinepi.2023.10.007_bib11 article-title: Disease-modifying anti-rheumatic drugs for rheumatoid arthritis: a systematic review and network meta-analysis publication-title: Cochrane Database Syst Rev contributor: fullname: Hazlewood – volume: 147 start-page: 224 year: 2007 ident: 10.1016/j.jclinepi.2023.10.007_bib5 article-title: How quickly do systematic reviews go out of date? A survival analysis publication-title: Ann Intern Med doi: 10.7326/0003-4819-147-4-200708210-00179 contributor: fullname: Shojania – volume: 66 start-page: S519 year: 2014 ident: 10.1016/j.jclinepi.2023.10.007_bib21 article-title: Effects of Tofacitinib on bone marrow edema, synovitis, and erosive damage in Methotrexate-naïve patients with early active rheumatoid arthritis (duration≤ 2 years): results of an exploratory phase 2 MRI study publication-title: Arthritis Rheumatol contributor: fullname: Conaghan – year: 2014 ident: 10.1016/j.jclinepi.2023.10.007_bib22 contributor: fullname: Noel-Storr – volume: 49 start-page: 1092 year: 2022 ident: 10.1016/j.jclinepi.2023.10.007_bib12 article-title: Canadian Rheumatology Association living guidelines for the pharmacological management of rheumatoid arthritis with disease-modifying anti-rheumatic drugs publication-title: J Rheumatol doi: 10.3899/jrheum.220209 contributor: fullname: Hazlewood – volume: 147 start-page: 142 year: 2022 ident: 10.1016/j.jclinepi.2023.10.007_bib14 article-title: Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: a mixed-methods study publication-title: J Clin Epidemiol doi: 10.1016/j.jclinepi.2022.03.019 contributor: fullname: Lee – volume: 133 start-page: 140 year: 2021 ident: 10.1016/j.jclinepi.2023.10.007_bib7 article-title: Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews publication-title: J Clin Epidemiol doi: 10.1016/j.jclinepi.2020.11.003 contributor: fullname: Thomas – volume: 24 start-page: 1165 year: 2017 ident: 10.1016/j.jclinepi.2023.10.007_bib10 article-title: Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach publication-title: J Am Med Inform Assoc doi: 10.1093/jamia/ocx053 contributor: fullname: Wallace – volume: 3 year: 2017 ident: 10.1016/j.jclinepi.2023.10.007_bib17 article-title: Biologics or tofacitinib for people with rheumatoid arthritis unsuccessfully treated with biologics: a systematic review and network meta-analysis publication-title: Cochrane Database Syst Rev contributor: fullname: Singh – volume: 76 start-page: 378 year: 1971 ident: 10.1016/j.jclinepi.2023.10.007_bib18 article-title: Measuring nominal scale agreement among many raters publication-title: Psychol Bull doi: 10.1037/h0031619 contributor: fullname: Fleiss – volume: 22 start-page: 1 year: 2022 ident: 10.1016/j.jclinepi.2023.10.007_bib2 article-title: Feasibility study for interactive reporting of network meta-analysis: experiences from the development of the MetaInsight COVID-19 app for stakeholder exploration, re-analysis and sensitivity analysis from living systematic reviews publication-title: BMC Med Res Methodol doi: 10.1186/s12874-022-01507-x contributor: fullname: Xin – start-page: 376 year: 2010 ident: 10.1016/j.jclinepi.2023.10.007_bib8 article-title: Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools contributor: fullname: Cohen – volume: 20 start-page: 37 year: 1960 ident: 10.1016/j.jclinepi.2023.10.007_bib19 article-title: A coefficient of agreement for nominal scales publication-title: Educ Psychol Meas doi: 10.1177/001316446002000104 contributor: fullname: Cohen – volume: 2008 year: 2008 ident: 10.1016/j.jclinepi.2023.10.007_bib4 article-title: When and how to update systematic reviews publication-title: Cochrane Database Syst Rev contributor: fullname: Moher – volume: 144 start-page: 22 year: 2022 ident: 10.1016/j.jclinepi.2023.10.007_bib6 article-title: Tools to support the automation of systematic reviews: a scoping review publication-title: J Clin Epidemiol doi: 10.1016/j.jclinepi.2021.12.005 contributor: fullname: Khalil – volume: 353 start-page: i1777 year: 2016 ident: 10.1016/j.jclinepi.2023.10.007_bib16 article-title: Methotrexate monotherapy and methotrexate combination therapy with traditional and biologic disease modifying antirheumatic drugs for rheumatoid arthritis: abridged Cochrane systematic review and network meta-analysis publication-title: BMJ doi: 10.1136/bmj.i1777 contributor: fullname: Hazlewood – volume: 14 start-page: 59 year: 2016 ident: 10.1016/j.jclinepi.2023.10.007_bib1 article-title: Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? publication-title: BMC Med doi: 10.1186/s12916-016-0596-4 contributor: fullname: Vandvik – volume: 346 start-page: f139 year: 2013 ident: 10.1016/j.jclinepi.2023.10.007_bib9 article-title: The automation of systematic reviews publication-title: BMJ doi: 10.1136/bmj.f139 contributor: fullname: Tsafnat – volume: 66 start-page: 15 year: 2014 ident: 10.1016/j.jclinepi.2023.10.007_bib20 article-title: Global molecular effects of tocilizumab therapy in rheumatoid arthritis synovium publication-title: Arthritis Rheumatol doi: 10.1002/art.38202 contributor: fullname: Ducreux – volume: 175 start-page: 1154 year: 2022 ident: 10.1016/j.jclinepi.2023.10.007_bib3 article-title: A framework for the development of living practice guidelines in health care publication-title: Ann Intern Med doi: 10.7326/M22-0514 contributor: fullname: El Mikati |
SSID | ssj0017075 |
Score | 2.4781106 |
SecondaryResourceType | review_article |
Snippet | To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a... ObjectivesTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA)... OBJECTIVESTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA)... |
SourceID | proquest crossref pubmed |
SourceType | Aggregation Database Index Database |
StartPage | 1 |
SubjectTerms | Accuracy Agreements Automation Classification Clinical medicine Clinical practice guidelines Clinical trials Crowdsourcing Efficiency Evaluation Learning algorithms Machine learning Reviews Rheumatoid arthritis Screening Steroids Workloads |
Title | Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review |
URI | https://www.ncbi.nlm.nih.gov/pubmed/37865299 https://www.proquest.com/docview/2900924860 https://search.proquest.com/docview/2880097207 |
Volume | 164 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT-MwEB7xkFZcVjx2oTwqI3FNIWkSxycEFQghgRACqbfITykVJIW2lz3uL9-Z2ClwWK5-JfLYM5_tb2YATjLpjM4KESkt8YBSKBdJkfDIoebLlUqdjsk5-e4-v3lOb8fZOFy4zQKtstOJraI2jaY78tNEUHggSpl0Pn2LKGsUva6GFBqrsB6ToSNP8dGS4hFzH2j3rBBZhLAg_-QhPBlMyPXQTqsBJRAftAwv_tU4_QdxtpbnehN-BsjILryMt2DF1tvw4y48iu_A3xEepU3UXsOjIWKyNkwu5o13SmROah-J2xqGaI9VJhCEfDW11gShP4oax9CCmea1-oN9Apn9hbq3q5VV2Iu9VHQVwbzryy94vr56Gt1EIbVCpGPiNYnc8hTBmVJc88w5kSPwcsIKmWqdmSKjsGWau7b-zCVaDa1JTcGHKk5crIa_Ya1uarsHLEYNhYKxRS6TlDuhEIDisacQLtUqF2kPTrs5Lac-gkbZUcsmZSeFkqRA5SiFHhx2U1-GHTUrP-Tfg-NlNe4FeuCQtW0W2AaVEYUjoiF2vciWnxzyIs_Q9u5_P_gBbNCfeMLKIazN3xf2CGHHXPVhlY95v11hfVi_vLp_ePwHrEna9w |
link.rule.ids | 314,780,784,12056,12223,21388,27924,27925,31719,31720,33266,33267,33744,33745,43310,43579,43805,73745,74014,74302 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3NT9swFLdGJwGXaeOzwIaRuKaQNInt04SqVd1GObUSt8ifUqqSFNpedtxfznux024HuPrZTuRnv_fz8_sg5DqTzuiMi0hpCRcUrlwkRcIiB5IvVyp1Osbg5PFDPpqmvx6zx2BwWwa3ylYmNoLa1Bpt5DeJwPRAWDLp--I5wqpR-LoaSmjskI-g9hnuaj7YuHjEzCfaveUii4Ce_xMhPOvNMPTQLsoeFhDvNR5e7H_l9AbibDTP8DP5FCAjvfM8_kI-2OqA7I7Do_gh-TuAq7SJGjM8KCIqK0PlelX7oETqpPaZuK2hgPZoaYKDkCdjb40QettUOwoazNRP5R8YE5zZ5zi82a20hFF0XqIpgvrQlyMyHf6YDEZRKK0Q6Rj9mkRuWQrgTCmmWeacyAF4OWGFTLXODM8wbZlmrqHfukSrvjWp4ayv4sTFqn9MOlVd2VNCY5BQwBjLc5mkzAkFABSuPVy4VKtcpF1y065psfAZNIrWtWxWtFwokAvYDlzokot26YtwopbFlv9dcrUhw1nABw5Z2XoNfUAYYToinOLEs2zzyT7jeQa69-z9yS_J3mgyvi_ufz78Pif7-FfeeeWCdFYva_sVIMhKfWv22Ss7HNty |
linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1JT-swELZYJMQFsdPHZiSuaUmaxPYJoULFLg4gcYu8SqkgKY_28o7vlzMTOwUOcPWWyGPPfLa_mSHkOJPO6IyLSGkJBxSuXCRFwiIHmi9XKnU6Rufku_v88im9fs6eA__pPdAqW53YKGpTa7wj7yUCwwNhyqSeC7SIh_Ph6fgtwgxS-NIa0mnMk0XMModx9PlgRveImQ-6e8JFFgFEyL94C4-6I3RDtOOyi8nEuw3bi303VD-gz8YKDVfJSoCP9MzLe43M2WqdLN2FB_IN8n8Ax2oTNVfyYJSorAyV00ntHRSpk9pH5baGAvKjpQlkIV-NrTXC6c-i2lGwZqZ-Lf9Bn0Bsf8HuzcqlJfSiLyVeS1DvBrNJnoYXj4PLKKRZiHSMHCeRW5YCUFOKaZY5J3IAYU5YIVOtM8MzDGGmmWvqT1yiVd-a1HDWV3HiYtXfIgtVXdkdQmPQViAky3OZpMwJBWAUjkBcuFSrXKQd0mvntBj7aBpFSzMbFa0UCpQCloMUOmSvnfoi7K734nMtdMjRrBr2BT52yMrWU2gDiglDE-EQ215ks0_2Gc8zsMN_fh_8kCzBEitur-5vdsky_pTnseyRhcnfqd0HNDJRB80y-wA0gd-X |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Crowd-sourcing+and+automation+facilitated+the+identification+and+classification+of+randomized+controlled+trials+in+a+living+review&rft.jtitle=Journal+of+clinical+epidemiology&rft.au=Kamso%2C+Mohammed+Mujaab&rft.au=Pardo%2C+Jordi+Pardo&rft.au=Whittle%2C+Samuel+L.&rft.au=Buchbinder%2C+Rachelle&rft.date=2023-12-01&rft.issn=0895-4356&rft.volume=164&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1016%2Fj.jclinepi.2023.10.007&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_jclinepi_2023_10_007 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0895-4356&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0895-4356&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0895-4356&client=summon |