Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review

To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclu...

Full description

Saved in:
Bibliographic Details
Published inJournal of clinical epidemiology Vol. 164; pp. 1 - 8
Main Authors Kamso, Mohammed Mujaab, Pardo, Jordi Pardo, Whittle, Samuel L, Buchbinder, Rachelle, Wells, George, Glennon, Vanessa, Tugwell, Peter, Deardon, Rob, Sajobi, Tolulope, Tomlinson, George, Elliott, Jesse, Kelly, Shannon E, Hazlewood, Glen S
Format Journal Article
LanguageEnglish
Published United States Elsevier Limited 01.12.2023
Subjects
Online AccessGet full text

Cover

Loading…
Abstract To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers. From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69). A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
AbstractList To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers. From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69). A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
OBJECTIVESTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR).METHODSRecords from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.RESULTSFrom 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).CONCLUSIONA screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
Author Wells, George
Glennon, Vanessa
Tugwell, Peter
Tomlinson, George
Hazlewood, Glen S
Kelly, Shannon E
Buchbinder, Rachelle
Deardon, Rob
Elliott, Jesse
Pardo, Jordi Pardo
Kamso, Mohammed Mujaab
Sajobi, Tolulope
Whittle, Samuel L
Author_xml – sequence: 1
  givenname: Mohammed Mujaab
  surname: Kamso
  fullname: Kamso, Mohammed Mujaab
  email: mohammedmujaab.kamso@ucalgary.ca
  organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada. Electronic address: mohammedmujaab.kamso@ucalgary.ca
– sequence: 2
  givenname: Jordi Pardo
  surname: Pardo
  fullname: Pardo, Jordi Pardo
  organization: Centre for Practice-Changing Research, Ottawa Hospital Research Institute, The Ottawa Hospital - General Campus, Ottawa, Canada
– sequence: 3
  givenname: Samuel L
  surname: Whittle
  fullname: Whittle, Samuel L
  organization: Department Rheumatology, The Queen Elizabeth Hospital, Adelaide, Australia; School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
– sequence: 4
  givenname: Rachelle
  surname: Buchbinder
  fullname: Buchbinder, Rachelle
  organization: School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
– sequence: 5
  givenname: George
  surname: Wells
  fullname: Wells, George
  organization: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
– sequence: 6
  givenname: Vanessa
  surname: Glennon
  fullname: Glennon, Vanessa
  organization: School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
– sequence: 7
  givenname: Peter
  surname: Tugwell
  fullname: Tugwell, Peter
  organization: Department of Medicine, University of Ottawa, Ottawa, Canada; Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; Department of Medicine, University of Ottawa, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Bruyere Research institute, Ottawa, Canada
– sequence: 8
  givenname: Rob
  surname: Deardon
  fullname: Deardon, Rob
  organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada; Department of Mathematics & Statistics, Faculty of Science, University of Calgary, Calgary, Canada
– sequence: 9
  givenname: Tolulope
  surname: Sajobi
  fullname: Sajobi, Tolulope
  organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
– sequence: 10
  givenname: George
  surname: Tomlinson
  fullname: Tomlinson, George
  organization: Department of Medicine, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada
– sequence: 11
  givenname: Jesse
  surname: Elliott
  fullname: Elliott, Jesse
  organization: Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada
– sequence: 12
  givenname: Shannon E
  surname: Kelly
  fullname: Kelly, Shannon E
  organization: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada
– sequence: 13
  givenname: Glen S
  surname: Hazlewood
  fullname: Hazlewood, Glen S
  organization: Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
BackLink https://www.ncbi.nlm.nih.gov/pubmed/37865299$$D View this record in MEDLINE/PubMed
BookMark eNpdkUFPHSEUhUljU5-2f8FM4qabeb0wwwBL89KqiYmbdk0YBlomDDxhRqPL_nIZn9qkK8I537mXcE7QUYjBIHSGYYsBd9_G7ai9C2bvtgRIU8QtAPuANpgzXlNB8BHaABe0bhvaHaOTnEcAzIDRT-i4YbyjRIgN-rtL8WGoc1ySduF3pcJQqWWOk5pdDJVV2nk3q9kM1fzHVG4wYXbW6YO90tqrnP9J0VapyHFyTyWjY5hT9H6NJ6d8rlxJVd7dr8uSuXfm4TP6aItjvryep-jXj-8_d1f1ze3l9e7iptaYc1GLzrC2o9D3TDNqregoxlYYoVqt6cBpW-6a2RcfLNF9Y4Z24KzpMbG4b07R18PcfYp3i8mznFzWxnsVTFyyJJwDCEaAFfT8P3QsHxTK6yQRBSIt76BQ3YHSKeacjJX75CaVHiUGubYkR_nWklxbWnV4GX_2On7pJzO8x95qaZ4B1naVNQ
Cites_doi 10.1186/s13643-015-0066-7
10.7326/0003-4819-147-4-200708210-00179
10.3899/jrheum.220209
10.1016/j.jclinepi.2022.03.019
10.1016/j.jclinepi.2020.11.003
10.1093/jamia/ocx053
10.1037/h0031619
10.1186/s12874-022-01507-x
10.1177/001316446002000104
10.1016/j.jclinepi.2021.12.005
10.1136/bmj.i1777
10.1186/s12916-016-0596-4
10.1136/bmj.f139
10.1002/art.38202
10.7326/M22-0514
ContentType Journal Article
Copyright Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.
2023. The Authors
Copyright_xml – notice: Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.
– notice: 2023. The Authors
DBID NPM
AAYXX
CITATION
3V.
7QL
7QP
7RV
7T2
7T7
7TK
7U7
7U9
7X7
7XB
88C
88E
8AO
8C1
8FD
8FI
8FJ
8FK
8G5
ABUWG
AFKRA
AZQEC
BENPR
C1K
CCPQU
DWQXO
FR3
FYUFA
GHDGH
GNUQQ
GUQSH
H94
K9.
KB0
M0S
M0T
M1P
M2O
M7N
MBDVC
NAPCQ
P64
PQEST
PQQKQ
PQUKI
Q9U
7X8
DOI 10.1016/j.jclinepi.2023.10.007
DatabaseName PubMed
CrossRef
ProQuest Central (Corporate)
Bacteriology Abstracts (Microbiology B)
Calcium & Calcified Tissue Abstracts
Nursing & Allied Health Database (ProQuest)
Health and Safety Science Abstracts (Full archive)
Industrial and Applied Microbiology Abstracts (Microbiology A)
Neurosciences Abstracts
Toxicology Abstracts
Virology and AIDS Abstracts
Health & Medical Collection
ProQuest Central (purchase pre-March 2016)
Healthcare Administration Database (Alumni)
Medical Database (Alumni Edition)
ProQuest Pharma Collection
Public Health Database
Technology Research Database
Hospital Premium Collection
Hospital Premium Collection (Alumni Edition)
ProQuest Central (Alumni) (purchase pre-March 2016)
Research Library (Alumni Edition)
ProQuest Central (Alumni)
ProQuest Central
ProQuest Central Essentials
AUTh Library subscriptions: ProQuest Central
Environmental Sciences and Pollution Management
ProQuest One Community College
ProQuest Central Korea
Engineering Research Database
Health Research Premium Collection
Health Research Premium Collection (Alumni)
ProQuest Central Student
Research Library Prep
AIDS and Cancer Research Abstracts
ProQuest Health & Medical Complete (Alumni)
Nursing & Allied Health Database (Alumni Edition)
Health & Medical Collection (Alumni Edition)
Health Management Database (Proquest)
Medical Database
Research Library
Algology Mycology and Protozoology Abstracts (Microbiology C)
Research Library (Corporate)
Nursing & Allied Health Premium
Biotechnology and BioEngineering Abstracts
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Academic
ProQuest One Academic UKI Edition
ProQuest Central Basic
MEDLINE - Academic
DatabaseTitle PubMed
CrossRef
Research Library Prep
ProQuest Central Student
Technology Research Database
ProQuest Central Essentials
ProQuest Health & Medical Complete (Alumni)
ProQuest Central (Alumni Edition)
ProQuest One Community College
Research Library (Alumni Edition)
ProQuest Pharma Collection
Environmental Sciences and Pollution Management
ProQuest Central
Health Research Premium Collection
Health and Medicine Complete (Alumni Edition)
ProQuest Central Korea
Bacteriology Abstracts (Microbiology B)
Algology Mycology and Protozoology Abstracts (Microbiology C)
AIDS and Cancer Research Abstracts
ProQuest Research Library
Health & Safety Science Abstracts
Industrial and Applied Microbiology Abstracts (Microbiology A)
ProQuest Medical Library (Alumni)
ProQuest Public Health
Virology and AIDS Abstracts
ProQuest Central Basic
Toxicology Abstracts
ProQuest One Academic Eastern Edition
ProQuest Health Management
ProQuest Nursing & Allied Health Source
ProQuest Hospital Collection
Health Research Premium Collection (Alumni)
Neurosciences Abstracts
ProQuest Hospital Collection (Alumni)
Biotechnology and BioEngineering Abstracts
Nursing & Allied Health Premium
ProQuest Health & Medical Complete
ProQuest Medical Library
ProQuest One Academic UKI Edition
ProQuest Health Management (Alumni Edition)
ProQuest Nursing & Allied Health Source (Alumni)
Engineering Research Database
ProQuest One Academic
Calcium & Calcified Tissue Abstracts
ProQuest Central (Alumni)
MEDLINE - Academic
DatabaseTitleList PubMed
MEDLINE - Academic
Research Library Prep
Database_xml – sequence: 1
  dbid: NPM
  name: PubMed
  url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
  sourceTypes: Index Database
– sequence: 2
  dbid: BENPR
  name: ProQuest Central
  url: https://www.proquest.com/central
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Medicine
EISSN 1878-5921
EndPage 8
ExternalDocumentID 10_1016_j_jclinepi_2023_10_007
37865299
Genre Journal Article
Review
GroupedDBID ---
--K
--M
-~X
.1-
.55
.FO
.GJ
.~1
0R~
1B1
1P~
1RT
1~.
1~5
29K
3V.
4.4
457
4CK
4G.
53G
5GY
5RE
5VS
7-5
71M
7RV
7X7
88E
8AO
8C1
8FI
8FJ
8G5
8P~
9JM
9JO
AABNK
AACTN
AAEDT
AAEDW
AAFJI
AAIKJ
AAKOC
AALRI
AAOAW
AAQFI
AAQXK
AAWTL
AAXKI
AAXUO
AAYJJ
ABBQC
ABFNM
ABIVO
ABJNI
ABLJU
ABMAC
ABMMH
ABMZM
ABOCM
ABUWG
ABXDB
ACDAQ
ACGFS
ACIUM
ACPRK
ACRLP
ADBBV
ADEZE
ADMUD
AEBSH
AEKER
AENEX
AEVXI
AFCTW
AFFNX
AFJKZ
AFKRA
AFKWA
AFRAH
AFRHN
AFTJW
AFXIZ
AGHFR
AGUBO
AGYEJ
AHHHB
AHMBA
AIEXJ
AIKHN
AITUG
AJOXV
AJRQY
AJUYK
AKRWK
ALIPV
ALMA_UNASSIGNED_HOLDINGS
AMFUW
AMRAJ
ANZVX
AOMHK
AQUVI
ASPBG
AVARZ
AVWKF
AXJTR
AZFZN
AZQEC
BENPR
BKEYQ
BKOJK
BLXMC
BNPGV
BPHCQ
BVXVI
CCPQU
CS3
D-I
DU5
DWQXO
EBS
EFJIC
EJD
EMOBN
EO8
EO9
EP2
EP3
EX3
F3I
F5P
FDB
FEDTE
FGOYB
FIRID
FNPLU
FYGXN
FYUFA
G-2
G-Q
GBLVA
GNUQQ
GUQSH
HEH
HMCUK
HMK
HMO
HVGLF
HZ~
IHE
J1W
KOM
L7B
M0T
M1P
M29
M2O
M3W
M41
MO0
N9A
NAPCQ
NPM
O-L
O9-
OAUVE
OD~
OHT
OO0
OZT
P-8
P-9
P2P
PC.
PQQKQ
PRBVW
PROAC
PSQYO
Q38
R2-
RIG
ROL
RPZ
SAE
SCC
SDF
SDG
SDP
SEL
SES
SEW
SPCBC
SSB
SSH
SSO
SSZ
SV3
T5K
UAP
UKHRP
WOW
WUQ
X7M
XPP
YHZ
Z5R
ZGI
~G-
AAYXX
ACRPL
ADNMO
CITATION
7QL
7QP
7T2
7T7
7TK
7U7
7U9
7XB
8FD
8FK
C1K
FR3
H94
K9.
M7N
MBDVC
P64
PQEST
PQUKI
Q9U
7X8
ID FETCH-LOGICAL-c1889-96e74650bb7c75ff96511f9e9a4cc5d854511c7fbb7c70f2cb3ed4d873b12f1b3
IEDL.DBID 8C1
ISSN 0895-4356
IngestDate Wed Dec 04 08:19:58 EST 2024
Thu Dec 05 17:25:13 EST 2024
Fri Dec 06 02:00:10 EST 2024
Sat Nov 02 12:25:19 EDT 2024
IsPeerReviewed true
IsScholarly true
Keywords Crowdsourcing
Living systematic reviews
Automation
Rheumatoid arthritis
Machine learning
Systematic reviews
Randomized controlled trials (RCTs)
Language English
License Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c1889-96e74650bb7c75ff96511f9e9a4cc5d854511c7fbb7c70f2cb3ed4d873b12f1b3
Notes ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Feature-3
content type line 23
ObjectType-Review-1
ObjectType-Undefined-4
ORCID 0000-0001-7417-7691
0000-0001-7709-3741
0000-0001-5062-0556
0000-0002-3985-5335
0000-0002-0597-0933
0000-0003-0436-1850
0000-0003-2792-7706
PMID 37865299
PQID 2900924860
PQPubID 105585
PageCount 8
ParticipantIDs proquest_miscellaneous_2880097207
proquest_journals_2900924860
crossref_primary_10_1016_j_jclinepi_2023_10_007
pubmed_primary_37865299
PublicationCentury 2000
PublicationDate 2023-Dec
2023-12-00
20231201
PublicationDateYYYYMMDD 2023-12-01
PublicationDate_xml – month: 12
  year: 2023
  text: 2023-Dec
PublicationDecade 2020
PublicationPlace United States
PublicationPlace_xml – name: United States
– name: Elmsford
PublicationTitle Journal of clinical epidemiology
PublicationTitleAlternate J Clin Epidemiol
PublicationYear 2023
Publisher Elsevier Limited
Publisher_xml – sequence: 0
  name: Elsevier Limited
References Lee (10.1016/j.jclinepi.2023.10.007_bib14) 2022; 147
Xin (10.1016/j.jclinepi.2023.10.007_bib2) 2022; 22
Ducreux (10.1016/j.jclinepi.2023.10.007_bib20) 2014; 66
Jonnalagadda (10.1016/j.jclinepi.2023.10.007_bib23) 2015; 4
Hazlewood (10.1016/j.jclinepi.2023.10.007_bib12) 2022; 49
Singh (10.1016/j.jclinepi.2023.10.007_bib17) 2017; 3
Conaghan (10.1016/j.jclinepi.2023.10.007_bib21) 2014; 66
Tsafnat (10.1016/j.jclinepi.2023.10.007_bib9) 2013; 346
Thomas (10.1016/j.jclinepi.2023.10.007_bib7) 2021; 133
Cohen (10.1016/j.jclinepi.2023.10.007_bib8) 2010
Vandvik (10.1016/j.jclinepi.2023.10.007_bib1) 2016; 14
Shojania (10.1016/j.jclinepi.2023.10.007_bib5) 2007; 147
Hazlewood (10.1016/j.jclinepi.2023.10.007_bib11) 2020; 2020
Khalil (10.1016/j.jclinepi.2023.10.007_bib6) 2022; 144
Fleiss (10.1016/j.jclinepi.2023.10.007_bib18) 1971; 76
Moher (10.1016/j.jclinepi.2023.10.007_bib4) 2008; 2008
Cohen (10.1016/j.jclinepi.2023.10.007_bib19) 1960; 20
Noel-Storr (10.1016/j.jclinepi.2023.10.007_bib22) 2014
Hazlewood (10.1016/j.jclinepi.2023.10.007_bib16) 2016; 353
El Mikati (10.1016/j.jclinepi.2023.10.007_bib3) 2022; 175
Wallace (10.1016/j.jclinepi.2023.10.007_bib10) 2017; 24
References_xml – volume: 4
  start-page: 1
  year: 2015
  ident: 10.1016/j.jclinepi.2023.10.007_bib23
  article-title: Automating data extraction in systematic reviews: a systematic review
  publication-title: Syst Rev
  doi: 10.1186/s13643-015-0066-7
  contributor:
    fullname: Jonnalagadda
– volume: 2020
  year: 2020
  ident: 10.1016/j.jclinepi.2023.10.007_bib11
  article-title: Disease-modifying anti-rheumatic drugs for rheumatoid arthritis: a systematic review and network meta-analysis
  publication-title: Cochrane Database Syst Rev
  contributor:
    fullname: Hazlewood
– volume: 147
  start-page: 224
  year: 2007
  ident: 10.1016/j.jclinepi.2023.10.007_bib5
  article-title: How quickly do systematic reviews go out of date? A survival analysis
  publication-title: Ann Intern Med
  doi: 10.7326/0003-4819-147-4-200708210-00179
  contributor:
    fullname: Shojania
– volume: 66
  start-page: S519
  year: 2014
  ident: 10.1016/j.jclinepi.2023.10.007_bib21
  article-title: Effects of Tofacitinib on bone marrow edema, synovitis, and erosive damage in Methotrexate-naïve patients with early active rheumatoid arthritis (duration≤ 2 years): results of an exploratory phase 2 MRI study
  publication-title: Arthritis Rheumatol
  contributor:
    fullname: Conaghan
– year: 2014
  ident: 10.1016/j.jclinepi.2023.10.007_bib22
  contributor:
    fullname: Noel-Storr
– volume: 49
  start-page: 1092
  year: 2022
  ident: 10.1016/j.jclinepi.2023.10.007_bib12
  article-title: Canadian Rheumatology Association living guidelines for the pharmacological management of rheumatoid arthritis with disease-modifying anti-rheumatic drugs
  publication-title: J Rheumatol
  doi: 10.3899/jrheum.220209
  contributor:
    fullname: Hazlewood
– volume: 147
  start-page: 142
  year: 2022
  ident: 10.1016/j.jclinepi.2023.10.007_bib14
  article-title: Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: a mixed-methods study
  publication-title: J Clin Epidemiol
  doi: 10.1016/j.jclinepi.2022.03.019
  contributor:
    fullname: Lee
– volume: 133
  start-page: 140
  year: 2021
  ident: 10.1016/j.jclinepi.2023.10.007_bib7
  article-title: Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews
  publication-title: J Clin Epidemiol
  doi: 10.1016/j.jclinepi.2020.11.003
  contributor:
    fullname: Thomas
– volume: 24
  start-page: 1165
  year: 2017
  ident: 10.1016/j.jclinepi.2023.10.007_bib10
  article-title: Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach
  publication-title: J Am Med Inform Assoc
  doi: 10.1093/jamia/ocx053
  contributor:
    fullname: Wallace
– volume: 3
  year: 2017
  ident: 10.1016/j.jclinepi.2023.10.007_bib17
  article-title: Biologics or tofacitinib for people with rheumatoid arthritis unsuccessfully treated with biologics: a systematic review and network meta-analysis
  publication-title: Cochrane Database Syst Rev
  contributor:
    fullname: Singh
– volume: 76
  start-page: 378
  year: 1971
  ident: 10.1016/j.jclinepi.2023.10.007_bib18
  article-title: Measuring nominal scale agreement among many raters
  publication-title: Psychol Bull
  doi: 10.1037/h0031619
  contributor:
    fullname: Fleiss
– volume: 22
  start-page: 1
  year: 2022
  ident: 10.1016/j.jclinepi.2023.10.007_bib2
  article-title: Feasibility study for interactive reporting of network meta-analysis: experiences from the development of the MetaInsight COVID-19 app for stakeholder exploration, re-analysis and sensitivity analysis from living systematic reviews
  publication-title: BMC Med Res Methodol
  doi: 10.1186/s12874-022-01507-x
  contributor:
    fullname: Xin
– start-page: 376
  year: 2010
  ident: 10.1016/j.jclinepi.2023.10.007_bib8
  article-title: Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools
  contributor:
    fullname: Cohen
– volume: 20
  start-page: 37
  year: 1960
  ident: 10.1016/j.jclinepi.2023.10.007_bib19
  article-title: A coefficient of agreement for nominal scales
  publication-title: Educ Psychol Meas
  doi: 10.1177/001316446002000104
  contributor:
    fullname: Cohen
– volume: 2008
  year: 2008
  ident: 10.1016/j.jclinepi.2023.10.007_bib4
  article-title: When and how to update systematic reviews
  publication-title: Cochrane Database Syst Rev
  contributor:
    fullname: Moher
– volume: 144
  start-page: 22
  year: 2022
  ident: 10.1016/j.jclinepi.2023.10.007_bib6
  article-title: Tools to support the automation of systematic reviews: a scoping review
  publication-title: J Clin Epidemiol
  doi: 10.1016/j.jclinepi.2021.12.005
  contributor:
    fullname: Khalil
– volume: 353
  start-page: i1777
  year: 2016
  ident: 10.1016/j.jclinepi.2023.10.007_bib16
  article-title: Methotrexate monotherapy and methotrexate combination therapy with traditional and biologic disease modifying antirheumatic drugs for rheumatoid arthritis: abridged Cochrane systematic review and network meta-analysis
  publication-title: BMJ
  doi: 10.1136/bmj.i1777
  contributor:
    fullname: Hazlewood
– volume: 14
  start-page: 59
  year: 2016
  ident: 10.1016/j.jclinepi.2023.10.007_bib1
  article-title: Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews?
  publication-title: BMC Med
  doi: 10.1186/s12916-016-0596-4
  contributor:
    fullname: Vandvik
– volume: 346
  start-page: f139
  year: 2013
  ident: 10.1016/j.jclinepi.2023.10.007_bib9
  article-title: The automation of systematic reviews
  publication-title: BMJ
  doi: 10.1136/bmj.f139
  contributor:
    fullname: Tsafnat
– volume: 66
  start-page: 15
  year: 2014
  ident: 10.1016/j.jclinepi.2023.10.007_bib20
  article-title: Global molecular effects of tocilizumab therapy in rheumatoid arthritis synovium
  publication-title: Arthritis Rheumatol
  doi: 10.1002/art.38202
  contributor:
    fullname: Ducreux
– volume: 175
  start-page: 1154
  year: 2022
  ident: 10.1016/j.jclinepi.2023.10.007_bib3
  article-title: A framework for the development of living practice guidelines in health care
  publication-title: Ann Intern Med
  doi: 10.7326/M22-0514
  contributor:
    fullname: El Mikati
SSID ssj0017075
Score 2.4781106
SecondaryResourceType review_article
Snippet To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a...
ObjectivesTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA)...
OBJECTIVESTo evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA)...
SourceID proquest
crossref
pubmed
SourceType Aggregation Database
Index Database
StartPage 1
SubjectTerms Accuracy
Agreements
Automation
Classification
Clinical medicine
Clinical practice guidelines
Clinical trials
Crowdsourcing
Efficiency
Evaluation
Learning algorithms
Machine learning
Reviews
Rheumatoid arthritis
Screening
Steroids
Workloads
Title Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review
URI https://www.ncbi.nlm.nih.gov/pubmed/37865299
https://www.proquest.com/docview/2900924860
https://search.proquest.com/docview/2880097207
Volume 164
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT-MwEB7xkFZcVjx2oTwqI3FNIWkSxycEFQghgRACqbfITykVJIW2lz3uL9-Z2ClwWK5-JfLYM5_tb2YATjLpjM4KESkt8YBSKBdJkfDIoebLlUqdjsk5-e4-v3lOb8fZOFy4zQKtstOJraI2jaY78tNEUHggSpl0Pn2LKGsUva6GFBqrsB6ToSNP8dGS4hFzH2j3rBBZhLAg_-QhPBlMyPXQTqsBJRAftAwv_tU4_QdxtpbnehN-BsjILryMt2DF1tvw4y48iu_A3xEepU3UXsOjIWKyNkwu5o13SmROah-J2xqGaI9VJhCEfDW11gShP4oax9CCmea1-oN9Apn9hbq3q5VV2Iu9VHQVwbzryy94vr56Gt1EIbVCpGPiNYnc8hTBmVJc88w5kSPwcsIKmWqdmSKjsGWau7b-zCVaDa1JTcGHKk5crIa_Ya1uarsHLEYNhYKxRS6TlDuhEIDisacQLtUqF2kPTrs5Lac-gkbZUcsmZSeFkqRA5SiFHhx2U1-GHTUrP-Tfg-NlNe4FeuCQtW0W2AaVEYUjoiF2vciWnxzyIs_Q9u5_P_gBbNCfeMLKIazN3xf2CGHHXPVhlY95v11hfVi_vLp_ePwHrEna9w
link.rule.ids 314,780,784,12056,12223,21388,27924,27925,31719,31720,33266,33267,33744,33745,43310,43579,43805,73745,74014,74302
linkProvider ProQuest
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3NT9swFLdGJwGXaeOzwIaRuKaQNInt04SqVd1GObUSt8ifUqqSFNpedtxfznux024HuPrZTuRnv_fz8_sg5DqTzuiMi0hpCRcUrlwkRcIiB5IvVyp1Osbg5PFDPpqmvx6zx2BwWwa3ylYmNoLa1Bpt5DeJwPRAWDLp--I5wqpR-LoaSmjskI-g9hnuaj7YuHjEzCfaveUii4Ce_xMhPOvNMPTQLsoeFhDvNR5e7H_l9AbibDTP8DP5FCAjvfM8_kI-2OqA7I7Do_gh-TuAq7SJGjM8KCIqK0PlelX7oETqpPaZuK2hgPZoaYKDkCdjb40QettUOwoazNRP5R8YE5zZ5zi82a20hFF0XqIpgvrQlyMyHf6YDEZRKK0Q6Rj9mkRuWQrgTCmmWeacyAF4OWGFTLXODM8wbZlmrqHfukSrvjWp4ayv4sTFqn9MOlVd2VNCY5BQwBjLc5mkzAkFABSuPVy4VKtcpF1y065psfAZNIrWtWxWtFwokAvYDlzokot26YtwopbFlv9dcrUhw1nABw5Z2XoNfUAYYToinOLEs2zzyT7jeQa69-z9yS_J3mgyvi_ufz78Pif7-FfeeeWCdFYva_sVIMhKfWv22Ss7HNty
linkToPdf http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1JT-swELZYJMQFsdPHZiSuaUmaxPYJoULFLg4gcYu8SqkgKY_28o7vlzMTOwUOcPWWyGPPfLa_mSHkOJPO6IyLSGkJBxSuXCRFwiIHmi9XKnU6Rufku_v88im9fs6eA__pPdAqW53YKGpTa7wj7yUCwwNhyqSeC7SIh_Ph6fgtwgxS-NIa0mnMk0XMModx9PlgRveImQ-6e8JFFgFEyL94C4-6I3RDtOOyi8nEuw3bi303VD-gz8YKDVfJSoCP9MzLe43M2WqdLN2FB_IN8n8Ax2oTNVfyYJSorAyV00ntHRSpk9pH5baGAvKjpQlkIV-NrTXC6c-i2lGwZqZ-Lf9Bn0Bsf8HuzcqlJfSiLyVeS1DvBrNJnoYXj4PLKKRZiHSMHCeRW5YCUFOKaZY5J3IAYU5YIVOtM8MzDGGmmWvqT1yiVd-a1HDWV3HiYtXfIgtVXdkdQmPQViAky3OZpMwJBWAUjkBcuFSrXKQd0mvntBj7aBpFSzMbFa0UCpQCloMUOmSvnfoi7K734nMtdMjRrBr2BT52yMrWU2gDiglDE-EQ215ks0_2Gc8zsMN_fh_8kCzBEitur-5vdsky_pTnseyRhcnfqd0HNDJRB80y-wA0gd-X
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Crowd-sourcing+and+automation+facilitated+the+identification+and+classification+of+randomized+controlled+trials+in+a+living+review&rft.jtitle=Journal+of+clinical+epidemiology&rft.au=Kamso%2C+Mohammed+Mujaab&rft.au=Pardo%2C+Jordi+Pardo&rft.au=Whittle%2C+Samuel+L.&rft.au=Buchbinder%2C+Rachelle&rft.date=2023-12-01&rft.issn=0895-4356&rft.volume=164&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1016%2Fj.jclinepi.2023.10.007&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_jclinepi_2023_10_007
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0895-4356&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0895-4356&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0895-4356&client=summon