A new method for testing reproducibility in systematic reviews was developed, but needs more testing

• Published in: BMC medical research methodology Vol. 21; no. 1; pp. 1 - 157
• Main Authors: Pieper, Dawid, Hess, Simone, Faggion, Clovis Mariano
• Format: Journal Article
• Language: English
• Published: London BioMed Central Ltd 29.07.2021; BioMed Central; BMC
• Subjects: Bias; Bibliographic data bases; Data extraction; Data mining; Information storage and retrieval; Medical research; Medicine, Experimental; Methodological quality; Methods; Reproducibility; Reproducibility of Results; Research methodology; Risk of bias; Secondary data analysis; Systematic review; Systematic reviews; Germany
• ISSN: 1471-2288; 1471-2288
• DOI: 10.1186/s12874-021-01342-6

Abstract Background To develop and test an approach to test reproducibility of SRs. Methods Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review. Results We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory. Conclusion Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.