Reporting bias inflates the reputation of medical treatments: A comparison of outcomes in clinical trials and online product reviews

• Published in: Social science & medicine (1982) Vol. 177; pp. 248 - 255
• Main Author: de Barra, Mícheál
• Format: Journal Article
• Language: English
• Published: England Elsevier Ltd 01.03.2017; Pergamon Press Inc
• Subjects: Averages; Bias; Cholesterol; Clinical outcomes; Clinical research; Clinical trials; Cultural evolution; Data Accuracy; Drugs; eHealth; Health care; Health informatics; Health problems; Health services; Humans; Hypercholesterolemia - drug therapy; Internet; Lactones - pharmacology; Lactones - therapeutic use; Low income groups; Medical literature; Medical overuse; Medical research; Medical treatment; Medicine; Mouth; Orlistat; Outcome Assessment (Health Care) - standards; Outcome Assessment (Health Care) - statistics & numerical data; Product reviews; Reputations; Research Design - standards; Sitosterols - pharmacology; Sitosterols - therapeutic use; Therapeutics - methods; Therapeutics - standards; Treatment methods; Treatment outcomes; Variance analysis; Weight loss; Word of mouth; Cultural evolution; Word of mouth; Medical overuse; Health informatics; eHealth
• ISSN: 0277-9536; 1873-5347; 1873-5347
• DOI: 10.1016/j.socscimed.2017.01.033

People often hold unduly positive expectations about the outcomes of medicines and other healthcare products. Here the following explanation is tested: people who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically and positively distorts the information available to others. If people with good treatment outcomes are more inclined to tell others, then they should also be more inclined to write online medical product reviews. Therefore, average treatment outcomes in these reviews should be more positive than those found in randomised controlled trials (RCTs). Data on duration of treatment and outcome (i.e., weight/cholesterol change) were extracted from user-generated health product reviews on Amazon.com and compared to RCT data for the same treatments using ANOVA. The sample included 1675 reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments and the primary outcome was cholesterol change (Bencol and CholestOff) or weight change (Orlistat). In three independent tests, average outcomes reported in the reviews were substantially more positive than the outcomes reported in the medical literature (η2 = 0.01 to 0.06; p = 0.04 to 0.001). For example, average cholesterol change following use of Benecol is −14 mg/dl in RCTs and −45 mg/dl in online reviews. People with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others. People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations. •Why do people overestimate how much a medical treatment will benefit them?•We compare average outcomes in clinical trials and online product reviews.•Average outcomes in online reviews are much more positive.•People who have good outcomes are more likely to write online reviews.•Beliefs based on (electronic) word-of-mouth will be positively distorted.