Applying generalized funnel plots to help design statistical analyses

Researchers across many fields routinely analyze trial data using Null Hypothesis Significance Tests with zero null and p < 0.05. To promote thoughtful statistical testing, we propose a visualization tool that highlights practically meaningful effects when calculating sample sizes. The tool re-pu...

Full description

Saved in:
Bibliographic Details
Published inStatistical papers (Berlin, Germany) Vol. 64; no. 1; pp. 355 - 364
Main Authors Aisbett, Janet, Drinkwater, Eric J., Quarrie, Kenneth L., Woodcock, Stephen
Format Journal Article
LanguageEnglish
Published Berlin/Heidelberg Springer Berlin Heidelberg 01.02.2023
Springer Nature B.V
Subjects
Cognition & reasoning
Economic Theory/Quantitative Economics/Mathematical Methods
Economics
Finance
Hypotheses
Insurance
Management
Mathematical analysis
Mathematics and Statistics
Operations Research/Decision Theory
Probability Theory and Stochastic Processes
Regular Article
Researchers
Software packages
Statistical analysis
Statistics
Statistics for Business
Visualization
Equivalence test
Sample size calculator
62-04
62-P10
Funnel plots
62-P15
62-08
62-P25
Significance level
Meaningful effect sizes
Superiority test
Online AccessGet full text

Cover

More Information
Summary:Researchers across many fields routinely analyze trial data using Null Hypothesis Significance Tests with zero null and p < 0.05. To promote thoughtful statistical testing, we propose a visualization tool that highlights practically meaningful effects when calculating sample sizes. The tool re-purposes and adapts funnel plots, originally developed for meta-analyses, after generalizing them to cater for meaningful effects. As with traditional sample size calculators, researchers must nominate anticipated effect sizes and variability alongside the desired power. The advantage of our tool is that it simultaneously presents sample sizes needed to adequately power tests for equivalence, for non-inferiority and for superiority, each considered at up to three alpha levels and in positive and negative directions. The tool thus encourages researchers at the design stage to think about the type and level of test in terms of their research goals, costs of errors, meaningful effect sizes and feasible sample sizes. An R-implementation of the tool is available on-line.
ISSN:0932-5026
1613-9798
DOI:10.1007/s00362-022-01322-y