Hypothesis-Testing Improves the Predicted Reliability of Neuroscience Research

Abstract Critics often cite statistical problems as prime contributors to the “reproducibility crisis” of science, expressing great concern about research that bases major conclusions on single p-valued statistical tests. The critics also argue that the predicted reliability of neuroscience research in particular is low because much of the work depends heavily on small experimental sample sizes and, hence, its statistical tests lack adequate “power.” It isn’t known how common the practice of basing major conclusions on single tests is in neuroscience or how the statistical criticisms affect the validity of conclusions drawn by laboratory research that evaluates hypotheses via multiple tests. I review a sample of neuroscience publications to estimate the prevalence and extensiveness of hypothesis-testing research. I then apply R.A. Fisher’s method for combining test results to show that the practice of testing multiple predictions of hypotheses increases the predicted reliability of neuroscience research.