The Multiple Hypothesis Testing Problem
If you test for significance enough, you’re going to find something significant
Published in
5 min readMar 7, 2022
You’re about to run an experiment you think will improve your website conversion rate. You state your null hypothesis, H0, and your alternative hypothesis, H1:
H0: There is no difference in conversion rate
H1: There is a difference in conversion rate
You design your experiment (calculate sample size and establish a significance level, for example, α = 0.05) and set it live in an…
