Many know that p-values follow an uniform distribution when the null hypothesis is true. But what about when the null isn't true? This visualization shows the distribution of p-values when comparing the means of two independent samples. Check out this blog post by Daniel Lakens for more information.

Slide to change ES (d = )

Slide to change sample size (n = per group)

Power

p-values in selection

Log x-axis

Limitations

This visualization is based on a two-sample Z-test, i.e. we assume that the true standard deviation is known. In real life this is often not the case, which is why t-tests are much more common. When the effect is nonzero p-values from t-tests follow a non-central t distribution. However, the formulas used here works quite well as normal approximation of the non-central t distribution, but it is slightly biased when n is small. As an educational tool this hardly makes a difference, since the take-home message is the amount of skew when the true effect is greater than zero. This approach is also used by Cumming (2008).