Many know that p-values follow an uniform distribution when the null hypothesis is true. But what about when the null isn't true? This visualization shows the distribution of p-values when comparing the means of two independent samples. Check out this blog post by Daniel Lakens for more information.

Slide to change ES (d = )

Slide to change sample size (n = per group)

Power

p-values in selection

Log x-axis

Limitations

This visualization is based on a two-sample Z-test, i.e. we assume that the true standard deviation is known. In real life this is often not the case, which is why t-tests are much more common. When the effect is nonzero p-values from t-tests follow a non-central t distribution. However, the formulas used here works quite well as normal approximation of the non-central t distribution, but it is slightly biased when n is small. As an educational tool this hardly makes a difference, since the take-home message is the amount of skew when the true effect is greater than zero. This approach is also used by Cumming (2008).

Support my work

Become a Patron!

If you enjoy my work, or maybe even use it in your own teaching, please consider supporting me on Patreon. My visualizations will always be free to use, but you can show your support by donating a dollar or two for every new visualization I create.

Suggestions

Have any suggestion? Send them to me, my contact info can be found here.