Understanding p-values Through Simulations

An Interactive Visualization

Created by Kristoffer Magnusson

Share

P-values are often misinterpreted or misused. My goal with this page is to explain p-values through an interactive simulation. (This is an early release that is still under development!).

Cohen's d

Drag

Observations per sample

n=5

Draw samples

# Draws: 0


Mean

Stats

Type I Error: 0.00

Power (true): 0.00

Effect size (true): 0.0

Effect size (sim): 0.0

Effect size (pub. bias): 0.0

Loading visualization

Written by Kristoffer Magnusson, a researcher in clinical psychology. You should follow him on Twitter and come hang out on the open science discord Git Gud Science.

Page under construction
This is an alpha version of this page, there might be some rough edges and missing features!

P HACK: add 1 observations to all samples that do not show a positive effect and reanalyze.

FAQ

Please report errors or suggestions by opening an issue on GitHub, if you want to ask a question use GitHub discussions

Contribute/Donate

There are many ways to contribute to free and open software. If you like my work and want to support it you can:

A huge thanks to the 79 supporters who've bought me a 179 coffees!

Jason Rinaldo bought ☕☕☕☕☕☕☕☕☕☕ (10) coffees

I've been looking for applets that show this for YEARS, for demonstrations for classes. Thank you so much! Students do not need to tolarate my whiteboard scrawl now. I'm sure they'd appreciate you, too.l

Someone bought ☕☕☕☕☕ (5) coffees

What a great site

@Daniel_Brad4d bought ☕☕☕☕☕ (5) coffees

Wonderful work!

David Loschelder bought ☕☕☕☕☕ (5) coffees

Terrific work. So very helpful. Thank you very much.

@neilmeigh bought ☕☕☕☕☕ (5) coffees

I am so grateful for your page and can't thank you enough!  

@giladfeldman bought ☕☕☕☕☕ (5) coffees

Wonderful work, I use it every semester and it really helps the students (and me) understand things better. Keep going strong.

Dean Norris bought ☕☕☕☕☕ (5) coffees

Sal bought ☕☕☕☕☕ (5) coffees

Really super useful, especially for teaching. Thanks for this!

dde@paxis.org bought ☕☕☕☕☕ (5) coffees

Very helpful to helping teach teachers about the effects of the Good Behavior Game

@akreutzer82 bought ☕☕☕☕☕ (5) coffees

Amazing visualizations! Thank you!

@rdh_CLE bought ☕☕☕☕☕ (5) coffees

So good!

Someone bought ☕☕☕ (3) coffees

Ruslan Klymentiev bought ☕☕☕ (3) coffees

@lkizbok bought ☕☕☕ (3) coffees

Keep up the nice work, thank you!

@TELLlab bought ☕☕☕ (3) coffees

Thanks - this will help me to teach tomorrow!

SCCT/Psychology bought ☕☕☕ (3) coffees

Keep the visualizations coming!

@elena_bolt bought ☕☕☕ (3) coffees

Thank you so much for your work, Kristoffer. I use your visualizations to explain concepts to my tutoring students and they are a huge help.

A random user bought ☕☕☕ (3) coffees

Thank you for making such useful and pretty tools. It not only helped me understand more about power, effect size, etc, but also made my quanti-method class more engaging and interesting. Thank you and wish you a great 2021!

@hertzpodcast bought ☕☕☕ (3) coffees

We've mentioned your work a few times on our podcast and we recently sent a poster to a listener as prize so we wanted to buy you a few coffees. Thanks for the great work that you do!Dan Quintana and James Heathers - Co-hosts of Everything Hertz 

Cameron Proctor bought ☕☕☕ (3) coffees

Used your vizualization in class today. Thanks!

eshulman@brocku.ca bought ☕☕☕ (3) coffees

My students love these visualizations and so do I! Thanks for helping me make stats more intuitive.

Someone bought ☕☕☕ (3) coffees

Adrian Helgå Vestøl bought ☕☕☕ (3) coffees

@misteryosupjoo bought ☕☕☕ (3) coffees

For a high school teacher of psychology, I would be lost without your visualizations. The ability to interact and manipulate allows students to get it in a very sticky manner. Thank you!!!

Chi bought ☕☕☕ (3) coffees

You Cohen's d post really helped me explaining the interpretation to people who don't know stats! Thank you!

Someone bought ☕☕☕ (3) coffees

You doing useful work !! thanks !!

@ArtisanalANN bought ☕☕☕ (3) coffees

Enjoy.

@jsholtes bought ☕☕☕ (3) coffees

Teaching stats to civil engineer undergrads (first time teaching for me, first time for most of them too) and grasping for some good explanations of hypothesis testing, power, and CI's. Love these interactive graphics!

@notawful bought ☕☕☕ (3) coffees

Thank you for using your stats and programming gifts in such a useful, generous manner. -Jess

Mateu Servera bought ☕☕☕ (3) coffees

A job that must have cost far more coffees than we can afford you ;-). Thank you.

@cdrawn bought ☕☕☕ (3) coffees

Thank you! Such a great resource for teaching these concepts, especially CI, Power, correlation.

Julia bought ☕☕☕ (3) coffees

Fantastic work with the visualizations!

@felixthoemmes bought ☕☕☕ (3) coffees

@dalejbarr bought ☕☕☕ (3) coffees

Your work is amazing! I use your visualizations often in my teaching. Thank you. 

@PsychoMouse bought ☕☕☕ (3) coffees

Excellent!  Well done!  SOOOO Useful!😊 🐭 

@whlevine bought ☕☕ (2) coffees

Thank you so much for these amazing visualizations. They're a great teaching tool and the allow me to show students things that it would take me weeks or months to program myself.

Someone bought ☕☕ (2) coffees

@notawful bought ☕☕ (2) coffees

Thank you for sharing your visualization skills with the rest of us! I use them frequently when teaching intro stats. 

@inthelabagain bought ☕ (1) coffee

Really wonderful visuals, and such a fantastic and effective teaching tool. So many thanks!

Someone bought ☕ (1) coffee

Someone bought ☕ (1) coffee

Yashashree Panda bought ☕ (1) coffee

I really like your work.

Ben bought ☕ (1) coffee

You're awesome. I have students in my intro stats class say, "I get it now," after using your tool. Thanks for making my job easier.

Gabriel Recchia bought ☕ (1) coffee

Incredibly useful tool!

Shiseida Sade Kelly Aponte bought ☕ (1) coffee

Thanks for the assistance for RSCH 8210.

@Benedikt_Hell bought ☕ (1) coffee

Great tools! Thank you very much!

Amalia Alvarez bought ☕ (1) coffee

@noelnguyen16 bought ☕ (1) coffee

Hi Kristoffer, many thanks for making all this great stuff available to the community!

Eran Barzilai bought ☕ (1) coffee

These visualizations are awesome! thank you for creating it

Someone bought ☕ (1) coffee

Chris SG bought ☕ (1) coffee

Very nice.

Gray Church bought ☕ (1) coffee

Thank you for the visualizations. They are fun and informative.

Qamar bought ☕ (1) coffee

Tanya McGhee bought ☕ (1) coffee

@schultemi bought ☕ (1) coffee

Neilo bought ☕ (1) coffee

Really helpful visualisations, thanks!

Someone bought ☕ (1) coffee

This is amazing stuff. Very slick. 

Someone bought ☕ (1) coffee

Sarko bought ☕ (1) coffee

Thanks so much for creating this! Really helpful for being able to explain effect size to a clinician I'm doing an analysis for. 

@DominikaSlus bought ☕ (1) coffee

Thank you! This page is super useful. I'll spread the word. 

Someone bought ☕ (1) coffee

Melinda Rice bought ☕ (1) coffee

Thank you so much for creating these tools! As we face the challenge of teaching statistical concepts online, this is an invaluable resource.

@tmoldwin bought ☕ (1) coffee

Fantastic resource. I think you would be well served to have one page indexing all your visualizations, that would make it more accessible for sharing as a common resource.

Someone bought ☕ (1) coffee

Fantastic Visualizations! Amazing way to to demonstrate how n/power/beta/alpha/effect size are all interrelated - especially for visual learners! Thank you for creating this?

@jackferd bought ☕ (1) coffee

Incredible visualizations and the best power analysis software on R.

Cameron Proctor bought ☕ (1) coffee

Great website!

Someone bought ☕ (1) coffee

Hanah Chapman bought ☕ (1) coffee

Thank you for this work!!

Someone bought ☕ (1) coffee

Jayme bought ☕ (1) coffee

Nice explanation and visual guide of Cohen's d

Bart Comly Boyce bought ☕ (1) coffee

thank you

Dr. Mitchell Earleywine bought ☕ (1) coffee

This site is superb!

Florent bought ☕ (1) coffee

Zampeta bought ☕ (1) coffee

thank you for sharing your work. 

Mila bought ☕ (1) coffee

Thank you for the website, made me smile AND smarter :O enjoy your coffee! :)

Deb bought ☕ (1) coffee

Struggling with statistics and your interactive diagram made me smile to see that someone cares enough about us strugglers to make a visual to help us out!😍 

Someone bought ☕ (1) coffee

@exerpsysing bought ☕ (1) coffee

Much thanks! Visualizations are key to my learning style! 

Someone bought ☕ (1) coffee

Sponsors

You can sponsor my open source work using GitHub Sponsors and have your name shown here.

Backers ✨❤️

Pull requests are also welcome, or you can contribute by suggesting new features, add useful references, or help fix typos. Just open a issues on GitHub.

Webmentions

1591 319

What's this?

Andrew Perfors
Andrew Perfors 2021-03-26
This is great!
Thom Baguley
Thom Baguley 2021-03-26
There's always stepwise regression ...
Schotz
Schotz 2021-03-26
Ive always wished my statistical software came with a “p-hack” button.
Kristoffer Magnusson
Yeah, they work great for calculations that can be truly run in parallel to the UI thread
Richard D. Morey
Richard D. Morey 2021-03-21
just figured out web workers for an app; the setup is annoying, but they're effective!
Tanner Delpier
Tanner Delpier 2021-03-20
This is so satisfying... rpsychologist.com/pvalue/
Nona
Nona 2021-03-20
Brilliant!
Kaleb Mathieu
Kaleb Mathieu 2021-03-13
rpsychologist.com/pvalue/ Fantastic visualization by @krstoffr to help understand what a p-value is. #AcademicTwitter #stats
Santiago Silvestrini
Muy buen recurso interactivo!
Daniel Wiczew
Daniel Wiczew 2021-03-09
Want to play with m̶e̶ some statistics ? rpsychologist.com/pvalue/ #visualization #Statistics
Per Damkier
Per Damkier 2021-03-08
Folks, this is rather awesome👇👏
Dominique Roche
Dominique Roche 2021-03-08
Amazing!
megan peters
megan peters 2021-03-08
the world can never have enough of these beautiful, simple, easily-understood visualizations of basic stats concepts ❤ stats is hard for many. i also used to find stats hard. so here's another tools to add to our repertoire, to help build our students' intuitive understanding.
megan peters
megan peters 2021-03-08
the world can never have enough of these beautiful, simple, easily-understood visualizations of basic stats concepts ❤ stats is hard for many. i also used to find stats hard. so here's another tools to add to our repertoire, to help build our students' intuitive understanding.
Dominique Roche
Dominique Roche 2021-03-08
Amazing!
Hacker News記事題日本語翻訳
シミュレーションによるP値の理解–インタラクティブな視覚化 rpsychologist.com/pvalue/
Hacker News 50
Hacker News 50 2021-03-08
Understanding P-Values Through Simulations – An Interactive Visualization rpsychologist.com/pvalue/ (news.ycombinator.com/item?id=263762…)
Doğu
Doğu 2021-03-08
This can truly come in handy rpsychologist.com/pvalue/
Zieloli
Zieloli 2021-03-07
P-values are often misinterpreted or misused. The goal of this page is to explain p-values through an interactive simulation. rpsychologist.com/pvalue/
AV Speech Processing
Do you ever have to explain p-values to students? (I know I do). Take a look at "Understanding p-values through simulations - an interactive visualization" by @krstoffr rpsychologist.com/pvalue/ 👏👏
Hacker News 20
Hacker News 20 2021-03-07
Understanding P-Values Through Simulations – An Interactive Visualization rpsychologist.com/pvalue/ (news.ycombinator.com/item?id=263762…)
HN Front Page
HN Front Page 2021-03-07
Understanding P-Values Through Simulations – An Interactive Visualization L: rpsychologist.com/pvalue/ C: news.ycombinator.com/item?id=263762…
Winson Tang
Winson Tang 2021-03-07
Understanding P-Values Through Simulations – An Interactive Visualization rpsychologist.com/pvalue/?utm_so…
Hacker News
Hacker News 2021-03-07
Understanding P-Values Through Simulations – An Interactive Visualization: rpsychologist.com/pvalue/ Comments: news.ycombinator.com/item?id=263762…
Ruben Dario Palacio
I'd say abandon p-values but this is how statistics should be taught!
Polina Beloborodova
The most straightforward demonstration of p-value issues that I’ve ever seen.
Mathieu M.J.E. Rebeaud
Oooh c'est un beau joujou pour les stateux.
Jorge Joo
Jorge Joo 2021-03-07
Understanding p-values through simulations - an interactive visualization by @krstoffr rpsychologist.com/pvalue/
Sarah H
Sarah H 2021-03-07
https://t.co/OXlkawMVOd
manulu
manulu 2021-03-07
Do you plan on implementing a simulation that shows the central limit theorem? :)
321
321 2021-03-06
Understanding p-values | R Psychologist rpsychologist.com/pvalue/
Sarah Myers
Sarah Myers 2021-03-05
THIS IS SO HELPFUL
Begüm Özkısaoğlu
This is cool!
Kristin
Kristin 2021-03-05
rpsychologist.com/pvalue/
Antoine 10km
Antoine 10km 2021-03-05
Voilà qui a autrement plus de tronche que mes petits graphiques tiens.
Duhyadi Oliva García
Puntitos
Dr. Jennifer Provencher, wears a mask
Ping @heathmacmillan
Oriane Armand
Oriane Armand 2021-03-05
Covering tests and plots in R will already be quite dense but I’ll see if I can touch upon it 😉
Kristoffer Magnusson
Thanks! I've opened an issue for this
Peter Dahlgren
Peter Dahlgren 2021-03-05
Jättebra som vanligt!
Nicolas
Nicolas 2021-03-05
😍
Nicolas
Nicolas 2021-03-05
@susyandrausdl
BOT RESERVA 😡 ( em ☁️ ) ( LEIA O FIXADO )
(¯ □ ¯ 」) baixaessaporra.com/video/13673862…
Constant Pieters
Constant Pieters 2021-03-05
Awesome, its strength is in the animations. Two suggestions: app crashes when n goes to 0; might be useful to update n when adding "p-hack" given its operationalization (add 1 observations to all samples that do not show a positive effect and reanalyze).
stasha.antonijevic
@Renagalway
Simon Dellicour
Simon Dellicour 2021-03-05
@FMassonnet
Dr Anastasia S Mihailidou FAHA FCSANZ
This is definitely #MedEd #CardioEd #scicomm💥& thank you @krstoffr
MammothReg l'Amish des 🐘
big fan! thanx
David Hajage #JeSuisVacciné
Poke @Nibor_Tolum @Clara_Locher @SagittariusHH C'est beau.
Sarah Myers
Sarah Myers 2021-03-05
THIS IS SO HELPFUL
louislongin
louislongin 2021-03-05
@oriane_armand maybe we can cover this next week..?
Valmir Matos
Valmir Matos 2021-03-05
Congrats from Brazil! It's awesome!!
David Colquhoun 💙
Another quotation from that. "Harold Jeffreys recommends the lump prior only to capture cases where a special value of a parameter is deemed plausible" Sadly in many experiments, zero (or near zero) effects are only too plausible. @dnunan79
Tiago André Marques
Cool visualization!
Homero San Juan
Homero San Juan 2021-03-04
Excelente visualización para entender qué significan los valores p en estadística, usando simulaciones
Luis Ortega Paz
Luis Ortega Paz 2021-03-04
#EAPCI
Begüm Özkısaoğlu
This is cool!
Polina Beloborodova
The most straightforward demonstration of p-value issues that I’ve ever seen.
David Colquhoun 💙
Steven Goodman pointed out years ago that p=0.05 corresponds to a likelihood ratio of about 3. Odds of 3:1 are not very impressive
David Colquhoun 💙
It isn't a *huge* prior -in the absence of good date, ie almost always, it's unreasonable to postulate a value >0.5
♕Deborah Mayo♕
Read: Who's exaggerating what? errorstatistics.files.wordpress.com/2020/05/ex4-ti…
David Colquhoun 💙
I have read it several times. You say "This might only mean it’s worth getting more data to probe for a real effect.". That is how I would interpret p=0.05. But it's far from the interpretation used in practice.
♕Deborah Mayo♕
posteriors using p= have the same deficiencies & are completely distinct from our goals in hypotheses tests
♕Deborah Mayo♕
giving the null a huge spiked prior is radically at odds with how we understand null hypotheses, not to mention, they aren't proper frequentist priors. but we've been through this too many times.
David Colquhoun 💙
PPV is different. It uses the p-less-than definition for likelihood ratio which is inappropriate for tests designed to tell you whether an effect is chance or real. For that you need p-equals calculation of likelihood ratio (or BF) royalsocietypublishing.org/doi/full/10.10…
David Colquhoun 💙
The screening test analogy is not necessary, nor is it an exact analogy, for the reasons I just gave.
David Colquhoun 💙
The crucial assumption in my approach and several others has nothing to do with screening tests. Rather it's that it's sensible to test a point (or near-point) null. I think that that is often the case (but not always).
♕Deborah Mayo♕
this is according to YOU, but not according to people who know how to use p-values correctly & are not trying to fit hypothesis testing into the mold for PPV screening measures.
Marce Cvallos
Marce Cvallos 2021-03-04
I love this! 😍
David Colquhoun 💙
The problem, surely, is that the simulation doesn't distinguish between type 1 error and false positive risk. So it doesn't really help people to understand what's bad about p values. tandfonline.com/doi/pdf/10.108… and fpr-calc.ucl.ac.uk
Dominik Liebl
Dominik Liebl 2021-03-04
P p pppp p-values. Explained through simulations. (Simulations: the only way to understand frequentist statistics ... since simulations allow frequent repetitions.)
Gavin Buckingham
Gavin Buckingham 2021-03-04
This is gorgeous
Luis Benites
Luis Benites 2021-03-04
Very nice! Congrats.
Kai-Markus Mueller
Super awesome, my students will love this!!!
dolores frias-navarro
Thank you
Gareth J. Harvey
Gareth J. Harvey 2021-03-04
So many people (students, lecturers & practitioners) really don't understand P values. This interactive simulation from @krstoffr should help make it a little clearer. I love how it features both p hacking & publication bias. rpsychologist.com/pvalue/
cbergenholtz
cbergenholtz 2021-03-04
Great illustration! Very timely, I can use it in my philosophy of science class today, where we discuss Carney's 'retraction' of her power pose paper (with Cuddy), and more generally how and why false positives appear in the literature.
Dominik Liebl
Dominik Liebl 2021-03-04
P p pppp p-values. Explained through simulations. (Simulations: the only way to understand frequentist statistics ... since simulations allow frequent repetitions.)
Julia Jerke
Julia Jerke 2021-03-04
Awesome! I really like that it integrates p-hacking (here optional stopping: sampling until significance is reached) to illustrate how it contributes to inflated results and, eventually, publication bias. I may borrow this for my teaching 😃
Gavin Buckingham
Gavin Buckingham 2021-03-04
This is gorgeous
Max Berggren 🐍🐼
A++ for under construction gif
Julia Jerke
Julia Jerke 2021-03-04
Very nice! I really like how it features p-hacking (here in the form of sampling until significance is reached) to illustrate how it contributes to inflated results and, eventually, publication bias.
♕Deborah Mayo♕
It would be great to give corresponding severity assessments.
Raimondo Bruno
Raimondo Bruno 2021-03-04
OMG this is GREAT!!!!

(Webmentions sent before 2021 will unfortunately not show up here.)

More Visualizations

Understanding p-values Through Simulations

An interactive simulation to help explain p-values

Maximum Likelihood

An interactive post covering various aspects of maximum likelihood estimation.

Cohen's d

An interactive app to visualize and understand standardized effect sizes.

Statistical Power and Significance Testing

An interactive version of the traditional Type I and II error illustration.

Confidence Intervals

An interactive simulation of confidence intervals

Bayesian Inference

An interactive illustration of prior, likelihood, and posterior.

Correlations

Interactive scatterplot that let's you visualize correlations of various magnitudes.

Equivalence and Non-Inferiority Testing

Explore how superiority, non-inferiority, and equivalence testing relates to a confidence interval

P-value distribution

Explore the expected distribution of p-values under varying alternative hypothesises.

t-distribution

Interactively compare the t- and normal distribution.