In this post I will use the theoretical and empirical sampling distribution of Cohen’s d to show the expected overestimation due to selective publishing. I will look at the overestimation for various sample sizes when the population effect is 0, 0.2, 0.5 and 0.8. The conclusion is that you should be weary of effect sizes from small samples, and that the issue is rather with type M (magnitude) errors than type I errors. At least is clinical psychology the pervasive problem is overestimation of effects and not falsely rejecting null hypothesis.
A common way of illustrating the idea behind statistical power in null hypothesis significance testing, is by plotting the sampling distributions of the null hypothesis and the alternative hypothesis. Typically, these illustrations highlight the regions that correspond to making a type II error, type I error and correctly rejecting the null hypothesis (i.e. the test’s power). In this post I will show how to create such “power plots” using both ggplot and R’s base graphics.
In this post I show some R-examples on how to perform power analyses for mixed-design ANOVAs. The first example is analytical—and adapted from formulas used in G*Power (Faul et al., 2007), and the second example is a Monte Carlo simulation.
Can you tell when error bars based on 95 % CIs or standard errors correspond to a significant p-value? Don’t fret if you think it’s hard, a study from 2005 showed that researchers in psychogoly, behavior neuroscience and medicine had a hard time judging when error bars from two independent groups signified a significant difference
Why are physicists talking about 5-sigma, and what’s it got to do with statistics? In this short post I’ll explain what 5-sigma is and why it’s not a measure of how certain scientist are that they’ve found the Higgs boson
When talking about confidence intervals, Jacob Cohen famously said: “I suspect that the main reason they are not reported is that they are so embarrassingly large!” (Cohen, 1994). In this post I’ll take a look at the relationship between the 95 % CI for Cohen’s d and it’s corresponding sample size.
Background I believe there’s some information to be gained from looking at publication trends over time. But it’s really troublesome to do it by hand; fortunately it’s not so troublesome to do it in R statistical software. Though, data like these should be interpreted with extreme caution ...