Heterogeneous treatment effects and homogeneous outcome variances

January 10, 2020

Recently there has been a couple of meta-analyses investigating heterogeneous treatment effects by analyzing the ratio of the outcome variances in the treatment and control group. The argument made in these articles is that if individuals differ in their response, then observed variances in the treatment and control group in RCTs should differ. For instance, Winkelbeiner et al. (2019) write:

The SDs of the pretreatment and posttreatment outcome difference scores in the treatment and control groups consist of the same variance components, including the within-patient variation. The treatment group, however, may also include the additional treatment-by-patient interaction, which could indicate the presence of individual response differences. Thus, in the case of a variable treatment effect, an increase of the variance in the treatment group, compared with the control group, should be observable.

Altough I agree with much of what’s written in these articles, I’m still not completely sold on the core argument. If we filter the argument through my (perhaps missinformed) understanding of the ideas, we can write the following model for the individual-level treatment effect (ITE):

where is the outcome after treatment and after placebo. For a single patient we can only observe one of these potential outcomes. We can see that the only difference between an individual’s treatment-free outcome and their outcome after treatment is the causal effect of the treatment. We have two variance components: 1) the treatment effects (), and 2) all other sources of variance (). Let’s assume they are bivariate normal,

Hence, the treatment-free outcome is potentially correlated with the individual-level treatment effect. Thus, in an RCT the observed variance in each arm would be:

It should be clear that all variance in the treatment effect is represented by ,

Now let’s investigate under what conditions we can have equal variances and treatment effect heterogeneity, i.e., when is

We can rewrite the expression above as,

We can see that this ratio will only equal 1 when the covariance exactly cancel the treatment effect variance, i.e., when

Under this model it seems quite unlikely that heterogeneous effects are present when the outcome variances are equal in magnitude, as they could only be present given a highly specific correlation and for all other values of the outcome variances will be heterogeneous. This result agrees with what Cortés et al. (2019) conclude in their supplement. However, why must we assume that the treatment variance is an entirely separate variance component? Let us see what happens if we change that assumption.

What if the treatment changes one of the variance components?

Let’s assume that there’s a causal mechanism that causes some or all of the variance in symptoms. This variable, M, is now a source of variance in both the treatment and the control group. However, if the treatment impacts the outcome by fixing a dysfunction in this causal mechanism (i.e., the treatment effect is mediated) then this can be a source of treatment effect heterogeneity without having to introduce a new variance component in the treatment group. Let’s write down this model, first we have an outcome,

where is an individuals assigned treatment, and the value of the mediator realized depending on the assigment . The potential outcomes after treatment or control is then,

and we assume that the potential outcome values of the mediator is correlated,

Now the outcome variances in each group can be written as,

Hence variances are equal in the treatment and control group if the variance of the mediator is the same in each group (and that the effect of the mediator on the outcome is the same). Let’s assume that’s the case so that we have . Then the variance of the individual-level effects is,

We can see that can only be 0 if . Thus in this example, the results are reversed, and homogeneous effects are now only possible if individuals’ potential outcomes on the mediator are perfectly (positively) correlated. The point here is not to claim that heterogeneous effects are likely and that “precision psychiatry” is the way forward. My point is simply that I’m not sure how much we can learn from looking at the ratio of outcome variances.

A Numerical Example

Here is a numerical example of equal outcome variances with varying degrees of heterogeneous individual-level treatment effects. Although a simulation is not needed here, I know some people prefer it over equations.

I base these values on Plöderl and Hengartner (2019). The SD in each group is 8, with an average treatment effect of -2 points on the Hamilton Depression Rating Scale (HDRS). I arbitrarily assume that 25% of the outcome variance is caused by The Causal Mechanism and that 50% of the total treatment effect is mediated.

TXmean(y)sd(y)cor(Y0_M0, Y1_M1)cor(M0, M1)mean(Y1_M1 - Y0_M0)sd(Y1_M1 - Y0_M0)

Let us plot as a function of the correlation of potential mediator outcomes.

unnamed chunk 3 1

We can also plot the proportion of “responders” (improve by more than 5 points on HDRS) as a function of the correlation.

unnamed chunk 4 1

We can also take a sample of participants and plot their potential outcomes.

unnamed chunk 5 1


  • Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., … Cobo, E. (2019). Does evidence support the high expectations placed in precision medicine? A bibliographic review. F1000Research, 7, 30. https://doi.org/10.12688/f1000research.13490.5
  • Munkholm, K. (n.d.). Individual response to antidepressants for depression in adults – a simulation study and meta-analysis. 10.
  • Plöderl, M., & Hengartner, M. P. (2019). What are the chances for personalised treatment with antidepressants? Detection of patient-by-treatment interaction with a variance ratio meta-analysis. BMJ Open, 9(12). https://doi.org/10.1136/bmjopen-2019-034816
  • Winkelbeiner, S., Leucht, S., Kane, J. M., & Homan, P. (2019). Evaluation of Differences in Individual Treatment Response in Schizophrenia Spectrum Disorders: A Meta-analysis. JAMA Psychiatry, 76(10), 1063–1073. https://doi.org/10.1001/jamapsychiatry.2019.1530