Something that never ceases to amaze (depress) me, is how extremely common it is to see causal claims in RCTs, that are not part of the randomization. For instance, the relationship between treatment adherence and outcome, or between alliance and outcome, are often analyzed but seldom experimentally manipulated. This is basically observational research disguised as experimental, but without DAGs, instrumental variables, propensity scores, or any other technique used in observational research. Hence, the effects will almost surely be biased in an unknown direction. This is a post I’ve been planning to write for a long time, but put off since the theory and methods are so well covered in, for instance, epidemiology. However, when brms added support for multivariate formulas, I figured I could show how some of the techniques can be implemented in R, and their Bayesian counterparts using brms. So, in this post I do three things:

1. Rant about how quickly clinical psychology researchers forget about causal inference, once they start analyzing non-randomized variables, such as the effect of treatment adherence (or “compliance”) on treatment outcome in a randomized controlled trial.
2. Show that common regression methods either, makes absolutely no sense, or result in badly biased estimates of the causal dose-response effect, when a third variable influences both adherence and outcome, i.e. when the dose-response relationship is confounded. While instrumental variable methods may allow for unbiased estimates in these scenarios.
3. Show how Bayesian intrumental variable regression can be easily fit using brms new multivariate syntax. And also do a small simulation study to compare the results to lavaan, AER::ivreg and biased lm() models.

Motivating example

The effect of treatment adherence (psychotherapy dose-response effects)

Let’s assume we’ve performed an RCT and compared some psychotherapy condition to some control condition. Patients in these types of trials seldom complete all scheduled sessions, or adhere 100 % to the treatment. It’s rather common for researchers to try to see if there is some relationship between adherence and treatment outcome. Which is a valid question, if there is no added effect of actually “receiving” more of the treatment, then that should be a problem for your theory, or at least some of the components of that theory. However, what usually happens is that researchers somehow forget that the variable “adherence to therapy” is not experimentally manipulated. Unless we have randomized patients to receive 1, 2, …, 10 sessions, or to do 1, 2, …, 10, exposure exercises (and somehow ensured adherence was 100% in all conditions)–then we are now longer in RCT country!

Never stop to acknowledge you’re doing non-randomized comparisons!

A simple model of confounded adherence effects

A really simple scenario with one know (or unknown) confounder is shown in the DAG below.

Each arrow represent causal influence of one variable on another. Y is the observed outcome, adherence is often measured by some proxy, like the number of sessions completed–that we assume is a good indicator of treatment “dose”, it could be any other proxy for treatment adherence. So, what the graph is saying, is that treatment allocation (randomization) influences the outcome (Y) via the amount of treatment received (adherence). Thus, unless something went horribly wrong, randomization should be strongly correlated with S, and only influence Y through the treatment received (S). By definition this makes randomization an instrumental variable. However, the problem is that the effect of S on Y is most likely confounded (U). For instance, baseline functioning might be related both to a better prognosis (better outcome) and treatment adherence. It is easy to come up with scenarios where the relationship could be either positive or negative. Higher baseline functioning could be related to a better prognosis, and to better treatment adherence, or, a patient could be too depressed to actually adhere to the treatment. It is also plausible that worse baseline functioning make you more motivated to adhere to your treatment. Whatever the relationship might be, it is clear that the causal effect of treatment adherence will be confounded.

We will deal with three types of scenarios where a variable $U$ confounds the estimate of treatment adherence:

• Observed confounder: $U$ is an observed variable that is adjusted for.
• Residual confounding: our known confounder $U$ is measured with error.
• Hidden confounding: $U$ is either unknown or unmeasured, and not adjusted for.

The data generating model

First, let’s remind ourselves what is commonly estimated in an RCT is the average treatment effect (ATE), or some related estimate,

Where $Y_i(C)$ is the would-be outcome for patient i after participating in the control condition, and $Y_i(Tx)$ the would be outcome for patient i after participating in the treatment condition. Obviously, we can never observe both $Y_i(C)$ and $Y_i(Tx)$ for the same patient. But thanks to randomization when can infer the ATE. We also assume that the individual treatment effect $Y_i(C)-Y_i(Tx)$ is the same for all individuals, given the same level of adherence.

The data generating model (DGM) is, thus, that adherence depends on $U_i$, the confounder variable,

Where $f()$ is just a function that centers and scales the confounding variable, so that the mean adherence is 7, and $e_i$ is just random variation that influence the relationship between $U_i$ and $S_i$. Moreover, the observed measure of adherence could be a noisy measure of a patient’s actual adherence, $S_i$, e.g. sessions completed,

To generate the outcome, we first simulate $Y_i(C)$, and then calculate $Y_i(Tx)$,

So that ${\beta }_S$ is the causal effect of treatment adherence,

Lastly, the outcome is generated as,

Where $R_i(Tx)$ is 1 for participants allocated to the treatment condition and 0 otherwise.

Different ways to answer the same question

Now, that we have a DGM, let’s generate some fake data and fit different models. We’ll start with a standard OLS regression, the first model would make sense in the absence of any hidden or residual confounding (extremely unlikely). The second model (“naïve OLS”) makes little sense, since it only analyze the treatment group, and as we’ll see, further confounds prognosis with the causal effect of treatment adherence. Sadly, this way of analyzing psychotherapy trials is really common… Anyway, lets generate some fake data. I’ll use a two-level random intercept-only model as a starting point, and assume that the confounder (baseline functioning), is measured without error.

OLS

First, we fit the two standard OLS models where we are adjusting for pretest scores.

Instrumental variables (IV)

Now, let’s try to use treatment assignment as an instrument to better uncover the causal effect of adherence. Two-stage least-squares (2SLS) is a common method to estimate an IV model. The point of this post is not to explain IV regression, it’s explained well in standard econometric textbooks (Wooldridge, 2010; Wooldridge, 2015). And in the context of psychotherapy research in the articles by Maracy and Dunn (2011), or Dunn and Bentall (2007). The very short explanation, is that 2SLS is basically to regress session on the instruments, then regress y on the predicted sessions from Step 1. This can be done using lm() to get point estimates, but the standard errors will be highly suspicious.

However, 2SLS is easier done with ivreg() from the package AER. The same structural model is fit using this code.

The model can also be thought of as a special case of a structural equation model, and fit with e.g. lavaan.

Lastly, using the same approach, a Bayesian IV model can be fit using brms (>2.0.0).

This allows extending the model a lot using brms highly flexible modelling syntax. Moreover, once you’ve exhausted brms capabilities you can get the stancode() and further customize the model yourself.

A small simulation study

Let’s also do a small simulation study, to better show the impact of observed confounding, residual confounding, and hidden confounding. We’ll use the OLS and IV models shown above. For each of the three scenarios, 5000 data sets were generated, with 50 patients per condition. The correlations between the variables are shown in the figure below, the actual values are arbitrary chosen, and the resulting bias, and its direction, is not meant to reflect real data, instead the values are meant to highlight the different types of confounding.

In the figure, s is the latent adherence variable, session the observed adherence, intercept_subject is the baseline confounder, that is measured with or without error by pre, and y is the outcome.

Relative bias

The relative bias of the estimated adherence effect is shown in the figure below.

In the figure, we see for the OLS models, that when there’s:

• Hidden confounding:
  • Adjusting for the baseline variable does not help, which is as expected, since it is not related to adherence.
• Observed confounder:
  • The estimates are biased when the confounder is omitted.
  • Adjusting for the confounder removes the bias. However, we do not recover the true causal effect, since session is a noisy proxy of adherence the effect is attenuated.
• Residual confounding:
  • Controlling for the confounder reduces bias, but due to residual confounding estimates are still biased.

And the overall pattern is that:

• the naïve OLS is as bad idea, and is the most affected by bias in all scenarios. We use control groups for a reason, don’t ignore the control group!
• IV estimates are unbiased.
  • However, large measurement errors in session would also attenuate the IV estimate.

We see the same results if look at the coverage probabilities of the confidence intervals.

Coverage probabilities of the CIs

When less is more–collider bias

Let’s throw in another example of how things can go wrong, this time by adjusting for a collider. We extend the first DAG, and add another covariate therapeutic alliance. A researcher might think that by controlling for X (alliance), the dose-response relationship will be more clearly revealed. Or they might just do a multivariable regression and think “hey, both therapeutic alliance and adherence probably predict treatment outcome, let’s throw both in and see what p-value we get”. However, the problem is that alliance might neither be a predictor of outcome, nor a confounder of the adherence and outcome relationship, instead it could be a common effect of adherence and a hidden confounder (i.e., a collider). Obviously, this example is made up and alliance is just a placeholder–there’s so much going on during a psychological treatment that it would be easy to come up with other colliders. In the DAG bellow, conditioning on alliance opens up a path between adherence and the hidden confounder, that bias the relationship between adherence and outcome even more.

If we also simulate this model, and assume that $U_1$ is measured with error, and use the same IV regressions, but add an OLS model that also includes the collider we get these results.

What we see is that:

• The OLS models that control for the baseline confounder is still biased due to residual confounding, and the naïve OLS is badly biased.
• By adjusting for the collider the bias increase, it would even have been better to not adjust for anything.
• The IV models are unbiased, and as you see below, their CIs have nominal coverage probabilities as well.

Summary

Lastly, to summarize this post:

• Don’t do naïve regressions on treatment adherence dose-response effects. It makes no sense even in the absence of confounding.
• Measurement error will bias your regression estimates even if you adjust for confounders.
• Confounders and colliders are everywhere, make your assumptions explicit. We don’t need more zombie studies claiming XYZ is related to treatment outcome.
• IV regression does not automagically solve all problems. But if you think hard about your data it can be a very useful tool in treatment studies.
• brms new multivariate syntax make it extremely easy to fit Bayesian IV models, that can be extended to much more complicated scenarios than what I covered in this article.

References

• Dunn, G., & Bentall, R. (2007). Modelling treatment-effect heterogeneity in randomized controlled trials of complex interventions (psychological treatments). Statistics in Medicine, 26(26), 4719–4745. https://doi.org/10.1002/sim.2891
• Mohammad Maracy, & Graham Dunn. (2011). Estimating dose-response effects in psychological treatment trials: the role of instrumental variables. Statistical Methods in Medical Research, 20(3), 191–215. https://doi.org/10.1177/0962280208097243
• Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data. MIT press.
• Wooldridge, J. M. (2015). Introductory econometrics: A modern approach. Nelson Education.

Code

Simulation functions

Setup and run the simulation.