Confounded dose-response effects of treatment adherence: fitting Bayesian instrumental variable models using brms

Something that never ceases to amaze (depress) me, is how extremely common it is to see causal claims in RCTs, that are not part of the randomization. For instance, the relationship between treatment adherence and outcome, or between alliance and outcome, are often analyzed but seldom experimentally manipulated. This is basically observational research disguised as experimental, but without DAGs, instrumental variables, propensity scores, or any other technique used in observational research. Hence, the effects will almost surely be biased in an unknown direction. This is a post I’ve been planning to write for a long time, but put off since the theory and methods are so well covered in, for instance, epidemiology. However, when brms added support for multivariate formulas, I figured I could show how some of the techniques can be implemented in R, and their Bayesian counterparts using brms. So, in this post I do three things:

  1. Rant about how quickly clinical psychology researchers forget about causal inference, once they start analyzing non-randomized variables, such as the effect of treatment adherence (or “compliance”) on treatment outcome in a randomized controlled trial.
  2. Show that common regression methods either, makes absolutely no sense, or result in badly biased estimates of the causal dose-response effect, when a third variable influences both adherence and outcome, i.e. when the dose-response relationship is confounded. While instrumental variable methods may allow for unbiased estimates in these scenarios.
  3. Show how Bayesian intrumental variable regression can be easily fit using brms new multivariate syntax. And also do a small simulation study to compare the results to lavaan, AER::ivreg and biased lm() models.

Motivating example

The effect of treatment adherence (psychotherapy dose-response effects)

Let’s assume we’ve performed an RCT and compared some psychotherapy condition to some control condition. Patients in these types of trials seldom complete all scheduled sessions, or adhere 100 % to the treatment. It’s rather common for researchers to try to see if there is some relationship between adherence and treatment outcome. Which is a valid question, if there is no added effect of actually “receiving” more of the treatment, then that should be a problem for your theory, or at least some of the components of that theory. However, what usually happens is that researchers somehow forget that the variable “adherence to therapy” is not experimentally manipulated. Unless we have randomized patients to receive 1, 2, …, 10 sessions, or to do 1, 2, …, 10, exposure exercises (and somehow ensured adherence was 100% in all conditions)–then we are now longer in RCT country!

"Confounding bat country meme"

Never stop to acknowledge you’re doing non-randomized comparisons!

A simple model of confounded adherence effects

A really simple scenario with one know (or unknown) confounder is shown in the DAG below.

Adherence dose-response DAG

Each arrow represent causal influence of one variable on another. Y is the observed outcome, adherence is often measured by some proxy, like the number of sessions completed–that we assume is a good indicator of treatment “dose”, it could be any other proxy for treatment adherence. So, what the graph is saying, is that treatment allocation (randomization) influences the outcome (Y) via the amount of treatment received (adherence). Thus, unless something went horribly wrong, randomization should be strongly correlated with S, and only influence Y through the treatment received (S). By definition this makes randomization an instrumental variable. However, the problem is that the effect of S on Y is most likely confounded (U). For instance, baseline functioning might be related both to a better prognosis (better outcome) and treatment adherence. It is easy to come up with scenarios where the relationship could be either positive or negative. Higher baseline functioning could be related to a better prognosis, and to better treatment adherence, or, a patient could be too depressed to actually adhere to the treatment. It is also plausible that worse baseline functioning make you more motivated to adhere to your treatment. Whatever the relationship might be, it is clear that the causal effect of treatment adherence will be confounded.

We will deal with three types of scenarios where a variable confounds the estimate of treatment adherence:

  • Observed confounder: is an observed variable that is adjusted for.
  • Residual confounding: our known confounder is measured with error.
  • Hidden confounding: is either unknown or unmeasured, and not adjusted for.

The data generating model

First, let’s remind ourselves what is commonly estimated in an RCT is the average treatment effect (ATE), or some related estimate,

Where is the would-be outcome for patient i after participating in the control condition, and the would be outcome for patient i after participating in the treatment condition. Obviously, we can never observe both and for the same patient. But thanks to randomization when can infer the ATE. We also assume that the individual treatment effect is the same for all individuals, given the same level of adherence.

The data generating model (DGM) is, thus, that adherence depends on , the confounder variable,

Where is just a function that centers and scales the confounding variable, so that the mean adherence is 7, and is just random variation that influence the relationship between and . Moreover, the observed measure of adherence could be a noisy measure of a patient’s actual adherence, , e.g. sessions completed,

To generate the outcome, we first simulate , and then calculate ,

So that is the causal effect of treatment adherence,

Lastly, the outcome is generated as,

Where is 1 for participants allocated to the treatment condition and 0 otherwise.

Different ways to answer the same question

Now, that we have a DGM, let’s generate some fake data and fit different models. We’ll start with a standard OLS regression, the first model would make sense in the absence of any hidden or residual confounding (extremely unlikely). The second model (“naïve OLS”) makes little sense, since it only analyze the treatment group, and as we’ll see, further confounds prognosis with the causal effect of treatment adherence. Sadly, this way of analyzing psychotherapy trials is really common… Anyway, lets generate some fake data. I’ll use a two-level random intercept-only model as a starting point, and assume that the confounder (baseline functioning), is measured without error.

OLS

First, we fit the two standard OLS models where we are adjusting for pretest scores.

Instrumental variables (IV)

Now, let’s try to use treatment assignment as an instrument to better uncover the causal effect of adherence. Two-stage least-squares (2SLS) is a common method to estimate an IV model. The point of this post is not to explain IV regression, it’s explained well in standard econometric textbooks (Wooldridge, 2010; Wooldridge, 2015). And in the context of psychotherapy research in the articles by Maracy and Dunn (2011), or Dunn and Bentall (2007). The very short explanation, is that 2SLS is basically to regress session on the instruments, then regress y on the predicted sessions from Step 1. This can be done using lm() to get point estimates, but the standard errors will be highly suspicious.

However, 2SLS is easier done with ivreg() from the package AER. The same structural model is fit using this code.

The model can also be thought of as a special case of a structural equation model, and fit with e.g. lavaan.

Lastly, using the same approach, a Bayesian IV model can be fit using brms (>2.0.0).

This allows extending the model a lot using brms highly flexible modelling syntax. Moreover, once you’ve exhausted brms capabilities you can get the stancode() and further customize the model yourself.

A small simulation study

Let’s also do a small simulation study, to better show the impact of observed confounding, residual confounding, and hidden confounding. We’ll use the OLS and IV models shown above. For each of the three scenarios, 5000 data sets were generated, with 50 patients per condition. The correlations between the variables are shown in the figure below, the actual values are arbitrary chosen, and the resulting bias, and its direction, is not meant to reflect real data, instead the values are meant to highlight the different types of confounding.

Confounding of treatment adherence effect, different scenarious

In the figure, s is the latent adherence variable, session the observed adherence, intercept_subject is the baseline confounder, that is measured with or without error by pre, and y is the outcome.

Relative bias

The relative bias of the estimated adherence effect is shown in the figure below.

"Treatment adherence confounding relative bias, OLS and instrumental variables

In the figure, we see for the OLS models, that when there’s:

  • Hidden confounding:
    • Adjusting for the baseline variable does not help, which is as expected, since it is not related to adherence.
  • Observed confounder:
    • The estimates are biased when the confounder is omitted.
    • Adjusting for the confounder removes the bias. However, we do not recover the true causal effect, since session is a noisy proxy of adherence the effect is attenuated.
  • Residual confounding:
    • Controlling for the confounder reduces bias, but due to residual confounding estimates are still biased.

And the overall pattern is that:

  • the naïve OLS is as bad idea, and is the most affected by bias in all scenarios. We use control groups for a reason, don’t ignore the control group!
  • IV estimates are unbiased.
    • However, large measurement errors in session would also attenuate the IV estimate.

We see the same results if look at the coverage probabilities of the confidence intervals.

Coverage probabilities of the CIs

Treatment adherence confounding confidence interval coverage, OLS and instrumental variables

When less is more–collider bias

Let’s throw in another example of how things can go wrong, this time by adjusting for a collider. We extend the first DAG, and add another covariate therapeutic alliance. A researcher might think that by controlling for X (alliance), the dose-response relationship will be more clearly revealed. Or they might just do a multivariable regression and think “hey, both therapeutic alliance and adherence probably predict treatment outcome, let’s throw both in and see what p-value we get”. However, the problem is that alliance might neither be a predictor of outcome, nor a confounder of the adherence and outcome relationship, instead it could be a common effect of adherence and a hidden confounder (i.e., a collider). Obviously, this example is made up and alliance is just a placeholder–there’s so much going on during a psychological treatment that it would be easy to come up with other colliders. In the DAG bellow, conditioning on alliance opens up a path between adherence and the hidden confounder, that bias the relationship between adherence and outcome even more.

Adherence dose-response DAG

If we also simulate this model, and assume that is measured with error, and use the same IV regressions, but add an OLS model that also includes the collider we get these results.

Treatment adherence confounding and collider relative bias, OLS and instrumental variables

What we see is that:

  • The OLS models that control for the baseline confounder is still biased due to residual confounding, and the naïve OLS is badly biased.
  • By adjusting for the collider the bias increase, it would even have been better to not adjust for anything.
  • The IV models are unbiased, and as you see below, their CIs have nominal coverage probabilities as well.

Treatment adherence confounding and collider CI coverage, OLS and instrumental variables

Summary

Lastly, to summarize this post:

  • Don’t do naïve regressions on treatment adherence dose-response effects. It makes no sense even in the absence of confounding.
  • Measurement error will bias your regression estimates even if you adjust for confounders.
  • Confounders and colliders are everywhere, make your assumptions explicit. We don’t need more zombie studies claiming XYZ is related to treatment outcome.
  • IV regression does not automagically solve all problems. But if you think hard about your data it can be a very useful tool in treatment studies.
  • brms new multivariate syntax make it extremely easy to fit Bayesian IV models, that can be extended to much more complicated scenarios than what I covered in this article.

References

Code

Simulation functions

Setup and run the simulation.


Written by Kristoffer Magnusson, a researcher in clinical psychology. You should follow him on Twitter and come hang out on the open science discord Git Gud Science.


Share:

Published February 01, 2018 (View on GitHub)

Buy Me A Coffee

A huge thanks to the 64 supporters who've bought me a 143 coffees!

SCCT/Psychology bought ☕☕☕ (3) coffees

Keep the visualizations coming!

@noelnguyen16 bought ☕ (1) coffee

Hi Kristoffer, many thanks for making all this great stuff available to the community!

Eran Barzilai bought ☕ (1) coffee

These visualizations are awesome! thank you for creating it

Someone bought ☕ (1) coffee

@elena_bolt bought ☕☕☕ (3) coffees

Thank you so much for your work, Kristoffer. I use your visualizations to explain concepts to my tutoring students and they are a huge help.

A random user bought ☕☕☕ (3) coffees

Thank you for making such useful and pretty tools. It not only helped me understand more about power, effect size, etc, but also made my quanti-method class more engaging and interesting. Thank you and wish you a great 2021!

@hertzpodcast bought ☕☕☕ (3) coffees

We've mentioned your work a few times on our podcast and we recently sent a poster to a listener as prize so we wanted to buy you a few coffees. Thanks for the great work that you do!Dan Quintana and James Heathers - Co-hosts of Everything Hertz 

Chris SG bought ☕ (1) coffee

Very nice.

@whlevine bought ☕☕ (2) coffees

Thank you so much for these amazing visualizations. They're a great teaching tool and the allow me to show students things that it would take me weeks or months to program myself.

Someone bought ☕☕ (2) coffees

Cameron Proctor bought ☕☕☕ (3) coffees

Used your vizualization in class today. Thanks!

@Daniel_Brad4d bought ☕☕☕☕☕ (5) coffees

Wonderful work!

Gray Church bought ☕ (1) coffee

Thank you for the visualizations. They are fun and informative.

eshulman@brocku.ca bought ☕☕☕ (3) coffees

My students love these visualizations and so do I! Thanks for helping me make stats more intuitive.

Qamar bought ☕ (1) coffee

Someone bought ☕☕☕ (3) coffees

Adrian Helgå Vestøl bought ☕☕☕ (3) coffees

David Loschelder bought ☕☕☕☕☕ (5) coffees

Terrific work. So very helpful. Thank you very much.

Tanya McGhee bought ☕ (1) coffee

@neilmeigh bought ☕☕☕☕☕ (5) coffees

I am so grateful for your page and can't thank you enough!  

@schultemi bought ☕ (1) coffee

@giladfeldman bought ☕☕☕☕☕ (5) coffees

Wonderful work, I use it every semester and it really helps the students (and me) understand things better. Keep going strong.

Dean Norris bought ☕☕☕☕☕ (5) coffees

@misteryosupjoo bought ☕☕☕ (3) coffees

For a high school teacher of psychology, I would be lost without your visualizations. The ability to interact and manipulate allows students to get it in a very sticky manner. Thank you!!!

Sal bought ☕☕☕☕☕ (5) coffees

Really super useful, especially for teaching. Thanks for this!

Neilo bought ☕ (1) coffee

Really helpful visualisations, thanks!

Chi bought ☕☕☕ (3) coffees

You Cohen's d post really helped me explaining the interpretation to people who don't know stats! Thank you!

Someone bought ☕☕☕ (3) coffees

You doing useful work !! thanks !!

@ArtisanalANN bought ☕☕☕ (3) coffees

Enjoy.

Someone bought ☕ (1) coffee

This is amazing stuff. Very slick. 

Someone bought ☕ (1) coffee

Sarko bought ☕ (1) coffee

Thanks so much for creating this! Really helpful for being able to explain effect size to a clinician I'm doing an analysis for. 

@jsholtes bought ☕☕☕ (3) coffees

Teaching stats to civil engineer undergrads (first time teaching for me, first time for most of them too) and grasping for some good explanations of hypothesis testing, power, and CI's. Love these interactive graphics!

@DominikaSlus bought ☕ (1) coffee

Thank you! This page is super useful. I'll spread the word. 

Someone bought ☕ (1) coffee

dde@paxis.org bought ☕☕☕☕☕ (5) coffees

Very helpful to helping teach teachers about the effects of the Good Behavior Game

@notawful bought ☕☕☕ (3) coffees

Thank you for using your stats and programming gifts in such a useful, generous manner. -Jess

Mateu Servera bought ☕☕☕ (3) coffees

A job that must have cost far more coffees than we can afford you ;-). Thank you.

Melinda Rice bought ☕ (1) coffee

Thank you so much for creating these tools! As we face the challenge of teaching statistical concepts online, this is an invaluable resource.

@tmoldwin bought ☕ (1) coffee

Fantastic resource. I think you would be well served to have one page indexing all your visualizations, that would make it more accessible for sharing as a common resource.

@cdrawn bought ☕☕☕ (3) coffees

Thank you! Such a great resource for teaching these concepts, especially CI, Power, correlation.

Someone bought ☕ (1) coffee

Fantastic Visualizations! Amazing way to to demonstrate how n/power/beta/alpha/effect size are all interrelated - especially for visual learners! Thank you for creating this?

@akreutzer82 bought ☕☕☕☕☕ (5) coffees

Amazing visualizations! Thank you!

@rdh_CLE bought ☕☕☕☕☕ (5) coffees

So good!

@jackferd bought ☕ (1) coffee

Incredible visualizations and the best power analysis software on R.

Julia bought ☕☕☕ (3) coffees

Fantastic work with the visualizations!

@felixthoemmes bought ☕☕☕ (3) coffees

@dalejbarr bought ☕☕☕ (3) coffees

Your work is amazing! I use your visualizations often in my teaching. Thank you. 

@notawful bought ☕☕ (2) coffees

Thank you for sharing your visualization skills with the rest of us! I use them frequently when teaching intro stats. 

Cameron Proctor bought ☕ (1) coffee

Great website!

Someone bought ☕ (1) coffee

Hanah Chapman bought ☕ (1) coffee

Thank you for this work!!

Someone bought ☕ (1) coffee

Jayme bought ☕ (1) coffee

Nice explanation and visual guide of Cohen's d

Bart Comly Boyce bought ☕ (1) coffee

thank you

Dr. Mitchell Earleywine bought ☕ (1) coffee

This site is superb!

Florent bought ☕ (1) coffee

Zampeta bought ☕ (1) coffee

thank you for sharing your work. 

Mila bought ☕ (1) coffee

Thank you for the website, made me smile AND smarter :O enjoy your coffee! :)

Deb bought ☕ (1) coffee

Struggling with statistics and your interactive diagram made me smile to see that someone cares enough about us strugglers to make a visual to help us out!😍 

Someone bought ☕ (1) coffee

@exerpsysing bought ☕ (1) coffee

Much thanks! Visualizations are key to my learning style! 

@PsychoMouse bought ☕☕☕ (3) coffees

Excellent!  Well done!  SOOOO Useful!😊 🐭 

Someone bought ☕ (1) coffee

Sponsors

You can sponsor my open source work using GitHub Sponsors and have your name shown here.

Backers ✨❤️

Questions & Comments

Please use GitHub Discussions for any questions related to this post, or open an issue on GitHub if you've found a bug or wan't to make a feature request.

Webmentions

There are no webmentions for this page

(Webmentions sent before 2021 will unfortunately not show up here.)

Archived Comments (2)

D
devangna raj 2018-07-10

Great post dear. It definitely has increased my knowledge on R Programming. Please keep sharing similar write ups of yours. You can check this too for R Programming tutorial as i have recorded this recently on R Programming. and i'm sure it will be helpful to you.https://www.youtube.com/watch?v=rgF...

A
Aki Vehtari 2018-02-05

”Rant about how quickly clinical psychology researchers forget about casual inference"

Casual inference?