# Confounded dose-response effects of treatment adherence: fitting Bayesian instrumental variable models using brms

Something that never ceases to amaze (depress) me, is how extremely common it is to see causal claims in RCTs, that are not part of the randomization. For instance, the relationship between treatment adherence and outcome, or between alliance and outcome, are often analyzed but seldom experimentally manipulated. This is basically observational research disguised as experimental, but without DAGs, instrumental variables, propensity scores, or *any* other technique used in observational research. Hence, the effects will *almost surely* be biased in an unknown direction. This is a post I’ve been planning to write for a long time, but put off since the theory and methods are so well covered in, for instance, epidemiology. However, when `brms`

added support for multivariate formulas, I figured I could show how some of the techniques can be implemented in R, and their Bayesian counterparts using `brms`

. So, in this post I do three things:

- Rant about how quickly clinical psychology researchers forget about
**causal inference**, once they start analyzing non-randomized variables, such as the effect of**treatment adherence**(or “compliance”) on treatment outcome in a randomized controlled trial. - Show that common regression methods either, makes absolutely no sense, or result in badly biased estimates of the causal dose-response effect, when a third variable influences both adherence and outcome, i.e. when the dose-response relationship is confounded. While
**instrumental variable**methods*may*allow for unbiased estimates in these scenarios. - Show how
**Bayesian intrumental variable**regression can be easily fit using`brms`

new multivariate syntax. And also do a small simulation study to compare the results to`lavaan`

,`AER::ivreg`

and biased`lm()`

models.

# Motivating example

## The effect of treatment adherence (psychotherapy dose-response effects)

Let’s assume we’ve performed an RCT and compared some psychotherapy condition to some control condition. Patients in these types of trials seldom complete all scheduled sessions, or adhere 100 % to the treatment. It’s rather common for researchers to try to see if there is some relationship between adherence and treatment outcome. Which is a valid question, if there is no added effect of actually “receiving” more of the treatment, then that *should* be a problem for your theory, or at least some of the components of that theory. However, what usually happens is that researchers somehow forget that the variable “adherence to therapy” is not experimentally manipulated. Unless we have randomized patients to receive 1, 2, …, 10 sessions, or to do 1, 2, …, 10, exposure exercises (and somehow ensured adherence was 100% in all conditions)–then we are now longer in RCT country!

*Never stop to acknowledge you’re doing non-randomized comparisons!*

## A simple model of confounded adherence effects

A really simple scenario with one know (or unknown) confounder is shown in the DAG below.

Each arrow represent *causal* influence of one variable on another. `Y`

is the observed outcome, `adherence`

is often measured by some proxy, like the number of sessions completed–that we assume is a good indicator of treatment “dose”, it could be any other proxy for treatment adherence. So, what the graph is saying, is that treatment allocation (randomization) influences the outcome (Y) via the amount of treatment received (adherence). Thus, unless something went horribly wrong, randomization should be strongly correlated with S, and only influence Y through the treatment received (S). By definition this makes randomization an *instrumental variable*. However, the problem is that the effect of S on Y is most likely confounded (U). For instance, baseline functioning might be related both to a better prognosis (better outcome) and treatment adherence. It is easy to come up with scenarios where the relationship could be either positive or negative. Higher baseline functioning could be related to a better prognosis, and to better treatment adherence, or, a patient could be too depressed to actually adhere to the treatment. It is also plausible that worse baseline functioning make you more motivated to adhere to your treatment. Whatever the relationship might be, it is clear that the causal effect of treatment adherence will be confounded.

We will deal with three types of scenarios where a variable $U$ confounds the estimate of treatment adherence:

**Observed confounder**: $U$ is an observed variable that is adjusted for.**Residual confounding**: our known confounder $U$ is measured with error.**Hidden confounding**: $U$ is either unknown or unmeasured, and not adjusted for.

# The data generating model

First, let’s remind ourselves what is commonly estimated in an RCT is the *average treatment effect* (ATE), or some related estimate,

Where $Y_{i}(C)$ is the would-be outcome for patient *i* after participating in the control condition, and $Y_{i}(Tx)$ the would be outcome for patient *i* after participating in the treatment condition. Obviously, we can never observe both $Y_{i}(C)$ and $Y_{i}(Tx)$ for the same patient. But thanks to randomization when can infer the ATE. We also assume that the individual treatment effect $Y_{i}(C)−Y_{i}(Tx)$ is the same for all individuals, given the same level of adherence.

The data generating model (DGM) is, thus, that adherence depends on $U_{i}$, the confounder variable,

$S_{i}=7−f(U_{i})+e_{1i}.$Where $f()$ is just a function that centers and scales the confounding variable, so that the mean adherence is 7, and $e_{i}$ is just random variation that influence the relationship between $U_{i}$ and $S_{i}$. Moreover, the *observed* measure of adherence could be a noisy measure of a patient’s actual adherence, $S_{i}$, e.g. sessions completed,

To generate the outcome, we first simulate $Y_{i}(C)$, and then calculate $Y_{i}(Tx)$,

$Y(Tx)_{i}=Y(C)_{i}−β_{S}S_{i}.$So that $β_{S}$ is the causal effect of treatment adherence,

$E(Δ_{i}∣S_{i})=β_{s}S_{i}.$Lastly, the outcome is generated as,

$Y_{i}=Y_{i}(C)[1−R_{i}(Tx)]+Y_{i}(Tx)R_{i}(Tx).$Where $R_{i}(Tx)$ is 1 for participants allocated to the treatment condition and 0 otherwise.

# Different ways to answer the same question

Now, that we have a DGM, let’s generate some fake data and fit different models. We’ll start with a standard OLS regression, the first model would make sense in the absence of any hidden or residual confounding (extremely unlikely). The second model (“naïve OLS”) makes little sense, since it only analyze the treatment group, and as we’ll see, further confounds prognosis with the causal effect of treatment adherence. Sadly, this way of analyzing psychotherapy trials is really common… Anyway, lets generate some fake data. I’ll use a two-level random intercept-only model as a starting point, and assume that the confounder (baseline functioning), is measured without error.

## OLS

First, we fit the two standard OLS models where we are adjusting for pretest scores.

## Instrumental variables (IV)

Now, let’s try to use treatment assignment as an *instrument* to better uncover the causal effect of adherence. Two-stage least-squares (2SLS) is a common method to estimate an IV model. The point of this post is not to explain IV regression, it’s explained well in standard econometric textbooks (Wooldridge, 2010; Wooldridge, 2015). And in the context of psychotherapy research in the articles by Maracy and Dunn (2011), or Dunn and Bentall (2007). The *very* short explanation, is that 2SLS is basically to regress `session`

on the instruments, then regress `y`

on the predicted sessions from Step 1. This can be done using `lm()`

to get point estimates, but the standard errors will be highly suspicious.

However, 2SLS is easier done with `ivreg()`

from the package `AER`

. The same structural model is fit using this code.

The model can also be thought of as a special case of a structural equation model, and fit with e.g. `lavaan`

.

Lastly, using the same approach, a Bayesian IV model can be fit using `brms`

(>2.0.0).

This allows extending the model a lot using `brms`

highly flexible modelling syntax. Moreover, once you’ve exhausted `brms`

capabilities you can get the `stancode()`

and further customize the model yourself.

# A small simulation study

Let’s also do a small simulation study, to better show the impact of observed confounding, residual confounding, and hidden confounding. We’ll use the OLS and IV models shown above. For each of the three scenarios, 5000 data sets were generated, with 50 patients per condition. The correlations between the variables are shown in the figure below, the actual values are arbitrary chosen, and the resulting bias, and its direction, is not meant to reflect real data, instead the values are meant to highlight the different types of confounding.

In the figure, *s* is the latent adherence variable, *session* the observed adherence, *intercept_subject* is the baseline confounder, that is measured with or without error by *pre*, and *y* is the outcome.

## Relative bias

The relative bias of the estimated adherence effect is shown in the figure below.

In the figure, we see for the OLS models, that when there’s:

**Hidden confounding**:- Adjusting for the baseline variable does not help, which is as expected, since it is not related to
*adherence*.

- Adjusting for the baseline variable does not help, which is as expected, since it is not related to
**Observed confounder**:- The estimates are biased when the confounder is omitted.
- Adjusting for the confounder removes the bias. However, we do not recover the true causal effect, since
*session*is a noisy proxy of adherence the effect is attenuated.

**Residual confounding**:- Controlling for the confounder reduces bias, but due to residual confounding estimates are still biased.

And the overall pattern is that:

- the naïve OLS is as bad idea, and is the most affected by bias in all scenarios. We use control groups for a reason, don’t ignore the control group!
- IV estimates are unbiased.
- However, large measurement errors in
*session*would also attenuate the IV estimate.

- However, large measurement errors in

We see the same results if look at the coverage probabilities of the confidence intervals.

## Coverage probabilities of the CIs

## When less is more–collider bias

Let’s throw in another example of how things can go wrong, this time by adjusting for a collider. We extend the first DAG, and add another covariate *therapeutic alliance*. A researcher might think that by controlling for X (alliance), the dose-response relationship will be more clearly revealed. Or they might just do a multivariable regression and think *“hey, both therapeutic alliance and adherence probably predict treatment outcome, let’s throw both in and see what p-value we get”*. However, the problem is that alliance might neither be a predictor of outcome, nor a confounder of the adherence and outcome relationship, instead it could be a common effect of adherence and a hidden confounder (i.e., a collider). Obviously, this example is made up and *alliance* is just a placeholder–there’s so much going on during a psychological treatment that it would be easy to come up with other colliders. In the DAG bellow, conditioning on alliance opens up a path between adherence and the hidden confounder, that bias the relationship between adherence and outcome *even* more.

If we also simulate this model, and assume that $U_{1}$ is measured with error, and use the same IV regressions, but add an OLS model that also includes the collider we get these results.

What we see is that:

- The OLS models that control for the baseline confounder is still biased due to residual confounding, and the naïve OLS is badly biased.
- By adjusting for the collider the bias increase, it would even have been better to not adjust for anything.
- The IV models are unbiased, and as you see below, their CIs have nominal coverage probabilities as well.

# Summary

Lastly, to summarize this post:

- Don’t do naïve regressions on treatment adherence dose-response effects. It makes no sense even in the absence of confounding.
- Measurement error
*will*bias your regression estimates even if you adjust for confounders. - Confounders and colliders are everywhere, make your assumptions explicit. We don’t need more zombie studies claiming XYZ is related to treatment outcome.
- IV regression does not
*automagically*solve all problems. But if you think hard about your data it can be a very useful tool in treatment studies. `brms`

new multivariate syntax make it extremely easy to fit Bayesian IV models, that can be extended to much more complicated scenarios than what I covered in this article.

# References

- Dunn, G., & Bentall, R. (2007). Modelling treatment-effect heterogeneity in randomized controlled trials of complex interventions (psychological treatments). Statistics in Medicine, 26(26), 4719–4745. https://doi.org/10.1002/sim.2891
- Mohammad Maracy, & Graham Dunn. (2011). Estimating dose-response effects in psychological treatment trials: the role of instrumental variables. Statistical Methods in Medical Research, 20(3), 191–215. https://doi.org/10.1177/0962280208097243
- Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data. MIT press.
- Wooldridge, J. M. (2015). Introductory econometrics: A modern approach. Nelson Education.

# Code

## Simulation functions

## Setup and run the simulation.

Written by **Kristoffer Magnusson**, a researcher in clinical psychology. You should follow him on Twitter and come hang out on the open science discord Git Gud Science.

Published February 01, 2018 (View on GitHub)

### Buy Me A Coffee

A huge thanks to the **100** supporters who've bought me a **225** coffees!

Jason Rinaldo bought ☕☕☕☕☕☕☕☕☕☕ (10) coffees

I've been looking for applets that show this for YEARS, for demonstrations for classes. Thank you so much! Students do not need to tolarate my whiteboard scrawl now. I'm sure they'd appreciate you, too.l

@LinneaGandhi bought ☕☕☕☕☕ (5) coffees

This is awesome! Thank you for creating these. Definitely using for my students, and me! :-)

@ICH8412 bought ☕☕☕☕☕ (5) coffees

very useful for my students I guess

@KelvinEJones bought ☕☕☕☕☕ (5) coffees

Preparing my Master's student for final oral exam and stumbled on your site. We are discussing in lab meeting today. Coffee for everyone.

Someone bought ☕☕☕☕☕ (5) coffees

What a great site

@Daniel_Brad4d bought ☕☕☕☕☕ (5) coffees

Wonderful work!

David Loschelder bought ☕☕☕☕☕ (5) coffees

Terrific work. So very helpful. Thank you very much.

@neilmeigh bought ☕☕☕☕☕ (5) coffees

I am so grateful for your page and can't thank you enough!

@giladfeldman bought ☕☕☕☕☕ (5) coffees

Wonderful work, I use it every semester and it really helps the students (and me) understand things better. Keep going strong.

Dean Norris bought ☕☕☕☕☕ (5) coffees

Sal bought ☕☕☕☕☕ (5) coffees

Really super useful, especially for teaching. Thanks for this!

dde@paxis.org bought ☕☕☕☕☕ (5) coffees

Very helpful to helping teach teachers about the effects of the Good Behavior Game

@akreutzer82 bought ☕☕☕☕☕ (5) coffees

Amazing visualizations! Thank you!

@rdh_CLE bought ☕☕☕☕☕ (5) coffees

So good!

Someone bought ☕☕☕ (3) coffees

@PhysioSven bought ☕☕☕ (3) coffees

Amazing illustrations, there is not enough coffee in the world for enthusiasts like you! Thanks!

Cheryl@CurtinUniAus bought ☕☕☕ (3) coffees

🌟What a great contribution - thanks Kristoffer!

vanessa moran bought ☕☕☕ (3) coffees

Wow - your website is fantastic, thank you for making it.

Someone bought ☕☕☕ (3) coffees

mikhail.saltychev@gmail.com bought ☕☕☕ (3) coffees

Thank you Kristoffer This is a nice site, which I have been used for a while. Best Prof. Mikhail Saltychev (Turku University, Finland)

Someone bought ☕☕☕ (3) coffees

Ruslan Klymentiev bought ☕☕☕ (3) coffees

@lkizbok bought ☕☕☕ (3) coffees

Keep up the nice work, thank you!

@TELLlab bought ☕☕☕ (3) coffees

Thanks - this will help me to teach tomorrow!

SCCT/Psychology bought ☕☕☕ (3) coffees

Keep the visualizations coming!

@elena_bolt bought ☕☕☕ (3) coffees

Thank you so much for your work, Kristoffer. I use your visualizations to explain concepts to my tutoring students and they are a huge help.

A random user bought ☕☕☕ (3) coffees

Thank you for making such useful and pretty tools. It not only helped me understand more about power, effect size, etc, but also made my quanti-method class more engaging and interesting. Thank you and wish you a great 2021!

@hertzpodcast bought ☕☕☕ (3) coffees

We've mentioned your work a few times on our podcast and we recently sent a poster to a listener as prize so we wanted to buy you a few coffees. Thanks for the great work that you do!Dan Quintana and James Heathers - Co-hosts of Everything Hertz

Cameron Proctor bought ☕☕☕ (3) coffees

Used your vizualization in class today. Thanks!

eshulman@brocku.ca bought ☕☕☕ (3) coffees

My students love these visualizations and so do I! Thanks for helping me make stats more intuitive.

Someone bought ☕☕☕ (3) coffees

Adrian Helgå Vestøl bought ☕☕☕ (3) coffees

@misteryosupjoo bought ☕☕☕ (3) coffees

For a high school teacher of psychology, I would be lost without your visualizations. The ability to interact and manipulate allows students to get it in a very sticky manner. Thank you!!!

Chi bought ☕☕☕ (3) coffees

You Cohen's d post really helped me explaining the interpretation to people who don't know stats! Thank you!

Someone bought ☕☕☕ (3) coffees

You doing useful work !! thanks !!

@ArtisanalANN bought ☕☕☕ (3) coffees

Enjoy.

@jsholtes bought ☕☕☕ (3) coffees

Teaching stats to civil engineer undergrads (first time teaching for me, first time for most of them too) and grasping for some good explanations of hypothesis testing, power, and CI's. Love these interactive graphics!

@notawful bought ☕☕☕ (3) coffees

Thank you for using your stats and programming gifts in such a useful, generous manner. -Jess

Mateu Servera bought ☕☕☕ (3) coffees

A job that must have cost far more coffees than we can afford you ;-). Thank you.

@cdrawn bought ☕☕☕ (3) coffees

Thank you! Such a great resource for teaching these concepts, especially CI, Power, correlation.

Julia bought ☕☕☕ (3) coffees

Fantastic work with the visualizations!

@felixthoemmes bought ☕☕☕ (3) coffees

@dalejbarr bought ☕☕☕ (3) coffees

Your work is amazing! I use your visualizations often in my teaching. Thank you.

@PsychoMouse bought ☕☕☕ (3) coffees

Excellent! Well done! SOOOO Useful!😊 🐭

Dan Sanes bought ☕☕ (2) coffees

this is a superb, intuitive teaching tool!

@whlevine bought ☕☕ (2) coffees

Thank you so much for these amazing visualizations. They're a great teaching tool and the allow me to show students things that it would take me weeks or months to program myself.

Someone bought ☕☕ (2) coffees

@notawful bought ☕☕ (2) coffees

Thank you for sharing your visualization skills with the rest of us! I use them frequently when teaching intro stats.

Someone bought ☕ (1) coffee

Michael Hansen bought ☕ (1) coffee

ALEXANDER VIETHEER bought ☕ (1) coffee

mather bought ☕ (1) coffee

Someone bought ☕ (1) coffee

Bastian Jaeger bought ☕ (1) coffee

Thanks for making the poster designs OA, I just hung two in my office and they look great!

@ValerioVillani bought ☕ (1) coffee

Thanks for your work.

Someone bought ☕ (1) coffee

Great work!

@YashvinSeetahul bought ☕ (1) coffee

Someone bought ☕ (1) coffee

Angela bought ☕ (1) coffee

Thank you for building such excellent ways to convey difficult topics to students!

@inthelabagain bought ☕ (1) coffee

Really wonderful visuals, and such a fantastic and effective teaching tool. So many thanks!

Someone bought ☕ (1) coffee

Someone bought ☕ (1) coffee

Yashashree Panda bought ☕ (1) coffee

I really like your work.

Ben bought ☕ (1) coffee

You're awesome. I have students in my intro stats class say, "I get it now," after using your tool. Thanks for making my job easier.

Gabriel Recchia bought ☕ (1) coffee

Incredibly useful tool!

Shiseida Sade Kelly Aponte bought ☕ (1) coffee

Thanks for the assistance for RSCH 8210.

@Benedikt_Hell bought ☕ (1) coffee

Great tools! Thank you very much!

Amalia Alvarez bought ☕ (1) coffee

@noelnguyen16 bought ☕ (1) coffee

Hi Kristoffer, many thanks for making all this great stuff available to the community!

Eran Barzilai bought ☕ (1) coffee

These visualizations are awesome! thank you for creating it

Someone bought ☕ (1) coffee

Chris SG bought ☕ (1) coffee

Very nice.

Gray Church bought ☕ (1) coffee

Thank you for the visualizations. They are fun and informative.

Qamar bought ☕ (1) coffee

Tanya McGhee bought ☕ (1) coffee

@schultemi bought ☕ (1) coffee

Neilo bought ☕ (1) coffee

Really helpful visualisations, thanks!

Someone bought ☕ (1) coffee

This is amazing stuff. Very slick.

Someone bought ☕ (1) coffee

Sarko bought ☕ (1) coffee

Thanks so much for creating this! Really helpful for being able to explain effect size to a clinician I'm doing an analysis for.

@DominikaSlus bought ☕ (1) coffee

Thank you! This page is super useful. I'll spread the word.

Someone bought ☕ (1) coffee

Melinda Rice bought ☕ (1) coffee

Thank you so much for creating these tools! As we face the challenge of teaching statistical concepts online, this is an invaluable resource.

@tmoldwin bought ☕ (1) coffee

Fantastic resource. I think you would be well served to have one page indexing all your visualizations, that would make it more accessible for sharing as a common resource.

Someone bought ☕ (1) coffee

Fantastic Visualizations! Amazing way to to demonstrate how n/power/beta/alpha/effect size are all interrelated - especially for visual learners! Thank you for creating this?

@jackferd bought ☕ (1) coffee

Incredible visualizations and the best power analysis software on R.

Cameron Proctor bought ☕ (1) coffee

Great website!

Someone bought ☕ (1) coffee

Hanah Chapman bought ☕ (1) coffee

Thank you for this work!!

Someone bought ☕ (1) coffee

Jayme bought ☕ (1) coffee

Nice explanation and visual guide of Cohen's d

Bart Comly Boyce bought ☕ (1) coffee

thank you

Dr. Mitchell Earleywine bought ☕ (1) coffee

This site is superb!

Florent bought ☕ (1) coffee

Zampeta bought ☕ (1) coffee

thank you for sharing your work.

Mila bought ☕ (1) coffee

Thank you for the website, made me smile AND smarter :O enjoy your coffee! :)

Deb bought ☕ (1) coffee

Struggling with statistics and your interactive diagram made me smile to see that someone cares enough about us strugglers to make a visual to help us out!😍

Someone bought ☕ (1) coffee

@exerpsysing bought ☕ (1) coffee

Much thanks! Visualizations are key to my learning style!

Someone bought ☕ (1) coffee

## Sponsors

You can sponsor my open source work using GitHub Sponsors and have your name shown here.

Backers ✨❤️

#### Questions & Comments

Please use GitHub Discussions for any questions related to this post, or open an issue on GitHub if you've found a bug or wan't to make a feature request.

## Archived Comments (2)

Great post dear. It definitely has increased my knowledge on R Programming. Please keep sharing similar write ups of yours. You can check this too for R Programming tutorial as i have recorded this recently on R Programming. and i'm sure it will be helpful to you.https://www.youtube.com/watch?v=rgF...

”Rant about how quickly clinical psychology researchers forget about casual inference"

Casual inference?