Ben Locwin03.06.19
Synthetic Control Arms: What are they?
If you haven’t heard of them, Synthetic Control Arms are control arms of a clinical trial—think treatment vs. control—which don’t actually involve enrolling patients into a trial to receive placebo against which to compare a legitimate therapeutic treatment.
Why they’re not so good
Using an idea to incorporate Real World Evidence (RWE) into clinical trials, the thinking goes that we can take the ‘idea’ of a control group and make a leap that this group can be artificially extracted from society.
Where these data are intended to be pulled from: Electronic health records (EHR/EMR), administrative claims data, patient generated data from fitness trackers(!), home medical equipment, disease registries, and historical trial data. Keep in mind that much of these data aren’t validated.
Expansion of the discussion
Just because one is attempting to implement a concept in a new way doesn’t necessarily mean that it should be done. We can celebrate novel ideas executed well, and should ruthlessly and relentlessly controvert (falsify) those ideas which are ascientific. The impoverished thinking which forms the backbone of this synthetic control arm idea hasn’t adequately considered the most important parts of a control arm within clinical trials.
When ‘control’ provides more control than you thought
Far from being inert, a placebo treatment whether it’s a sugar pill, a sham interaction with a healthcare provider, or a sham surgery, is still a ‘treatment.’ Patients spontaneously get better when they take sugar pills, when a healthcare authority talks to them and listens to them, and when they undergo a surgical ‘non-procedure’ where the surgeon opens an incision, then closes it with no therapeutic surgery performed whatsoever. In each of these three cases, the patient experiences what we call an expectancy effect—that is, they ‘expect’ that their treatment modality will help them, and things get better. In every drug trial, there is a placebo group which responds directionally similarly with the actual treatment group (perhaps except for the overall magnitude (size) of the change). Don’t believe me? Check out this graphic:
Where the red line is the actual performance of the drug therapy, and the green line is the actual performance of the placebo (sugar pill) treatment. This is NOT ‘no treatment,’ and the positive response is NOT trivial.
We almost never run randomized controlled trials in the clinic where the control arm is actually ‘no treatment’—that is, the patient doesn’t undergo any exposure to a sugar pill, a positive discussion with the provider, nothing. In these cases, where there is admittedly a paucity of data, patients don’t perform the same as those in a placebo group—and of course also, then, much differently from the drug or device treatment group.
So the synthetic control arm approach is comparing apples and oranges. We try to make the administration of a medical treatment and placebo treatment the same in as many ways as are practically controllable: same treatment center, same environmental/ambient administration conditions, same provider interactions, same diagnostic/prognostic discussions.*
This approach allows us to see what exactly were the differences between the real treatment modality and the same rituals, but subtracting out the treatment modality.
When we try to move this to having a treatment group, and a ‘no treatment-lazily-measured-out-in-the-wild’ control group, not only does the data variance skyrocket astronomically for that other group, making inferences difficult, but finding what is the ‘true’ treatment effect becomes all but impossible to determine.
Of what value are increased trial speed and fewer patients recruited if the signals we need to detect for safety and efficacy and approval are impossible to confidently determine. This just shifts the uncertainty of well-run clinical trials to the Post-Marketing Surveillance phase to clean up the mess of poorly designed trials.
To attempt to address some of this, FDA has appointed a new deputy commissioner to look into how to best tackle this challenge. Her role will undoubtedly be a difficult one, and include needing to develop various checklists or matrices to demonstrate the evidentiary value of different types of ‘real world’ data, most of which is nonsense. Another assumption within the synthetic control arm approach is that disease progressions are at best case linear, and at worst case, self-similar within disease states. This is another falsifiable assumption, and one made messier within RWE because of varying standard of care treatments.
*Of course, in this list when I say “same” I clearly mean “an attempt at no difference.” These terms are NOT synonymous! The environment and interactions will never be the same, and this adds to the necessary noise in the data across all patient-data-points and all clinical trials. If you hear yourself saying something is ‘the same’ as something else, please stop. The whole point of hypothesis testing is to test against the hypothesis of ‘no difference’ and no two things are exactly ‘the same.’ Except for subatomic particles, where as far as the universe is concerned, one electron or photon is indistinguishable from another, provided they are prespecified to have certain characteristics, such as “spin,” which are identical.
References
Kirsch, I. (2009). The emperor’s new drugs.
Kaptchuk, T., Miller F. (2015). Placebo Effects in Medicine. New England Journal of Medicine; 373;1 8-9.
Fisher, R. (1956). Statistical Methods and Scientific Inference.
Locwin, B. (2019). I say placebo, you say potato: Misunderstandings and misgivings of the placebo phenomenon.
Ben Locwin
Ben Locwin, PhD, MBA, MS, MBB began his foray into healthcare decades ago after he started out as an astrophysicist. He is a popularizer and communicator of science, and has worked in a variety of pharmaceutical organizations (small molecule and biologics), medical device organizations, and within hospitals, clinics, and emergent care centers bringing better healthcare to the end user (i.e., the patients!).
If you haven’t heard of them, Synthetic Control Arms are control arms of a clinical trial—think treatment vs. control—which don’t actually involve enrolling patients into a trial to receive placebo against which to compare a legitimate therapeutic treatment.
Why they’re not so good
Using an idea to incorporate Real World Evidence (RWE) into clinical trials, the thinking goes that we can take the ‘idea’ of a control group and make a leap that this group can be artificially extracted from society.
Where these data are intended to be pulled from: Electronic health records (EHR/EMR), administrative claims data, patient generated data from fitness trackers(!), home medical equipment, disease registries, and historical trial data. Keep in mind that much of these data aren’t validated.
Expansion of the discussion
Just because one is attempting to implement a concept in a new way doesn’t necessarily mean that it should be done. We can celebrate novel ideas executed well, and should ruthlessly and relentlessly controvert (falsify) those ideas which are ascientific. The impoverished thinking which forms the backbone of this synthetic control arm idea hasn’t adequately considered the most important parts of a control arm within clinical trials.
When ‘control’ provides more control than you thought
Far from being inert, a placebo treatment whether it’s a sugar pill, a sham interaction with a healthcare provider, or a sham surgery, is still a ‘treatment.’ Patients spontaneously get better when they take sugar pills, when a healthcare authority talks to them and listens to them, and when they undergo a surgical ‘non-procedure’ where the surgeon opens an incision, then closes it with no therapeutic surgery performed whatsoever. In each of these three cases, the patient experiences what we call an expectancy effect—that is, they ‘expect’ that their treatment modality will help them, and things get better. In every drug trial, there is a placebo group which responds directionally similarly with the actual treatment group (perhaps except for the overall magnitude (size) of the change). Don’t believe me? Check out this graphic:
Where the red line is the actual performance of the drug therapy, and the green line is the actual performance of the placebo (sugar pill) treatment. This is NOT ‘no treatment,’ and the positive response is NOT trivial.
We almost never run randomized controlled trials in the clinic where the control arm is actually ‘no treatment’—that is, the patient doesn’t undergo any exposure to a sugar pill, a positive discussion with the provider, nothing. In these cases, where there is admittedly a paucity of data, patients don’t perform the same as those in a placebo group—and of course also, then, much differently from the drug or device treatment group.
So the synthetic control arm approach is comparing apples and oranges. We try to make the administration of a medical treatment and placebo treatment the same in as many ways as are practically controllable: same treatment center, same environmental/ambient administration conditions, same provider interactions, same diagnostic/prognostic discussions.*
This approach allows us to see what exactly were the differences between the real treatment modality and the same rituals, but subtracting out the treatment modality.
When we try to move this to having a treatment group, and a ‘no treatment-lazily-measured-out-in-the-wild’ control group, not only does the data variance skyrocket astronomically for that other group, making inferences difficult, but finding what is the ‘true’ treatment effect becomes all but impossible to determine.
Of what value are increased trial speed and fewer patients recruited if the signals we need to detect for safety and efficacy and approval are impossible to confidently determine. This just shifts the uncertainty of well-run clinical trials to the Post-Marketing Surveillance phase to clean up the mess of poorly designed trials.
To attempt to address some of this, FDA has appointed a new deputy commissioner to look into how to best tackle this challenge. Her role will undoubtedly be a difficult one, and include needing to develop various checklists or matrices to demonstrate the evidentiary value of different types of ‘real world’ data, most of which is nonsense. Another assumption within the synthetic control arm approach is that disease progressions are at best case linear, and at worst case, self-similar within disease states. This is another falsifiable assumption, and one made messier within RWE because of varying standard of care treatments.
*Of course, in this list when I say “same” I clearly mean “an attempt at no difference.” These terms are NOT synonymous! The environment and interactions will never be the same, and this adds to the necessary noise in the data across all patient-data-points and all clinical trials. If you hear yourself saying something is ‘the same’ as something else, please stop. The whole point of hypothesis testing is to test against the hypothesis of ‘no difference’ and no two things are exactly ‘the same.’ Except for subatomic particles, where as far as the universe is concerned, one electron or photon is indistinguishable from another, provided they are prespecified to have certain characteristics, such as “spin,” which are identical.
References
Kirsch, I. (2009). The emperor’s new drugs.
Kaptchuk, T., Miller F. (2015). Placebo Effects in Medicine. New England Journal of Medicine; 373;1 8-9.
Fisher, R. (1956). Statistical Methods and Scientific Inference.
Locwin, B. (2019). I say placebo, you say potato: Misunderstandings and misgivings of the placebo phenomenon.
Ben Locwin
Ben Locwin, PhD, MBA, MS, MBB began his foray into healthcare decades ago after he started out as an astrophysicist. He is a popularizer and communicator of science, and has worked in a variety of pharmaceutical organizations (small molecule and biologics), medical device organizations, and within hospitals, clinics, and emergent care centers bringing better healthcare to the end user (i.e., the patients!).
