A New Reanalysis of STAR*D Data
How does it inform our current understanding of antidepressants?
A team of research investigators has reanalyzed the patient-level data set of the famous ‘Sequenced Treatment Alternatives to Relieve Depression’ (STAR*D) study, following the analysis plan as specified by the original research protocol and related publications. The analysis by Pigott et al., published earlier this year in BMJ Open, reports substantially lower remission rates than the original STAR*D articles had reported.1
Almost everyone in psychiatry is familiar with the broad details of STAR*D, the largest and longest clinical trial conducted to evaluate medication treatment of depression. See this Wikipedia article for a quick overview, or this 2006 summary paper in American Journal of Psychiatry for scientific results as reported originally. It was conducted across 41 clinical sites (a mix of psychiatric and primary care clinics) funded by the NIMH. The trial had four treatment steps, with different options available in steps 2-4 (with the exception of a very underutilized cognitive behavioral therapy option in step 2, all other interventions were medications). After the four treatment steps, subjects were followed for another 12 months, during which their care was guided by their usual providers and not by the researchers.
Clinical trials, especially those conducted by pharmaceutical companies, rely on volunteers recruited through advertising and exclude patients with medical and psychiatric comorbidities. This limits our ability to generalize those results to “real-world patients.” STAR*D recruited patients who were seeking help in primary care and psychiatric settings and included patients with medical and psychiatric comorbidities. The trial was open-label, and treatment allocation was based on patient preference (it wasn’t randomized).
What most people remember, if they remember anything from the results of STAR*D, is the figure of 67%: after undergoing four treatment steps, 67% of people who completed treatment were in remission from their depression. This statistic is highly cited in academic as well as mainstream publications. In contrast, Pigott et al. reported a cumulative remission rate of 35.0% when using the protocol-stipulated assessment measure and inclusion/exclusion criteria.
What explains this discrepancy? And what does this mean, if anything, for our current understanding and clinical use of antidepressant medications?
There are two kinds of discussions we can have about this STAR*D data reanalysis. The first is focused on the deviations from the original protocol and the lack of transparency about the significance of these deviations in scientific reporting. This is an important story to tell, and I’m glad that it has been told. (I am doubtful that it constitutes scientific misconduct or fraud, as some critics have alleged, but the lack of transparency is certainly concerning and difficult to defend.)
The second is focused on clinical significance. How does this change what we now know about antidepressants? My impression is… not much. STAR*D is two decades old at this point. We have more treatment options now than existed when STAR*D was planned and executed. We have a more somber assessment of the efficacy of traditional antidepressants in general, and partly as a result of STAR*D itself, attention has shifted onto “treatment-resistant depression.” In this sense, the re-analysis, important as it is for the integrity of the scientific record, has little impact on clinical decision-making and serves more as a cautionary tale of how yet another beloved research statistic doesn’t stand up to rigorous scrutiny.
The STAR*D re-analysis, important as it is for the integrity of the scientific record, has little impact on clinical decision-making and serves more as a cautionary tale of how yet another beloved research statistic doesn’t stand up to rigorous scrutiny.
The discrepancy between the original STAR*D publications and the re-analysis by Pigott et al. comes down to three major factors:
STAR*D original publications used QIDS-SR (a self-reported, non-blinded assessment) to report remission and response rates in their summary article. It wasn’t disclosed that the original protocol specifically excluded all non-blinded/
clinic-administered assessments such as the QIDS-SR. The primary outcome measure, as specified by the protocol, was the Hamilton Rating Scale for Depression (HRSD). The re-analysis reported outcomes using HRSD (and considered QIDS-SR in those cases, while a final HRSD assessment was missing).
Although the protocol is silent on this, it was stated in the STAR*D publications that patients with missing exit HRSD scores would automatically be classified as non-remitters. The 67% cumulative remission figure was a “theoretical” rate calculated by assuming that there had been no study drop-outs (i.e. based on the assumption that those who exited the study would have had the same remission rates as those who stayed in the protocol, which isn’t a valid assumption to make.) This becomes significant because the drop-rate in STAR*D was massive. 53.7% of patients dropped out of the study during the acute treatment phase. The recalculated cumulative remission rate of 35.0% treats dropouts as non-remitters, regardless of how many treatment steps they completed.
There are some considerations around the inclusion and exclusion of patients from the analysis. For example, patients with a baseline HRSD score of less than 14 should’ve been excluded from the analysis but weren’t. There are other examples of such deviations from protocol as well which you can read in the BMJ Open paper.
Although the 67% statistic was superficially comforting, even that indicated that more than 30% depressed patients in the study had failed to remit despite four treatment steps. Furthermore, it was highlighted from the very beginning that remission and response rates decreased dramatically in step 3 and step 4, leading to a consolidation of the definition of treatment-resistant depression as a failure to respond to two adequate trials of antidepressant medications (about 50% of patients in STAR*D were treatment-resistant by this definition). Furthermore, there were very high relapse rates in the 12-month naturalistic follow-up period, indicating that even patients who got well did not stay well in the study. According to the AJP summary article, relapse rates were 40%, 55%, 65%, and 71% for those who entered follow-up at each step of the treatment.
The re-analysis doesn’t provide a cumulative response rate, but we can easily calculate it from the data provided in the paper. It is 53.9% after 4 steps. For 54% of patients, their depression severity was reduced to at least half of their initial depression severity. The rest either didn’t respond or dropped out.
For the psychiatric community, STAR*D’s findings have represented a ‘glass half-full’ situation from the moment they were published.
For the psychiatric community, STAR*D’s findings have represented a ‘glass half-full’ situation from the moment they were published. Available pharmacological treatments help a substantial number of depressed individuals, but they fail to offer meaningful relief or sustained relief to many. Numerous interventions have been explored and developed to address it (second generation antipsychotics, transcranial magnetic stimulation, ketamine, esketamine, psilocybin, deep brain stimulation, etc.), with varying degrees of success. In other words, STAR*D’s message or impact was not one of complacency, that all is well, but one that more work is needed. The fact that STAR*D ushered in an era of research and treatment focused on treatment-resistant depression ameliorates the impact of the re-analysis, for better or worse.
It is important to keep in mind that subjects in STAR*D generally had chronic, recurrent depression with multiple psychiatric and medical comorbidities. 56% had two more comorbid psychiatric disorders, and the average patient had 2.5 comorbid medical conditions. The average length of the current major depressive episode was 26 months! Many had already received antidepressant treatment before. The mean illness duration was 16 years. The average number of depressive episodes during their lifetimes was 4.4.
The patients in STAR*D were substantially sicker than patients in usual clinical trials of antidepressants, which is why the remission and response rates were considerably lower.
Peter Kramer writes in Ordinarily Well:
“The intent [in STAR*D] was to attract a range of patients. Only the sickest came… STAR*D showed how psychiatry does with its failures… Commentators considered this outcome disappointing, but is it? A decade and a half into a career of depression, during an episode that’s lasted two years despite treatment, medically complex, sometimes alcoholic patients have a fifty-fifty chance of responding to the first drug they’re offered. Does any specialty do better with its difficult cases?”
Based on the Pigott et al. re-analysis, this would be a 40-60 chance of responding to the first drug, but the point stands. STAR*D remains a glass half-full.
The clinical significance of the drop-outs is debatable. Do drop-outs in clinical trials correspond to drop-outs from actual clinical care? How should we take drop-outs into account when considering, ‘These are your chances of remission or response if you complete all the treatment steps’? Patients drop out of clinical trials for a variety of reasons. In acute treatment, this can be because of side effects, burdensome nature of the research assessments, or the inflexibility of the research protocol to meet their needs; in the follow-up phase, it may even be because patients are well enough that they don’t think they need further treatment and continued participation isn’t worth their time, in addition to the above. Nonetheless, as we have seen above, it is clear that the 67% cumulative figure is misleading in important ways. And it is about time that it is retired.2
Update:
Following the publication of the Pigott re-analysis, there was another development that I was not anticipating at all. A paper in World Psychiatry by Sakurai et al. tried to account for missing data in STAR*D with a new statistical analysis. They estimate the cumulative remission rate by using the inverse probability of censoring weighted (IPCW) Kaplan-Meier method. Furthermore, they compare cumulative remission rates among individuals with and without prior antidepressant treatment history during the ongoing episode. Their analysis is based on QIDS-SR16, and takes the burden of side effects into account, as well as a variety of other factors, such as the baseline depression score. They estimate the cumulative remission rate to be 53.8% at 90 days, 74.5% at 180 days, and 87.5% at 360 days! Cumulative remission for those who had received no antidepressant prior to study entry for the current episode is even higher at 89.1%.
STAR*D authors had estimated a cumulative remission rate of 67% by assuming that those who exited the study would have had the same remission rates as those who stayed in the protocol. Pigott et al. treated all dropouts as non-remitters and calculated a rate of 35%. The Sakurai et al. analysis suggests that the remission rate of drop-outs may actually be higher than that of those who stayed in the study! A somewhat counterintuitive finding, but not without precedent. My suspicion is that the method used by Sakurai et al. overestimates the cumulative remission rate, but their analysis does seem to converge with findings from other studies, such as NESDA which reported a depression remission rate of 79.5% over a two-year period and a median time to remission of 6 months. On the whole, I am of the view that while Pigott et al.’s analysis is faithful to the protocol, it seems unlikely to be the best interpretation of what the STAR*D data shows.
P.S. In addition to the 67% figure, another misleading statistic is the 3% stay-well rate commonly cited in critical spaces such as Mad in America. Robert Whitaker, for example, frequently points out that only 3% of patients in STAR*D remitted and were still well at the end of one year of follow-up. This is misleading because, due to massive drop-outs, this doesn’t translate into the assertion that only 3% of patients with depression remain well in the year after treatment. According to the 2006 summary article, there was data from only 132 patients in the 9-12 months naturalistic follow-up time period. The extremely high rate of drop-outs essentially renders the 3% figure meaningless, and anyone who cites it with a straight face is being deceptive. As an illustration that this 3% figure doesn’t represent real-world outcomes, we can see that in the Lewis et al. 2021 study of antidepressant maintenance treatment published in NEJM, by 52 weeks, relapse had occurred in 39% of patients who had stayed on their antidepressant medication.
P.P.S. STAR*D was not designed to address skepticism around the efficacy of antidepressants. Since there was no placebo comparison group, it cannot be determined how STAR*D outcomes fare in comparison to placebo effects or the passage of time.
Ed Pigott has re-analyzed STAR*D data in past publications as well (e.g. see this and this for an overview), so these results would not be a surprise to those who have been following Pigott’s work over the years; this latest analysis, however, is based on patient-level data obtained from NIMH and offers a more granular picture.
I noticed that in the latest October 2023 issue of World Psychiatry, a big chunk of which is devoted to treatment-resistant depression, the 67% figure isn’t cited even once, although various other figures from STAR*D reports are cited.
This non-medic found this from you very helpful, thank you!
What explains this discrepancy between 67% remission rate that was reported and the 35% that was the reality? Well, look at the financial conflicts of interest. They were massive. This paper is a shining example of the corruption in psychiatry by drug companies, and the breezy and cheerful lies that they told, such as doubling the remission rate, to sell their pills. It is the researchers without fCOI who try to get at the truth.