Nev Jones, Lauren Fowler, and Shannon Pagdon
PathLab University of Pittsburgh, School of Social Work
In response to recent public characterizations that minimize the evaluation’s documentation of harm and due process failures, evaluation co-lead Dr. Nev Jones and evaluation project members Lauren Fowler and Shannon Pagdon offer the following account of what the evaluation and their qualitative work actually found. Note that this post should not be construed as representing the views or opinions of evaluation partner HSRI.
Background. Through ‘Kendra’s Law,’ a 1999 statute designed to enable involuntary outpatient treatment, New York became one of the first states in the US to implement what is commonly referred to as “Assisted Outpatient Treatment” (or AOT). Due to the complex civil rights issues such legislation involved, lawmakers enacted the law with sunset clauses (expiration of the law unless extended by the state legislature). The original statute explicitly required state-funded, independent empirical studies (sometimes referred to as “re-authorization studies”) to present objective findings to lawmakers prior to re-authorization votes. The first such evaluation report was led by a research team at Duke University with a final report published in 2009 followed by a series of academic articles. (The evaluation primarily reports pre-post analyses found improvements in administrative outcomes such as hospitalization and medication adherence and increased use of/access to services). When Kendra’s Law was extended again in 2022, the New York State Legislature explicitly required a new, comprehensive independent study before the law came up for re-authorization again, leading to a competitive bidding process and award of a new (second) evaluation contract to a team co-led by the Human Services Research Institute (HSRI) and University of Pittsburgh’s PathLab.
Study Context. The new evaluation spanned approximately 2.5 years with one year (pre-contract) spent conducting background reviews, site visits and consultation with stakeholders across NY. For example, the evaluation team met with every NYS field office, with the leadership of every Mental Hygiene Legal Services division, with county leadership in large urban areas including the Department of Health – Mental Health (DOHMH) in New York City, and with groups of service users with direct experience of AOT and family members. These extended consultations in turn laid the groundwork for the formal data collection (in depth audio-recorded interviews and focus groups) conducted in Year 2. Ultimately, in Year 2 members of the research team interviewed 46 service users, and over 150 psychiatrists, providers, administrators, family members, legal systems staff and state-level advocates.
Ultimately, the independent evaluation represented an enormous and complex undertaking, tasked with grappling with the tremendous variation in AOT implementation across NY counties, and emerging with numerous serious concerns regarding both implementation and AOT’s impacts. In the relatively short time since its release (access full report here), numerous statements and public reporting have mischaracterized actual findings and, in this post, we seek to set the record straight.
The mischaracterizations: Many public statements and reporting, including statements made by the New York Office of Mental Health or its leadership (e.g. here and here), have, in our reading, mischaracterized, misrepresented or overtly “spun” the evaluation study, its findings and their complexity. For example, all OMH statements to date have led with the putative benefits of AOT, while minimizing (or wholly failing to acknowledge) documented harms, due process failures and the fact that the state’s failure to measure and monitor the implementation- and outcome-related harms that were documented is precisely what prompted the team to adopt a mixed methods approach, and Pitt’s research team specifically to heavily invest in qualitative data collection and analysis, most critically concerning the experiences of individuals (/service users) directly impacted by AOT.
Summary of the (Actual) Major Findings — Quantitative. The evaluation (as publicly reported) compared people under court orders (AOT orders involving ACT services) to a matched group receiving the same intensive services voluntarily (i.e. voluntary ACT). Both groups improved on the administrative outcomes used —the voluntary group did as well as the AOT group on most outcomes and better on arrests. Moreover, a more careful reading of the quantitative findings warrants a much weaker reading of the putative benefits of AOT relative to voluntary ACT. In fact, for example, homelessness fell far more sharply in the voluntary ACT group (29.2% to 17.6%, an 11.6-point drop, cutting homelessness by roughly 40%) than in the AOT group (10.9% to 8.5%, a 2.4-point drop). AOT is coded as “significantly better” on housing only in the evaluation’s adjusted, propensity-weighted rate of change comparison — a statistical (relative slope) artifact that stems from the AOT group starting with far better housing outcomes. In absolute terms the voluntary group achieved by far the larger reduction in homelessness, which is the opposite of what claims that AOT “improves housing stability” imply. The reason the adjusted comparison still favors AOT is that it models a relative rate of improvement after weighting and adjusting for age and treatment duration, not the real-world size of the drop: because the AOT cohort started from a much lower baseline rate of homelessness (11.2%), even a small 2.4 point decline represents a steep proportional slope, whereas the voluntary cohort’s far larger 11.6 point decline is measured against a much higher starting point (29.2%) and leaves a bigger residual — thus the test statistic used rewards the group that had less room for change (AOT), not the group that actually housed more people (voluntary ACT).
Intake characteristics merit the same scrutiny, troubling public claims that AOT is reserved for those who “would not accept services voluntarily”: at intake, medication non-adherence was higher in the voluntary ACT group not lower : only 40.8% of the voluntary cohort were taking medications exactly as prescribed, versus 51.3% of the AOT cohort. The voluntary group also started far more disadvantaged on the very outcomes AOT is credited with fixing — 30.3% homeless or unstably housed at intake versus 11.2% of the AOT group — yet reached comparable or better outcomes without any court order. Those absolute numbers should give every New Yorker pause: the cohort that would meet the obvious criteria for “harder to engage” — less adherent, far less stably housed — did as well or better under voluntary ACT. This finding alone is difficult to reconcile with the claim that coercion is not only necessary but also what makes engagement (and by extension service access) possible.
Major Findings — Qualitative. Turning next to the qualitative side of the evaluation, as briefly noted above, we conducted lengthy audio recorded interviews with 46 people with current or recent past direct AOT experience, as well as over 150 psychiatrists, providers, administrators, family members, legal system staff, and state-level advocates. What those directly impacted by AOT described to our team is not a program, at least in the majority of the state, that “supports individual autonomy and dignity,” as an Office of Mental Health representative recently put it. Instead, the majority of service user participants we interviewed — both those on and recently off AOT — described adhering to treatment out of fear of the consequences of what refusal might lead to rather than agreement with it; being placed under orders during hospital discharge and told they could go home only once they signed away a federal right to a hearing that they didn’t even understand as a right; surveillance experienced as invasive and generative of distrust; and medication effects ranging from serious metabolic and neurological harm to emotional flattening. Others reported psychiatrists who would not allow medication changes and AOT as a legal status that negatively impacted housing applications, employment, immigration, and custody proceedings long after the precipitating crisis, or even the period of AOT itself, had ended.
Many participants broke down crying or even sobbing during interviews when describing how it actually felt to be stripped of rights and not understand what was happening to them and why. Others reported that they had simply stopped raising concerns (or even hoping for them), having learned that it changed nothing. (Several participants told us simply that it would be “futile” to publicly raise or express concerns.). And still others asked some version of ‘why was it necessary to force me under AOT in order to access services I wanted all along?’ (Almost invariably when we asked participants if they would have voluntarily embraced the housing, case management, therapy and other psychosocial services they had access to under AOT, they said yes.)
And yet even among those who described deep gratitude to their case managers or ACT team members, we were alarmed by how few individuals were able to describe any access to high quality / high fidelity psychosocial rehabilitation, a reality sadly reflected in part in staggeringly high unemployment rates. (Viz. competitive, integrated employment rates increased a mere 4 percentage points from 4.1% → 8.2%). The overwhelming majority of participants were unable to describe any attempt on the part of their providers to promote social inclusion or community integration, and supervised residential services all too often functioned as “community” housing in name only.
As one parent, deeply saddened by the lack of attention to quality of life in her child’s case put it:
“I get that they don’t want to be made fools out of with the AOT, and they want to say, “Well, look, we kept her out of the hospital for two years now.” Yeah, but you’re medicating her so much that she can hardly move. She can’t tie her own shoes because she’s so big. It’s like you’re not giving her quality life at all.”
Pivoting to judicial process, particularly due process, half of those interviewed said their (Mental Hygiene Legal Services) attorney was present but did not advocate in any meaningful way, and another 30% had no positive or proactive attorney involvement at all. Legal representation did not lead to modification of any order in our qualitative sample, nor any successful challenge or successful objection to a renewal. High -level statewide advocates concurred that a frequent complaint from both service users and family is inadequate access to sufficiently resourced and responsive legal counsel. Per available statewide administrative data on this point, more than 95% of petitions and 98% of renewals are approved. The odds of fighting an AOT order and succeeding are staggeringly low. Indeed, even psychiatrists who strongly defended AOT in principle noted surprise and dismay at the average legal process, as the following quotes depict:
Psychiatrist 1: “The system, though, the court, although AOTs rarely involve the patient [actually] being in court, when a patient is in court, the entire system is rigged against them.... I’ll tell you how. Every person there, the judge is a state judge; the lawyers are state attorneys; the doctor is a state doctor. We’re, you know, we’re all power. We’re all dressed in a certain way. We all talk a certain way. We all behave a certain way. And we, you know, we’re not all powerful, but in the system of, in the entire system, we know all the procedures, we control all the shots. We can talk to each other. I can talk to the judge privately if I really wish to. And the patient, some of them, are, you know, it’s like a bull in a bull ring. You know, I’ve been to a bull fight, so I know how cruel it is. That they’re just totally, bewildered, like, what’s going on?
Psychiatrist 2: “So for me, like, that was actually disappointing to see that when an AOT is contested, it doesn’t feel like it is contested to me. So you show up there, [but] it goes the same way. No one’s really putting up a real fight, which is okay. I mean, because I believe in what I’m trying to do. However, I did question, like, in my mind, who’s fighting for his rights? You know, who helped him? And so you can look at certain points in the process to understand, well, the lawyer was here, the lawyer advised him, etcetera etcetera. But by the time you get to court, there’s really no way [to win], no chance.”
Replication is central to strong science and the truth is that these findings are not remarkable or different from the extant literature. The only other investigation of due process under Kendra’s Law to date, Candice Player’s 2015 due process study (based on her dissertation) documented findings as troubling as our own — with psychiatrists in the sample describing judges who simply deferred to the psychiatrist, service users who had no real chance of winning an appeal, and significant stigma in the ways that service users were talked about across the legal continuum.
Nor does our qualitative work contradict our quantitative work. When it comes to benefits, participants with direct experience repeatedly credited service access, including access to housing, and those providers who did treat them in caring, compassionate ways, as the source of these benefits, not a coercive mandate. Exactly the ‘elaboration’ and triangulation of the quantitative patterns that increases our confidence in the interpretation that service access is what is most critical, with very limited support for the benefits of coercion (particularly when weighed against harms).
This interpretation with respect to the central role of service access is also by no means discordant with the international literature; indeed many international field leaders have argued that that service access is in fact what drives most (if not all) benefits as evidence bed the repeated finding of seemingly robust pre-post benefits that attenuate substantially in quasi-experimental studies (i.e. that partially control for service access) and then disappear entirely in clinical trials in which service access (and sometimes accountability) is fully controlled for. We’ll quote from the most recent high-quality systematic review and meta-analysis of impacts of AOT (compulsory outpatient treatment) on aggression and criminal outcomes just to drive home the point:
“Results for all outcomes were non-significant, the effect size declining as study design improved from non-randomised data on self-reported criminal behaviour, through third party criminal justice records and finally to RCTs. Similarly, there was no significant finding in the subgroup analysis of serious criminal behaviour.” (Kisely, Bull & Gill, 2025)
In fact, the international evidence on AOT is anything but a testament to effectiveness, as Jorun Rugkasa (2016) notes in a summary review:
“Meta-analyses pooling patient data from RCTs and high quality nonrandomized studies also find no evidence of patient benefit, and systematic reviews come to the same conclusion.”
County Variation. It is also worth underscoring that implementation varies enormously across the state — as multiple providers and administrators put it, “if you’ve seen AOT in one county, you’ve seen AOT in one county” -- and this variation can teach us a lot about what is going right and wrong. For example, we noted particularly pronounced variation in the use of AOT diversion and in the application of least restrictive alternative standards: some administrators reported that “least restrictive” is barely mentioned, much less factored in, before they pursue or approve an order, while other counties — typically those with substantially lower utilization of AOT — did take these standards seriously. Similarly, use of enhanced voluntary agreements (pre-AOT diversion) is wildly uneven: New York City, which accounts for a large share of the state’s orders, never uses them to divert people before AOT, while some smaller counties make substantial use of them, engaging people voluntarily first and pursuing a court order only when everything else has genuinely failed. In such counties, administrators told the evaluation team they believe they could keep many or most AOT eligible individuals out of court entirely through voluntary diversion — again contradicting official characterizations of AOT as a “last resort” (ironically claims that have absolutely no evidentiary basis behind them). Note that the only published research to date that empirically addresses the question of pre-AOT diversion – a Los Angeles County study focused on diversion policy stemming from the County’s interpretation of least restrictive alternatives standards – found that, given a 6 month voluntary diversion support intervention – only a minority of individuals initially eligible ended up receiving an AOT order (Starks et al., 2020).
The Politics of Qualitative Findings. Unfortunately, qualitative findings that speak to serious problems with implementation of a given intervention or harms that result are easily (and all too often) dismissed or downplayed on the basis of “anecdote,” smaller sample size, or “non-generalizability.” Here it is important to reiterate that no data on fairness and due process, or any of the harms the evaluation documents, are monitored or collected by OMH. There is no data that documents how many voluntary service agreements a given county has pursued, whether an AOT recipient was meaningfully involved in developing their plan, whether they met an attorney for an adequate amount of time, whether their questions were answered, or whether they would have accepted the same services voluntarily through an enhanced agreement. And when it comes to harms, there is no statewide monitoring of polypharmacy or dosing, or monitoring of whether requests to modify medications have been approved; no reporting on harms associated with law enforcement involvement, involuntary transport (generally in handcuffs), inpatient hospitalization, or medication over objection; no tracking of serious medication side effects occurring under AOT orders; no measurement of collateral losses to housing, employment, or custody, or of the erosion of agency that renders service users “compliant” on paper while in fact continuing to experience their life as a form of house arrest or de facto institutionalization in the community. Where there is no data — and qualitative findings are minimized or ignored — there can be no accountability.
Participants who spoke to us were told that state officials and legislators wanted to learn from them. At the end of the day, the least they are owed is a public response from state leaders, public accounting and news reports that do not minimize or invalidate findings that raise serious concerns about how AOT has been implemented in New York and that document real harm to those directly impacted. If our goal is to improve the notoriously awful outcomes that most people labelled with serious mental illness experience in the United States (and certainly also New York) our shared path surely involves clear-eyed acceptance of the many ways we continue to fail people and a commitment to undoing harms of our own making. That would be a step towards justice.
See also:
References
Kisely, S., Bull, C., & Gill, N. (2025). A systematic review and meta-analysis of the effect of community treatment orders on aggression or criminal behaviour in people with a mental illness. Epidemiology and psychiatric sciences, 34, e12.
Munetz, M. R., Ritter, C., Teller, J. L., & Bonfine, N. (2014). Mental health court and assisted outpatient treatment: Perceived coercion, procedural justice, and program impact. Psychiatric Services, 65(3), 352-358.
Player, C. T. L. (2015). Outpatient commitment and procedural due process. International Journal of Law and Psychiatry, 38, 100-113.
Rugkåsa, J. (2016). Effectiveness of community treatment orders: the international evidence. The Canadian Journal of Psychiatry, 61(1), 15-24.
Starks, S. L., Kelly, E. L., Castillo, E. G., Meldrum, M. L., Bourgois, P., & Braslow, J. T. (2022). Client outreach in Los Angeles County’s Assisted Outpatient Treatment program: strategies and barriers to engagement. Research on social work practice, 32(7), 839-854.
Recommended Additional Readings
Barnett, P., Matthews, H., Lloyd-Evans, B., Mackay, E., Pilling, S., & Johnson, S. (2018). Compulsory community treatment to reduce readmission to hospital and increase engagement with community care in people with mental illness: a systematic review and meta-analysis. The Lancet Psychiatry, 5(12), 1013-1022.
Cossu, G., Kalcev, G., Sancassiani, F., Primavera, D., Gyppaz, D., Zreik, T., & Carta, M. G. (2024). The long‐term adherence following the end of community treatment order: A systematic review. Acta Psychiatrica Scandinavica, 150(2), 78-90.
Johnston, E. L., & Klein, A. (2024). Assisted Outpatient Treatment: A State-by-State Comparative Review. Clevland Student Law Review, 73, 723.
Johnston, E. L. (2025). Coercive Compassion: Theorizing Assisted Outpatient Treatment. University of Florida Levin College of Law Research Paper Forthcoming.
Kisely, S. R., & Campbell, L. A. (2015). Compulsory community and involuntary outpatient treatment for people with severe mental disorders. Schizophrenia Bulletin, 41(3), 542-543.
Maylea, C., Zirnsak, T. M., Edan, V., Armitage, P., Robert, H., & Brophy, L. (2026). Ensuring compulsory treatment is used as a last resort: a narrative review of the knowledge about Community Treatment Orders. Psychiatry, Psychology and Law, 33(3), 583-602.
Thanks for the overview, Nev.
My assumption is that there would be pretty major differences in the types of patients comprising the AOT and voluntary groups, but in the opposite direction from what the report indicates! Much demographic data suggests that the AOT individuals at baseline are probably more functional (and perhaps less ill?) than the voluntary cohort. Any thoughts as to why this is? I could imagine, for example, better resourced families pushing harder for AOT, or clinicians/judges looking at a younger, healthier individual and thinking that there is more reason to try and prevent further deterioration by being more aggressive.
While I find the functional outcomes dismal, I am somewhat impressed by the rather large reductions in risk of harm to self and others. Reducing the rate of physical violence in the AOT group from 30% to 9% is pretty significant.
My takeaway from the study is that AOT and voluntary treatment are practically the same in most major outcomes, which brings me to my next question. Are all individuals in the AOT group given the ability to opt-in to voluntary treatment? If no, then I think that obviously needs to change given the lack of clear practical differences between outcomes. If yes, then I think the question is whether or not we find the outcomes to be worth forcing people into AOT who would not otherwise voluntarily participate. I am skeptical of that for many reasons, but would also like to see if there are particular sub-populations that seem to respond particularly well to AOT.
Thank you for sharing further insight on this study. I work in homelessness policy and so was interested to see that the report looked at housing outcomes. In your discussion of the quantitative results here, I wasn’t sure what you meant about the “proportional slope” of the decline in homelessness for AOT clients. Since the decline for the non-AOT group is larger in both absolute and relative % terms, wouldn’t a comparison of slopes or an unadjusted regression estimate favor them? In any case, the way the inverse weighted estimates are presented leaves some important questions including: What was the magnitude of the statistically significant differences in outcomes? How do the estimates in marginal AOT and non-AOT change over time compare to the magnitude of estimates for other covariates? How did the AOT/non-AOT balance on characteristics look before and after inverse weighting? Further technical reporting on such items would allow outside researchers to better weigh the reliability and practical implications of the findings.