Mental Health Workers Say Algorithmic Triage Is Hurting Patients

hhn_acker about 3 hours ago 18 commentsRead Article on capitalandmain.com

⚡ Community Insights

Discussion Sentiment

62% Positive

Analyzed from 1468 words in the discussion.

Discussion (18 Comments)Read Original on HackerNews

exmadscientist•about 2 hours ago

One of the most basic safeguards -- necessary, not sufficient -- for AI and even general technology in healthcare should be transparency. Keep the model a black box (if you really must), but every single input, output, and decision should be logged and available to the patient on demand, no questions asked.

That makes it a lot easier to deal with things like the bad AI transcriptions going around. Get burned by AI that can't hear what you're saying? It's in your file, what it heard and then what it screwed up. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

Want to know what the useless AI saw when it sent you to group therapy for acute mania? It's in your file. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

bewareofscams•about 2 hours ago

This was the premise of GDPR (which I wholeheartedly support), but Europe caved in to US dictatorship (like Europe always does, cause it has no agency).

curuinor•about 3 hours ago

LLM can't have legitimacy in doing things to human beings, because it's impossible for LLM to have political legitimacy. Political legitimacy in modern societies is already a vicious zero-sum game with a tiny elite having any in the first place, none will be given to objects. Mental health is inextricably linked with violence and political legitimacy therefore, so the triage lacking efficacy or not doesn't even matter - it doesn't have legitimation. The violence inherent in triage is not justified.

kelseyfrog•about 2 hours ago

It's impossible for corporations to have political legitimacy. Arguably, AI illegitimacy is a subset of that broader umbrella of illegitimacy.

scld•about 3 hours ago

  This change has coincided with a sharp increase in the number of patients who are upset by the time they speak to her. On a typical day, as many as a third of her almost two dozen triage calls are with patients who have struggled to access appropriate care.

This seems like it's working, actually. Similar to automated Amazon warehouses experiencing a seemingly paradoxical increase in injury rates compared to non-automated ones.....it's very possible this is simply selection bias and the patient triage overall is improved greatly.

LordDragonfang•about 3 hours ago

Not sure that assertion holds; looking into the Amazon thing, it looks like the mechanism there was that it automated the "safe" parts of the job (notably walking), and greatly increased the rates at which workers were expected to perform the parts likely to result in injury/RSI, with less downtime between. That's not a flattering comparison to be making for any health system.

ofjcihen•about 3 hours ago

This is ignoring that patients were being routed correctly before. This isn’t someone at a help desk. This is a mental health clinician being given the same patients after a worse intake process.

watwut•about 2 hours ago

Being upset over obtuse process by the time you speek to a doctor is not a sign of good triage. Good triage does not implies angry patients.

Also, amazon injury rates were not paradoxical. If you dont care about people when designing process, it is unsurprising when you end up hurting them.

harvey9•about 2 hours ago

I mostly agree, except in some circumstances good triage can imply angry patients (assuming demand will exceed supply sometimes) : Lots of people present inappropriately at ER and then complain that they have to wait a long time.

piltdownman•about 3 hours ago

It's an amazing and disingenuous posit that the problem here lies with the Algorithm as opposed to the consequences under which it was introduced - cost-cutting. The barriers to care for patients who are already struggling with serious mental conditions are not a new phenomenon related to automated tooling derived from best practice corpus. Rather, they are a direct result of a decline in triage staffing.

Take cardiologist Dr. Lee Goldman’s chest pain decision-making algorithm, originally devised at Cook County Hospital to diagnose heart attacks.

Effectively a poster in the ER illustrating a simple decision tree, it outperformed traditional doctor diagnoses; achieving over 95% accuracy and proving 70% better at identifying patients not having a heart attack. This landmark clinical protocol represented a quantum-leap in outcomes, eventually refined to the Revised Cardiac Risk Index (RCRI) used as the gold-standard today.

So not only are algorithmic approaches old-hat in clinical settings, they are in fact a core component of contemporary best practice medicine. But that's somewhat besides the point when much of the issues raised in the article above refer to issues of executive dysfunction - areas in which low-context Agentic AI are basically designed to remedy, e.g.

// Instead of being referred directly by primary care doctors and contacted to schedule appointments, patients were routed through an app, nonclinical call center staff or simply given a phone number to call. “Now the onus is on the patients to be their own care coordinators,” she said... This dynamic can be particularly challenging for some patients. “Lack of motivation and lack of follow-through are the most common symptoms of depression on the planet,”

So the answer here is clearly an increase in appropriate AI resources, rather than throwing the baby out with the bathwater. If we trust a decision-tree poster to do cardiovascular triage better than the instincts of cardiovascular specialists, and OCR models to interpret results better than veteran Radiologists, why not Agentic AI to do the basic work of scheduling appointments?

NoDodgeQuestion•about 2 hours ago

RCRI is deterministic calculator, proved improve outcomes, not patient directly use

Is same true for AI triage?

piltdownman•about 2 hours ago

Well, yes. Not a huge breadth of research, but all conclusions support my statements above. Two such relevant and recent studies:

Evaluation of an Artificial Intelligence and Online Psychotherapy Initiative to Improve Access and Efficiency in an Ambulatory Psychiatric Setting https://pmc.ncbi.nlm.nih.gov/articles/PMC12316677/

//The overall wait time to receive care decreased by 71.43% due to this initiative. Additionally, participants received psychological care within three weeks after completing the triage module. In 71.29% of the cases, the artificial intelligence-assisted triage program and the psychiatrist suggested the same treatment intensity and psychotherapy program. Additionally, 63.29% of participants allocated to lower-intensity treatment plans by the AI-assisted triage program did not require psychiatric consultation later.

Potential of ChatGPT in youth mental health emergency triage: Comparative analysis with clinicians

https://pubmed.ncbi.nlm.nih.gov/40673126/

//This study suggests that GPT-4 models could be leveraged as a support tool in mental health telephone triage, particularly for psychiatric emergencies. Although response variability across iterations was minimal, most discrepancies in admission decisions were identified as false positives, reflecting that GPT Models may have a tendency to over-triage relative to clinician judgment. While findings are promising, further research is required to confirm clinical relevance.

A few others that may be of interest. The results from Raita et al.’s study that used a large dataset of adult ED visits revealed that four ML models outperformed the Emergency Severity Index (ESI) in forecasting outcomes of critical care and hospitalization, with higher discriminatory abilities and reduced under-triaged patients in levels three to five of ESI triage

The rest of the studies below reinforc the superiority of developed ML models over conventional triage systems - e.g. by demonstrating an AUROC of 0.991 for predicting critical outcomes in pediatric ED visitors, or another study showing LLMs surpassing the performance of the ESI and vital sign triggers

- Yilanli M, McKay I, Jackson DI, Sezgin E Large Language Models for Individualized Psychoeducational Tools for Psychosis: a cross‐sectional study. 2024.07.26.24311075. Preprint at medRxiv. 2024.

-Hwang S, Lee B: Machine learning-based prediction of critical illness in children visiting the emergency department. PLoS One. 2022, 17:e0264184. 10.1371/journal.pone.0264184

-Joseph JW, Leventhal EL, Grossestreuer AV, et al.: Deep-learning approaches to identify critically Ill patients at emergency department triage using limited information. J Am Coll Emerg Physicians Open. 2020,

-Liu Y, Gao J, Liu J, et al.: Development and validation of a practical machine-learning triage algorithm for the detection of patients in need of critical care in the emergency department. Sci Rep. 2021, 11:24044.

-Raita Y, Goto T, Faridi MK, Brown DF, Camargo CA Jr, Hasegawa K: Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019,

-Wolff P, Rios SA, Grana M: Setting up standards: a methodological proposal for pediatric triage machine learning model construction based on clinical outcomes. Expert Syst Appl. 2019, 138:12.

-Ivanov O, Wolf L, Brecher D, et al.: Improving Ed emergency Severity Index acuity assignment using machine learning and clinical natural language processing. J Emerg Nurs. 2021, 47:265-278.e7.

harvey9•about 2 hours ago

This is still ER - clinicians using AI tools. The MH triage flow is patient interacts with an AI by typing or speaking. It could work - maybe the systems in the article are just not very well set-up.

LordDragonfang•about 2 hours ago

> This dynamic can be particularly challenging for some patients. “Lack of motivation and lack of follow-through are the most common symptoms of depression on the planet,” Marcucci-Morris said. “We wouldn’t tell a paraplegic, ‘Hey, walk down the hall in order to get your wheelchair.’”

This was something I really struggled with getting an ADHD diagnosis as adult. I needed an incredible amount of executive function (dozens of phone calls, navigating the mire of health insurance to figure out who I was allowed to go to, who would take me, being bounced around) to be allowed medication to treat my executive dysfunction. I recall wondering how someone less functional than I was was ever expected to get help.

MisterTea•about 1 hour ago

Yup. With ADHD you constantly feel behind the 8-ball, forever in a deficit of agency. Worse when it's paired with anxiety so you never feel you are in control or make the right decision. Most of your time is spent worrying about what you didn't do and what you might have missed. Some things have higher priority to worry about so "small" things like calling that ADHD doctor fall by the wayside. Inevitably the chaos catches up where you forget an important event, forget to pay a bill, etc, and you're in panic emergency mode trying to fix it. So of course you forget to call that ADHD specialist. All day every day is like this.

kg•about 2 hours ago

One problem with Kaiser (probably not only Kaiser) when it comes to specialists like mental health workers is that at least in California there was a policy of not allowing specialists to refuse new patients. So over time, specialists' backlogs get bigger and bigger and they are juggling more and more patients and it becomes harder for the specialist to keep track of what's going on. Meanwhile the wait for a new appointment becomes longer as a result - typically months. As a Kaiser patient I personally had multiple specialists quit Kaiser to work elsewhere and at least one cited this as the reason when they told me they were leaving.

It seems really nasty to pair this with reductions in triage staff and the use of algorithmic triage. Their people are already overworked.

UpsideDownRide•about 3 hours ago

What the everliving..... There is no world in which current LLMs are even non harmful for people with actual mental issues.

People making such decisions should be barred from any kind of social adjacent decision-making.

Bluestein•about 3 hours ago

(Heck, if I may: they have, at best, not been disproven to be non-harmful for people without mental issues, while - it seems, more and more - proving quite harmful to those "borderline" or with underlying, unsurfaced issues.-)