Error by AI scribe during medical appointment leaves patient devastated

The cargo cult around AI on this website is making me second guess a career choice that has always been obvious to me.

In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.

That's besides point, the point is that classical software bugs are contained and we have process and understanding allowing us to discover them, triqge and fix. For AI based sw you can guess and try another prompt.

The software is heavily regulated for medical devices. Saying an MRI machine has bad software seems highly unlikely to me. This is of course different from Epic, but even then as a patient MyChart is really not that bad

I recommend reading table 3 in this paper:

https://pmc.ncbi.nlm.nih.gov/articles/PMC7284300/

"It was stated in notes that I had lung cancer. I do not and never have had lung cancer."

"I did receive a referral for physical therapy, it was for the wrong body part"

"Doctor reported that I did not claim to have pain in my hand. I am a pianist and I went specifically because pain was in my hand."

"I have been complaining of difficulty breathing [for over 3 mo]... notes saying my breathing is normal"

And plenty more

To this day we have no idea why or how, but years ago someone at a chain of walk-in clinics once put a note in my wife's file indicating that she was a habitual cocaine user. She had no idea until she went in one time and they were like "So, are you still using cocaine?"

(I do think this is extraordinarily rare compared to how often AI hallucinations happen, but it's such a bizarre story that I have to mention it.)

Medical transcription isn't the same thing as medical scribing. Transcription has been done primarily by speech recognition software with human review for decades. The output is just a text chart note, which isn't particularly useful because none of the clinical findings are coded. The coding can either be done up front by a human or AI scribe, or as a second step after transcription. In any case the human attending clinician is legally accountable for reviewing and approving the output before it's officially added to the patient chart.

Cost of medical care is already sky high (in the US). Are you sure you want to introduce even more administrative overhead?

Think about what this literally means. There are a million doctors in the US. Doctors spend their day consulting patients, so, you need a million transcription reviewers. Let's say you pay each of them $50k/yr (aka rock bottom US entry-level wage).

That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.

It used to be common in some places to outsource medical transcription to India (obviously not real-time recording). I spent time in India years ago working with a company in Mumbai (not medical transcription) and there were stories in Indian newspapers about mistakes being made due to the transcribers having no medical knowledge at all. One instance that I've never forgotten is the one where the Doctor had diagnosed "phlebitis" in the patient's leg, and the transcriber had written "a flea bite his leg".

You seem to have misread my message and kinda forced it into the Pro AI/Anti AI culture war shape.

Or at least my LLM thinks that you did.

It seems like this entire category of program is designed to make the same work take more time and more human attention.

I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.

The article doesn't meet a bug report standard. How would you feel about a bug report that doesn't mention which software were used, which versions were used, but it extensively describes all the harms that were caused by the lost messages.

I treat it as it is: a trendy hit piece against AI. There's nothing like "Why they still use Whisper Large V2?" in sight. If this article pushes the establishment to be more transparent about the AI tools, good. But right now there's not much to discuss.

It's not about giving AI a free pass - obviously we should try to remove these errors. It's about whether one reasonable option for removing these errors is to stop using AI or not. If AI increases error rates that would be a reasonable option. If AI doesn't increase the error rates, or even reduces them, it might not be.

And hence comparing AI use to a fictional situation where no errors happens is not meaningful.

I've personally caught multiple errors that were not just transcription errors, but elementary reasoning errors done by specialists I've seen - the baseline error rate from healthcare providers is far above zero.

Yeah, but also, if we can make AI do the same work cheaper than doctors, we can probably reduce the wages of doctors and save a bunch of money, which will both increase profits for health companies and reduce costs for patients. At some point, sufficient savings is better for patients than more doctors.

> Everything should be reviewed for correctness.

Have you ever actually read your notes on file?

This is basically living in a fantasy world. Errors are routine.

One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again.

Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.

> even if you require and get > 99.9% accuracy [...] there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.

This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work.

There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines.

I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.

> I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.

This happens every single day and is effectively never reported. For far more nefarious reasons than a simple scribing error.

Drug seeking behavior enters notes all the time without much evidence and based entirely on a random doctor's (or even a triage nurse) hunch. A significant portion of those notes are outright false and incorrect. Once that is on your file and in a given medical system, you are marked for life.

Exactly.

When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review.

When AI makes mistakes, what happens? How is it held accountable?

You know what? I visited multiple PCPs and psychiatric providers over the course of several years, and during those years I was repeatedly being subjected to secondhand THC, because one of my neighbors was always smoking pot, and it was tangibly, perceptibly seeping into my bedroom. While I slept, while I worked, whatever, whenever I could not avoid it, I was inhaling it for hours on end, and I know that it affected me in subtle ways.

So I was always up-front with these providers. I figured they had a right to know. They asked if I was using drugs, and I would explain the situation. Obviously it is not voluntary use.

But, sure as shootin', "habitual marijuana use" showed up in my chart. That kind of shit is hard to deny and it'd be impossible to appeal or erase that. So I live with it. I have come around to the viewpoint that it is wise to conceal many material facts from your doctors. Conceal as much as possible. Lying is better than the alternative.