The 90% Error Rate They Shipped Anyway
Insurance companies are using AI to deny care at scale. A 90% error rate sounds like a broken system. It’s only broken if the errors cost something. For 99.8% of patients, they don’t.
Margaret is 73. She had a hip replacement in October. Her surgeon recommended ten days of inpatient rehabilitation. Her Medicare Advantage plan only approved two.
She didn’t know she could appeal. The denial came as a one-page letter with a phone number for member services. She called twice, held both times, and eventually stopped. She went home with a walker and instructions to follow up with outpatient physical therapy three times a week, if she could get there.
She couldn’t always get there.
I’m starting with Margaret because I want to be specific about who this affects before we talk about systems. The systems conversation is important. But Margaret is the reason it matters.
I want to say something upfront to the people in this audience who work in insurance or health systems, because I know some of you do.
The business case for AI in utilization management isn’t irrational. Claims volume is enormous. Human reviewers are inconsistent. An AI that processes thousands of claims quickly, at low cost, with consistent criteria, sounds like exactly the kind of infrastructure investment that makes sense.
The problem isn’t that the technology exists. The problem is what happens when the system’s error rate stops mattering, because the incentives run the other way.
That’s what I want to trace here.
Last year, a federal class action lawsuit against UnitedHealthcare alleged that the company used an AI model called nH Predict to evaluate post-acute care claims for Medicare Advantage patients. The plaintiffs alleged the model had a 90% error rate. That it denied care even when treating physicians had documented medical necessity. That UnitedHealthcare’s denial rate for post-hospital care doubled after the tool was deployed, from 10.9% to 22.7%.
UnitedHealthcare disputes all of it, and the case is still in court. I want to be clear about that. What I’m describing are allegations, not findings. I’m citing them because they’re in the public record, and the questions they raise are worth asking regardless of how the litigation resolves.
The number that’s not in dispute: 90% of UnitedHealthcare’s denials get overturned when a patient actually appeals to a federal judge.
Only 0.2% of patients ever do.
Here’s the math the system runs on: you don’t need the AI to be right. You just need the denial to be intimidating enough that most people don’t fight it. And most people don’t.
I manage technology infrastructure for a global academic community. Error rates are something I think about constantly. If a process in my environment was failing 90% of the time, we’d take it down. We’d fix it. Scaling it to millions of patients would not be on the table.
The only context in which you scale a 90% error rate is when the errors are profitable.
And I can’t get past this: when a denial saves money and only 0.2% of those denials are ever challenged, the incentive structure doesn’t reward accuracy. It rewards volume. The AI doesn’t have to be good. It has to be good enough that the economics work.
The conversation about AI in healthcare focuses on the wrong question. Everyone wants to know: is the model accurate? That’s not the question that matters. The question is: what happens when it’s wrong? Who catches it? What does the error cost? For whom?
For Margaret, the cost was going home without the care her surgeon recommended. For the insurer, the cost of that error was zero. She didn’t appeal.
UnitedHealthcare isn’t the only one. In 2023, a ProPublica and CBS News investigation found that Cigna physicians reviewed more than 300,000 claims in a two-month period using automated decision support tools. Cigna disputes the characterization and says physicians exercise independent medical judgment. The volume isn’t disputed. 300,000 claims in two months.
The American Medical Association found that 71% of insurers now use AI for utilization management. Not a pilot. Standard practice, at scale, with no federal framework governing accuracy requirements, bias audits, transparency, or appeals.
We require clinical trials before a drug reaches the market. Years of them. Proof of efficacy. Mandatory side effect disclosure. Post-market surveillance. A drug with a 90% error rate doesn’t get approved.
An AI system making life-or-death coverage decisions for Medicare patients has none of those requirements. We find out that a model has a 90% error rate because a class action lawsuit surfaces internal documents. That’s not a system working. That’s luck, and it only helps the people who can afford to sue.
The denial-rate problem is one layer. The bias problem sits underneath it, and they’re not separate issues.
Cedars-Sinai researchers found that AI clinical decision support tools recommend inferior psychiatric treatment options when a patient is identified as Black. The AI’s algorithm doesn’t invent the disparity. It’s inherited from training data that reflects decades of documented inequity in how care has been delivered and recorded.
Think of it this way. If a physician consistently recommended worse psychiatric options to Black patients out of habit rather than clinical evidence, we’d expect accountability. When an AI trained on that physician’s historical decisions does the same thing, accountability becomes nearly impossible to locate. The company points to the data. The data reflects historical practice. Historical practice reflects structural inequity. No single actor is responsible because everyone can point to the layer below them.
I don’t think that’s an acceptable answer. It is, for now, where the law leaves us.
For anyone in this audience who is deploying clinical AI: if you haven’t audited your training data for demographic disparities, you’re not just carrying an ethical risk. You’re carrying a liability risk that the courts haven’t finished mapping yet.
The hospitals are using AI. The physician groups. The appeals coordinators. The pharmacy benefit managers. At least four separate AI systems may touch a single claim between initial authorization and final decision. Each has its own training data, its own error rate, its own optimization target.
We don’t have standards for how those systems interact. We don’t require that a denial disclose that it was AI-generated. In some states, we don’t require a licensed physician to review an AI recommendation before it becomes a coverage decision communicated to a patient.
In my article, Show Me the Receipts, I wrote about observability as a human right: the ability to see what an AI system did, why it did it, and who is accountable when it’s wrong. Healthcare is where that absence costs the most. When an AI denies care and the patient doesn’t appeal, there’s no record that the denial was probably wrong. The case closes. The error disappears.
The 0.2% who appeal get overturned almost every time. The 99.8% who don’t are invisible in the data.
That’s worth sitting with. If appeals succeed at that rate, the AI was wrong far more often than anyone’s tracking.
What You Actually Have the Right to Do
When a claim is denied, you have the right to appeal. The internal appeal is step one, not the final word. If the internal appeal fails, you have the right to an external independent review conducted by an organization with no relationship to your insurer. Insurers lose those reviews at high rates. Most people never get there because no one tells them the internal appeal isn’t the ceiling.
Your denial letter is required by law to cite the specific reason for denial and the policy language used. You can also request the clinical criteria the insurer applied, typically from a system called InterQual or MCG. That document is what your appeal actually needs to respond to. Most people never ask for it.
Deadlines matter. Appeals for most commercial plans must be filed within 180 days of the denial date. Medicare Advantage has its own timeline with enforceable federal rules. The clock starts when you receive the denial letter, not when you find out you have the right to fight it.
If you’re on Medicare and your health is at risk, you have the right to an expedited appeal. The decision is supposed to come within 72 hours. Most Medicare patients have never heard of it.
The system isn’t designed to surface these rights. You have to know that you can ask.
So I Built Something
After I posted about insurance AI denial rates on LinkedIn, someone replied: “I wonder if it will level the playing field when someone creates an appeals AI algorithm and insurers experience a 100% appeals rate.” A few other people said similar things. I’d been thinking about it for a while and decided to go for it.
I built it with Claude. I’m building it in public, with feedback from the people who actually need it and my audience, many of whom work for insurers.
It’s called Overturned. Free. No login required. Your documents are never stored or used to train any model. You upload your denial letter, the tool reads it, identifies the specific grounds for denial, and generates an appeal letter that responds to those grounds directly. It shows you exactly what it found before you send anything.
Upload your denial. Review the findings. Refine the letter. Appeal.
About three minutes. About $0.07 in AI processing costs, which I cover through donations and my consulting practice. No paywalls. No ads. No data selling.
Insurance companies have been using AI to deny claims at scale for years. Overturned is AI writing back.
It’s a live beta, which means it’s not perfect. What it does well: it extracts the denial rationale, matches it to the cited clinical criteria, and drafts a letter in language that appeals reviewers recognize. What it doesn’t do: give legal advice. If your situation is complex or the stakes are high, talk to a patient advocate or a healthcare attorney.
There’s a public roadmap on the tool page where you can vote on features and suggest what we build next. If something doesn’t work the way you need it to, that feedback is how it gets better.
The appeal you don’t file is the one you can’t win.
What I Don’t Have Answers To
I’m not arguing AI has no role in utilization management. The claims volume is real and human reviewers make errors too. What I’m arguing is that we need a standard of evidence before deployment: a required accuracy floor, mandatory demographic bias audits, and clear rules about who reviews AI decisions before they become final. None of that currently exists at the federal level.
I also don’t know how to solve the 0.2% problem without making the appeals process dramatically simpler. Overturned helps with the letter. Someone still has to know they can fight, decide it’s worth it, and follow through. That’s not a technology problem. It’s a structural one.
And I genuinely don’t know what accountability looks like when the bias lives in the training data. Who audits the model? Who has standing to require changes? The Cedars-Sinai finding is one data point. How many other health systems have run that same analysis?
My guess: not many.
If you’ve had a claim denied and fought it, or you’ve tried Overturned, I want to hear about it. The comment section is open. That’s how the tool gets better, and honestly, it’s how I understand what’s actually happening out there.
Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build s