Dr. Elisabeth Potter is roughly $5 million in debt.

Her husband cashed out his 401(k) to keep her surgery center running. UnitedHealthcare, the second-largest insurer in Texas, declined to add the center to its in-network list. Potter remains in the network as a surgeon. Her facility is not. She cannot operate there on UHC patients without leaving them with the full bill.

This is the same Dr. Potter who scrubbed out of a breast cancer surgery in January 2025 to take a phone call from UnitedHealthcare. The surgery had been pre-approved. The caller wanted to know if the patient asleep on the table needed an overnight stay. UHC denied the overnight anyway. According to Potter, her patient went home eight hours after a bilateral DIEP flap reconstruction. UnitedHealthcare disputes her account, attributing the call to a clerical error.

I wrote about that call in Vol. 12. What I want to write about now is what came after.

UHC sent Potter a defamation threat letter from Clare Locke, the firm that served as lead counsel in Dominion’s $787.5 million settlement against Fox News, demanding she take her TikTok down.

She didn’t. UHC then declined the network contract for her surgery center.

Same insurance company. Four different decisions. Four different parts of its operation. One surgeon. Same outcome.

Authorize. Deny. Threaten. Block.

That is not a process. That is an architecture.

A single Medicare Advantage claim can pass through four AI systems before a human ever opens the file.

The hospital codes the encounter. The payer evaluates the claim. The pharmacy benefit manager authorizes the prescription. And as of January 1, 2026, the Centers for Medicare & Medicaid Services (CMS) reviews it.

Each layer is built by a different vendor. Each is trained on different data. Each optimizes for a different metric. Each defines “human review” differently.

None of them coordinates with the others.

None of them is accountable for the patient.

I manage technology infrastructure for a global academic community. The first question I ask of any new vendor is what other systems their tool touches and where accountability lives when something goes wrong.

In healthcare claims right now, the honest answer is that nobody, not the insurer, not the hospital, not the patient, not the IT Director, has mapped the cascade end-to-end.

Let me try.

Layer One: The hospital’s AI codes the visit

It is not predicting clinical reality. It is predicting which diagnoses pay.

Blue Cross Blue Shield published an analysis in March 2026. One facility’s billing complexity rating jumped 6.7 percent after announcing it would adopt AI for medical coding. Other facilities in the same state moved 0.9 percent over the comparable period. Across hospitals identified as likely AI adopters, complex-coded admissions rose by an average of 13.1 percentage points.

Blue Cross attributes $663 million in additional inpatient spending to AI-driven coding over a three-year period.

The most damning specific finding: AI coding tools were classifying new mothers as having severe acute posthemorrhagic anemia in cases where no transfusion ever occurred. That single diagnostic pattern added $22 million to maternity admission costs in one year.

The model is not lying. The model is doing what it was built to do, which is to find the highest-paying compliant code for the documented encounter.

Whether the patient actually had the condition is a separate question and not one the model is asked to answer.

Layer Two: The payer’s AI evaluates the claim

It is not predicting medical necessity. It is predicting what a cost-pressured reviewer would have denied.

This is the Vol. 11 argument extended. Cigna’s PXDX, UnitedHealthcare’s nH Predict, EviCore’s prior auth tools. All trained on historical claims data.

Trained on historical claims data means trained on prior decisions. Which means trained on the outputs of human reviewers operating under the same cost-control incentives, the algorithm is now automating.

The model is not learning what good care looks like. It is learning what previous reviewers under quota pressure decided to deny.

In March 2026, a federal court ordered UnitedHealth to produce documents across six of seven discovery categories in the nH Predict class action. The discovery includes performance evaluations and compensation records for medical directors. The identities of the company’s internal AI review board. Documents back to January 2017, pre-dating the deployment of nH Predict.

The court rejected UnitedHealth’s argument that pre-deployment records were irrelevant. UnitedHealth disputes the plaintiffs’ characterization of the model.

What is not in dispute is that hospitals know exactly what is happening on the other side of the cascade.

Andrew Asher, Centene’s chief financial officer, said the quiet part out loud at the Deutsche Bank Healthcare Summit in September 2025. Hospitals had gotten better organized around AI for coding than payers had. “We’re going to catch up to that,” he said.

That is the cascade in one sentence, from a CFO. A documented arms race, named on both sides, with the patient sitting in the middle of a transaction nobody designed.

Layer Three: The pharmacy benefit manager’s AI authorizes the prescription

The voluntary commitment to reform prior auth does not cover this layer.

Optum Rx, UnitedHealth’s pharmacy benefit manager (PBM), runs a tool called PreCheck. According to UnitedHealth, it cut prescription approval time from over eight hours to a median of 29 seconds.

UnitedHealth projects nearly $1 billion in AI savings in 2026.

In June 2025, America’s Health Insurance Plans (AHIP) and the Blue Cross Blue Shield Association announced a voluntary commitment to streamline prior authorization. Fifty insurers covering roughly 257 million Americans signed on. By April 2026, the industry reported eliminating 11 percent of prior authorization requirements, or about 6.5 million fewer requests per year.

That is real. I want to acknowledge it.

I also want to be honest about what the pledge does not do.

The pledge has no enforcement. It does not specify what fraction of remaining authorizations are AI-decided. It does not require disclosure of which model decided, or what its overturn rate is.

And it does not cover pharmacy benefits, which is exactly where the PBM layer of the cascade sits.

CMS required payers to publish aggregated 2025 prior authorization metrics by the end of March 2026. KFF analyzed the data when it landed and found that it did not actually explain what drove approvals or denials.

Disclosure that does not let you trace a decision is not informing you. It is performing.

Layer Four: The regulator joined the cascade

On January 1, 2026, traditional Medicare became an experiment too.

CMS launched the Wasteful and Inappropriate Service Reduction model, called WISeR, in six states: Arizona, New Jersey, Ohio, Oklahoma, Texas, and Washington.

Six private technology vendors process prior authorization requests for fifteen Medicare Part B services: Cohere Health, Genzeon, Humata Health, Innovaccer, Virtix Health, and Zyter.

These vendors are not paid a flat fee.

They earn a percentage of the savings their denials generate.

Read that sentence again.

The vendor in Texas, Cohere Health, says its technology is “never used to deny care, but rather to automate approvals.” Healthcare Uncovered reported in April 2026 that 62 percent of WISeR prior authorizations in Texas were being approved on the first try.

Both of those things cannot be true at the same time.

The American Hospital Association asked CMS for a six-month delay before launch. The House Appropriations Committee approved an amendment to block WISeR funding. It did not survive final budget negotiations. Don Berwick, the former CMS administrator, called WISeR an import of “the bureaucratic, wasteful, and risky processes of permission-seeking” that have plagued Medicare Advantage for years.

CMS launched it on schedule.

For most of the last decade, the compliance posture for healthcare AI in the United States assumed traditional Medicare was the floor, and Medicare Advantage was the experiment.

WISeR inverts that. For services covered in those six states, traditional Medicare is now the experiment too.

When a denial reaches the patient, it cites one criterion and one policy clause.

It does not say which algorithm flagged the claim. It does not say which training data the algorithm used. It does not say whether a human ever opened the file. It does not say which vendor licensed the model to the insurer. It does not say what the appeal-overturn rate is for this category of denial.

The patient is supposed to appeal. The 0.2 percent who do appeal succeed at very high rates.

The other 99.8 percent never see the layer they were fighting.

This is not a transparency problem at the level of any single AI system. It is a transparency problem at the level of the architecture.

I want to be specific about something, because the shape of this argument matters.

If you work inside a payer, a hospital revenue cycle, or a PBM, you are not the villain of this piece. The incentives you operate inside are the villain.

Claims volume is real. Coding ambiguity is real. Fraud is real. An insurer that never reviews claims gets exploited. A hospital that does not optimize coding leaves money on the table that its competitors are taking.

I know that. What I want to trace is the gap between any single legitimate function and the industrial pattern they have become when stacked on top of one another.

The hospital’s coder is doing her job. The payer’s reviewer is doing his. The PBM’s algorithm is doing what it was built to do. The CMS pilot vendor is fulfilling a contract.

Every layer is operating within its own logic, against its own metric, defended by its own vendor’s compliance documentation. The aggregate effect is a denial cascade that no individual layer is responsible for.

In Vol. 11, I argued that an AI with a 90 percent error rate scales only when the errors are profitable.

The cascade extends that.

An AI cascade scales only when no single layer has to answer for the whole.

WISeR just adopted that architecture as federal policy.

What your AI vendor cannot answer

If you are procuring AI for a payer, a hospital, or a PBM in 2026, three questions will determine whether you have a vendor problem or a documentation problem that becomes a liability problem.

What is the documented overturn rate of this model on appeal, by category of decision?

What disclosure language do we provide to patients when this model contributed to a denial?

When the discovery order arrives, and discovery orders are arriving, what records can we produce about how this model was used, by whom, and against which performance metrics?

A legal analysis of the UnitedHealth discovery order from Stephenson Acquisto & Colman put it bluntly. Provider organizations that integrate AI into clinical workflows, prior authorization support, utilization management, or care coordination face meaningful legal risk if that AI functions as a decision-maker rather than a decision-support tool.

That distinction is doing a great deal of work in 2026 contracts. Most of the contracts I have read do not draw it cleanly.

If your vendor cannot answer those three questions, you do not have a vendor problem.

You have a documentation problem.

And the discovery order showing up in your industry just promoted that documentation problem to a liability problem.

What I do not know

I do not have a clean position on whether voluntary commitments like the AHIP pledge can produce the disclosure the cascade requires. The reductions are real. The transparency is not.

I do not know who audits across vendors. The same handful of payment integrity firms (Cotiviti, Optum, Zelis, MultiPlan, EquiClaim) operate across competing insurers. One model’s bias affects millions of patients across plans that appear to compete with each other but are running the same engine underneath. There is no public registry of which insurer uses which vendor for which decision type.

And I do not know how the Potter case ends. The retaliation arc, the network exclusion, the bankruptcy pressure, the defamation threats. None of it is settled. UnitedHealthcare disputes her account. The litigation is ongoing.

What I can say is that the pattern of single physicians being financially squeezed for documenting how the cascade affected their patients is not unique to her.

It is unusual only because she said it out loud.

If a bank approved your mortgage, let you close on the house, and three weeks later, a different department of the same bank sent you a letter saying it had re-reviewed the documents and actually you don’t qualify, and a third department flagged your account, and a fourth froze your line of credit, we would not call that a process.

We would call it fraud.

In healthcare, we call it utilization management.

And as of this year, four AI systems are running it in parallel.

If you have received a denial after a service was already authorized, or your prescription was denied at the pharmacy benefit manager, or your prior authorization was denied by a WISeR vendor in one of the six pilot states, the appeal rules are the same ones I described in Vol. 11. You have the right to appeal, the right to external review, and the right to see the specific clinical criteria the insurer applied.

Overturned will generate an appeal letter from your denial documents. It’s free, no login, no storage of your records.

The tool lives at rachelankerholz.com/tools/overturned. Cascade denials are explicitly in scope.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

Subscribe now

On January 7, 2025, Dr. Elisabeth Potter was in the middle of a bilateral DIEP flap breast reconstruction — a complex surgery for a cancer patient — when a call came into the operating room.

The caller was a UnitedHealthcare representative. Her patient, already asleep on the table, had been pre-approved for the surgery. What the rep wanted to know was whether the patient’s overnight inpatient stay was justified.

Potter scrubbed out to take the call. The UHC representative on the line did not have access to the patient’s medical records.

UnitedHealthcare denied the overnight stay anyway.

When Potter posted about it on TikTok, the video got 5.5 million views. She then received a defamation threat letter from Clare Locke, the same firm that won the $787 million Dominion judgment against Fox News, demanding she take it down and apologize.

She didn’t. UnitedHealthcare then declined to add her new surgery center to its in-network list, a decision she says has put her $5 million in debt and forced her husband to cash out his 401(k).

Scrubbing in is the point of no return. The patient has committed. The anesthesia team has committed. The decision the insurance company is making in that moment is not about whether the surgery happens. It is about who pays for how long the patient recovers.

Potter’s call is not an outlier. It is the loud version of a pattern that plays out quietly every day in post-op recovery rooms and billing offices across the country.

A patient gets home from the hospital. Eight days post-op. The mail arrives.

Her claim has been denied pending additional clinical review. The letter asks for medical records her surgeon’s office already sent during prior authorization.

She calls billing. The coordinator has heard this before. The coordinator has a folder of these letters going back years.

The only new document in the file is the procedure note. The same insurer that approved the surgery now wants to re-review whether the surgery it approved was medically necessary — using clinical information it already had, adding only the evidence that its approval was acted on.

Three weeks becomes three months. The patient’s credit takes a hit. The surgeon’s office writes off a percentage. Somebody hit a quarterly number.

The industry has a word for this. It almost never reaches the patient.

It is called retrospective denial. KFF Health News documented the pattern in a case involving the Markley family, who incurred medical debt after Anthem Blue Cross and Blue Shield revoked preapproval for a battery of tests performed at the Mayo Clinic. When insurers pull the decision to pay after the service is completed, patients are legally on the hook for the bill.

Martha Gaines, who directs the Center for Patient Partnerships at the University of Wisconsin Law School, co-authored a JAMA piece on this in 2020. “How broken can you get?” she asked. “How much more laid bare can it be that our health care insurance system is not about health, nor caring, but just for profit?”

That was 2020. It has not improved.

I want to be specific about something, because the shape of this argument matters.

If you work inside a health plan running utilization management, you are not the villain of this piece. The incentives you operate inside are the villain. Claims volume is real. Fraud is real. Documentation errors are real. An insurer that never reviews claims is an insurer that gets exploited.

I know that. I want to trace the gap between a legitimate review function and the industrial pattern it has become.

Retrospective review was originally a fraud control. It asked: Did the provider bill for a service they didn’t perform? Does the documentation match the procedure? Was the patient eligible on the date of service? These are good questions.

What retrospective review has become, at scale, is a second medical necessity decision on claims the plan has already approved. That is a different function. It runs on the assumption that the authorization was provisional and that the real decision happens after the money is at stake.

Two algorithms are running. The harm is in the gap between them.

The first algorithm approves. Cigna’s PXDX system processed approximately 300,000 denials in two months. Medical reviewers allegedly spent an average of 1.2 seconds per case.

At 1.2 seconds, there is no review happening. The algorithm makes the decision. The human clicks “approve” on what the algorithm has already decided, which is how the insurer meets the “human in the loop” requirement on paper while functionally automating the outcome. The human is not reviewing the algorithm. The human is ratifying it.

The second algorithm denies. Post-service claims get screened against different rules, often by different vendors. 46 percent of healthcare organizations already use AI for revenue cycle management. Another 49 percent plan to within a year.

Cotiviti partners with more than 100 health plans on payment accuracy, explicitly marketing retrospective review and AI-enabled clinical chart validation. Optum, the UnitedHealth subsidiary that acquired Change Healthcare, runs revenue cycle tools used across the industry. Zelis, MultiPlan, and EquiClaim work the same territory. One Cotiviti case study claims a Blue Plan achieved “triple its original projected findings” after adopting their AI-powered clinical review.

The AI making the denial is often not built by the insurer named on the letterhead. It is licensed from a vendor whose product, marketed in those exact terms, is more denials sustained on appeal.

Now, the training data question, which nobody is asking loudly enough.

The AI systems running authorization and retrospective denial are trained on historical claims data. Which means they are trained on prior decisions. Which means they are trained on the outputs of human reviewers who were themselves operating under the same cost-control incentives.

The model is not predicting medical necessity. The model is predicting what a cost-pressured reviewer would have denied.

That is a critical distinction. A model trained on historical denials learns to reproduce historical denials. If the training data contains systematic bias against expensive procedures, older patients, or complex cases, the model does too. The bias gets encoded, then laundered through a layer of algorithmic objectivity, then sold back to the same industry that generated it.

This is the pattern Cigna’s PXDX, UnitedHealthcare’s nH Predict, and EviCore’s prior-auth tools share. They are not performing a medical review. They are automating the inherited judgment of reviewers whose incentives were never aligned with the patient in the first place.

And now the detail that broke me.

According to reporting on UnitedHealth’s internal practices, the company explored using AI to predict which denials were likely to be appealed, and which of those appeals were likely to be overturned.

Read that sentence twice.

That is not an algorithm predicting medical necessity. That is an algorithm predicting who will fight back. And denying accordingly.

The logic is simple. If the model predicts an appeal is unlikely, deny. If the model predicts an appeal would likely succeed, deny anyway if the patient is unlikely to file one. A KFF analysis found that in 2021, only 0.2 percent of denied claims were ever appealed.

0.2 percent. The model does not need to be right. It needs to be right often enough to withstand the 0.2 percent of cases where someone fights.

nH Predict, the algorithm at the center of the UnitedHealth class action, has an alleged 90 percent reversal rate on appeal. Ninety percent. A coin flip would do better. The reason it gets deployed anyway is that nine out of ten denied patients never appeal.

A 90 percent error rate is only broken if the errors cost the company something. For 99.8 percent of patients, they don’t.

The speed asymmetry is the whole game.

Denial runs at algorithmic speed. Appeal runs at human speed. The insurer’s system flags a claim in milliseconds. The appeal takes the patient weeks of phone calls, records requests, and letters. The provider’s billing team appeals in aggregate because they don’t have the labor to fight every denial individually.

Between an algorithm that denies in milliseconds and a human who appeals in months, the house always wins. Not because the algorithm is right. Because the patient gave up, or died, or paid.

I manage technology infrastructure for a global academic community. When I think about what would happen if my systems operated the way insurance claims review operates, I do not have to guess. It would be a disaster.

Imagine a system that approved a user’s access to a resource, let them use it, and three weeks later revoked the approval retroactively and charged them for the time they had already spent. In my world, that is not a utilization management program. That is a breach of contract, an incident report, and a board conversation.

When I look at the retrospective denial pattern, I see the same infrastructure failure. A commitment was made. The commitment was relied on. The commitment was revoked after reliance had created harm.

A 2024 Premier survey found an average of 3.2 percent of all claims denied had been pre-approved via prior authorization. More than 54 percent of denied claims were ultimately paid after appeal. Denials skewed to high-cost claims, averaging charges of $14,000 and up.

Three details from that survey matter. First, 54 percent of denied claims eventually get paid, which tells you the denial was wrong. Second, the denials concentrate on expensive claims, which tells you the optimization target. Third, the denials concentrate on procedures that were already pre-approved, which tells you what authorization is actually worth.

A Senate Permanent Subcommittee on Investigations report criticized UnitedHealthcare, Humana, and CVS for using AI automation to deny Medicare Advantage post-acute care. The report raised concerns about substituting medical necessity with financial calculations.

That investigation focused on front-end denials. The back-end version — retrospective denial, running through licensed vendor AI — has not received the same scrutiny.

If a bank approved your mortgage, let you close on the house, and three weeks later sent a letter saying they had re-reviewed your application and actually you don’t qualify — we would not call that a process. We would call it fraud.

In healthcare, we call it utilization management. And we let AI run it.

What I Don’t Have Answers To

I don’t know how to distinguish, at policy scale, legitimate retrospective review from the pattern I am describing. Fraud happens. Upcoding happens. Documentation errors happen.

I also don’t know who audits the second algorithm. Cotiviti, Optum, Zelis, MultiPlan, EquiClaim — these vendors operate across dozens of insurers, which means one model’s bias affects millions of patients across plans that appear to compete with each other but are all running the same denial engine underneath.

When a denial letter cites “additional clinical review,” I cannot tell you which vendor’s model flagged the claim or what criteria it applied. Neither can the patient. Neither, in many cases, can the appeals coordinator at the insurance company.

And I don’t have a clean answer for Dr. Potter. She did the right thing. She scrubbed out to take the call because she knew if she didn’t, the stay would be denied. She is now $5 million in debt and facing defamation threats for saying so publicly. I don’t know how to build an incentive structure that doesn’t punish that decision.

If you have a denial letter for a service that was previously authorized, the appeal rules are the same ones I described in Vol 11. You have the right to appeal, the right to external review, and the right to see the specific clinical criteria the insurer applied.

Overturned will generate an appeal letter from your denial documents. It’s free, no login, no storage of your records.

The tool lives at rachelankerholz.com/tools/overturned. Retrospective denials are explicitly in scope.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

Your health system probably signed an ambient AI scribe contract in the last 18 months. Maybe it was Abridge. Maybe Nuance DAX Copilot. Maybe Ambience Healthcare. The deal came through clinical operations, or IT, or both. Your legal team may or may not have seen the final BAA. Your patients almost certainly saw nothing at all.

That gap is now a class action lawsuit.

In July 2025, Jose Saucedo went to Sharp Rees-Stealy Medical Group for a routine physical. He spoke with his doctor. He left. A few weeks later, he logged into his patient portal to review his visit notes. His medical record stated that he had been “advised” that the visit was being audio recorded. It said he had “consented.” Neither thing had happened. The recording had been made, transmitted to a third-party vendor’s cloud, and processed by an AI tool called Abridge. The consent documentation in his chart was, according to the lawsuit he filed in November 2025, false.

The proposed class action covers anyone who had a medical visit with Sharp on or after April 1, 2025, the date Sharp announced its Abridge partnership. That is potentially over 100,000 patients.

The AI didn’t just record him without his knowledge. It wrote a consent record proving he had agreed to something he never agreed to. That is not a failure at the edges of how this technology works. That is the system performing exactly as designed, with no one watching the output.

Before we get to the law, here’s the operational problem your board needs to understand.

Once the recording is made, your health system is in one of two bad positions, and your vendor contract put you there.

Delete the audio. Many vendors do this within 30 days. Privacy problem addressed, at least technically. But now there is no ground truth. If the AI hallucinated a dosage, a diagnosis, or a symptom the patient never mentioned, there is no recording left to check it against. The legal exposure here is concrete: a physician tells the patient they’re prescribing 0.5mg. The AI transcribes 5mg. The physician doesn’t catch it before the prescription is filled. Months later, a malpractice claim arrives. The audio is gone. The AI-generated note is the only record. The note is what the opposing attorney has.

Retain the audio. You get verification capability and an evidentiary trail. But now every raw recording, every draft transcript, every backend AI artifact is potentially discoverable. Defense attorneys are already warning that clinicians may find themselves defending not just the signed chart, but a parallel archive they did not author, edit, or control.

Delete it, no receipts. Keep it, the receipts may be used against you.

Neither option is clean. That is the architecture your vendor sold you, and your contract formalized it.

Here’s what I keep coming back to: the C-suite conversation about ambient AI in healthcare is almost entirely about efficiency, and almost entirely missing the question that will matter most when the next lawsuit lands.

The efficiency case is real. The Permanente Medical Group, Kaiser’s physician organization in Northern California, reported that ambient scribes saved its physicians the equivalent of 1,794 working days across more than 2.5 million patient encounters. Clinician burnout is a genuine operational crisis and the documentation burden is a meaningful driver of it.

What they’re less clear on is who owns the liability when the tool goes wrong.

The Larridin 2026 State of Enterprise AI Report, which surveyed more than 350 senior leaders at organizations with 1,000 or more employees, found that 92% of C-suite executives expressed full confidence in AI impact, while 58% said they couldn’t identify who in their organization owned AI performance accountability, and 62% lacked a comprehensive inventory of AI applications currently in use. The confidence in AI and the visibility into ownership and accountability are not moving together.

In the same month that the survey was published, Harvard Business Review ran a piece about a Fortune 500 insurance company whose CEO convened the C-suite to ask a single question: Who owns our AI initiatives? The CIO said it was obviously her domain. The COO said an agentic workforce is operations by definition. The CFO pointed out that an AI system was already making underwriting decisions with direct P&L impact. The Chief Risk Officer noted that autonomous decision-making is a major risk exposure. No one had a clean answer.

Lauren’s organization had a version of that meeting. The question of who owns the ambient scribe decision, and who owns the liability when that decision goes wrong, is not settled in most health systems. In many, it is not even fully articulated.

I manage technology infrastructure for a global academic community. When I evaluate a vendor relationship, my first questions are always about data: where does it go, who can see it, how long does it stay, and what does the contract say when I want to leave. These are basic infrastructure questions. They are also, as I’ve been arguing across this series, accountability questions.

Here’s what the Abridge contract structure actually looked like in practice: Sharp deployed the tool across its clinical network in April 2025. Per the vendor agreement, Abridge retained broad rights to access recordings and transcripts. The compliance obligations, including consent workflows, were placed on Sharp. When the lawsuit arrived, both names appeared in the complaint. The legal liability sat with the health system.

A February 2026 analysis in Medical Economics put it directly: many health systems are signing AI vendor agreements without clear answers to who owns the patient data, what happens if an AI output contributes to a clinical error, and what exiting the relationship actually looks like. The vendor’s broad disclaimers are standard language. Those disclaimers do not change the fact that under the current law in most states, the institution remains responsible for whatever makes it into patient care.

That is governance arbitrage. The vendor captures the revenue. The provider carries the risk. The contract made it so.

Sharp is not a one-off. It is one instance of a blueprint.

The consent architecture underneath these tools does not work the way most health system leaders assume it does.

In California, recording a conversation requires consent from all parties. Under the California Invasion of Privacy Act, violations carry $5,000 per incident in statutory damages. Applied across tens of thousands of patient encounters, that exposure can threaten an organization’s financial position, not just its reputation. Sharp’s case is still active.

But Sharp is in California. The picture outside California is more complicated, and in some ways more alarming.

In July 2025, a class action was filed against Heartland Dental, the largest dental support organization in the United States, alleging that patient phone calls were recorded, transcribed, analyzed for sentiment, and used operationally without patients ever being informed. The calls ran through RingCentral’s AI platform. Patients calling to schedule an appointment had their conversation recorded, summarized, and scored for emotional tone. No exam room. No ambient scribe on a device. Just a phone call, quietly routed through a system that was listening, and a vendor capturing the data to optimize its own product.

In January 2026, a federal court dismissed the original wiretapping claims, ruling that AI transcription fell within an “ordinary course of business” exception to the Federal Wiretap Act. Because recording and analyzing calls is what the product does, the court found, it is not eavesdropping under federal law. It is a feature.

The case is continuing. The plaintiff filed an amended complaint in February, arguing that RingCentral’s AI tools are a separate optional product from its phone service, not a core function, and that the company is using those patient calls to train its own models. That distinction, if it holds, could close the loophole. The outcome isn’t settled. But the original ruling is already on the books, and other courts will cite it.

This is the same blueprint as Sharp. RingCentral provides the service, captures the operational value, and sits behind a “core service” defense when liability appears. Sharp and Heartland are not two isolated lawsuits. They are the same contract architecture playing out in different rooms. Exam room. Phone call. The vendor captures the capability. The institution carries the exposure.

In most of the country, the Heartland ruling is currently the answer to whether recording your patients without telling them violates federal law. The legal protection patients might reasonably assume exists does not, if the AI recording is the vendor’s core service.

This is the part I think health system leadership underestimates. The risk is not confined to the exam room or the telehealth visit. It extends into scheduling calls, intake workflows, prior authorization conversations, and any touchpoint where an AI tool is listening and neither the patient nor the contracting organization fully understands the terms under which that listening is happening.

In an earlier piece in this series on consent, I wrote about how the current model of digital agreement has become a legal fiction: you agree to something general, the system does something very specific and seemingly out of scope of your original consent. The gap between those two things is where your accountability is unexamined. That piece focused on software consent flows. The ambient scribe problem is the same architecture applied to your exam room.

A piece I published on observability made a related argument: without a verifiable record of what a system did and when, there is no accountability infrastructure, only assurances. The Sharp case is both problems at once. The AI generated a false consent record, which is a consent failure. It simultaneously created a documentation archive that the health system cannot fully audit or control, which is an observability failure. The two are not separate issues. They are the same gap.

Your symptoms. Your medications. Your mental health history. The conversation you had with your doctor about your marriage because it was affecting your blood pressure. All of it, processed through a system your institution contracted for, under terms your patients never saw.

There is a clinical argument that runs the other direction, and it deserves examination, because dismissing it would cost real patients real harm.

AI systems with longitudinal memory, systems that retain and reason across multiple visits rather than treating each encounter in isolation, show meaningfully better diagnostic performance for slowly developing conditions, including early detection of neurodegenerative disease, than systems that treat each encounter in isolation. The pattern recognition that catches a slow-developing condition requires time and continuity. A system designed to delete everything after 30 days cannot see that the fatigue from October connects to the joint pain from March.

The system designed to protect your patients’ privacy is also the system least equipped to help them. I don’t have a clean answer for that tension yet. What I do know is that this tradeoff is being made right now, in procurement meetings, largely without patient input, and in many cases without the full involvement of legal and risk teams who would ask different questions than clinical and IT leaders ask. Patients are finding out the way Jose Saucedo found out: by reading their own records.

One more number worth sitting with: the ambient scribe market grew 2.4 times in 2025 alone, generating an estimated $600 million in revenue. Abridge, the vendor named in the Sharp lawsuit, is valued at $5.3 billion and is already deployed at more than 200 large health systems, including the VA, Johns Hopkins, and the University of Chicago. Industry projections put the market at nearly $3 billion annually by 2033.

If you are in healthcare leadership and you are not certain whether your organization uses one of these tools, the answer is probably yes. Which means the question is not whether to evaluate this risk. It is whether you are in the position Sharp was in before the lawsuit, or after it.

What I Don’t Have Answers To

The clinical utility argument is real. Ambient scribes are providing genuine value to overwhelmed physicians. The burnout crisis is not a talking point. If I were advising a health system’s IT leadership today, I would not tell them to stop evaluating these tools.

I also don’t know what meaningful consent looks like in emergency contexts. If a patient arrives unconscious, the consent framework breaks immediately.

I don’t have a clean read on where the federal preemption question lands. The Trump administration’s December 2025 executive order directed the DOJ to challenge onerous state AI laws and tasked agencies with developing preemptive standards to avoid a patchwork of fifty different state rules. If that effort succeeds, the California standard that creates Sharp’s exposure may be weakened nationally. If it doesn’t, health systems operating across state lines are navigating that patchwork now, with contracts that were often written before the legal landscape clarified.

And the Heartland case is still moving. If the amended complaint succeeds in arguing that RingCentral’s AI tools are a separate product from its phone service, the “ordinary course of business” loophole narrows. The legal ground is shifting faster than most procurement cycles.

What I’m more confident about: the default of deploying first and building consent infrastructure later is not a viable risk position. The Sharp case will not be the last. The health systems moving fastest without governance infrastructure are not just acquiring tools. They are acquiring liability at scale, signed into their own contracts.

The contracts are signed. The tools are running. The question is whether the governance has caught up.

If you’re in healthcare IT or operations leadership, here are the four questions worth pulling your Business Associate Agreement (BAA) out to answer today. Does your vendor agreement specify whether patient recordings or transcripts are used to train AI models? Who among your vendor’s staff can access those recordings, and under what conditions? What does your consent workflow look like for a patient who arrives without prior notice? And what happens to the data, all of it, when you terminate the contract?

If you don’t have clear written answers to all four, you have a gap your legal team needs to see before the next procurement cycle. Not after the next lawsuit.

In the next piece, I’ll step back from sector-specific cases and look at something that runs underneath all of them: what it would actually mean to build AI systems where the data stays with the people it came from. Not as a privacy feature. As a market mechanism that changes who benefits from the value your patients’ information creates.

If you’re working through these questions in your own organization, I’d be glad to have you along.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind.

It’s Wednesday morning. You haven’t opened your laptop yet. Your AI has already read 63 emails, decided four of them matter, and filed the rest. You start with the four. By 9:30, you’ve cleared them.

Somewhere in the 59 is a security disclosure from a third-party vendor your health system uses for billing. The vendor’s communications team uses AI to generate and distribute notices at scale. The email was formatted like their standard marketing correspondence, sent from the same bulk delivery domain they use for product updates and newsletters. Copilot read the signals correctly. It just didn’t know this message was different.

You find it on Thursday afternoon. By then, the vendor had followed up by phone, concerned no one had responded. The incident they disclosed happened on Tuesday. Your HIPAA-required risk assessment clock started the moment the email arrived.

This is the part of the AI communications story that doesn’t get discussed in the enterprise press.

We’ve been treating AI-generated spam as an inbox nuisance. Something to solve with better filters. But the filters are AI now, too. And the two systems are making decisions about each other without you in the room.

Here’s what the volume actually looks like. Google organic search traffic in the U.S. dropped 38% between November 2024 and November 2025, according to Chartbeat data cited in a January 2026 Reuters Institute report. Major publishers have lost 40 to 55% of their traffic. Some smaller ones have shut down entirely. Google now surfaces AI-generated summaries at the top of results, users stop clicking through, and the economic model that paid humans to write original content quietly collapses.

When the economics of human-written content break because the economics of AI-generated content work better, ad networks don’t distinguish, and algorithms don’t care. You can build an agent that sends email, browses the web, makes phone calls, and negotiates contracts from any laptop, using open-source frameworks, for free. The barrier to entry for mass AI-generated communications is basically zero.

Recently, someone who runs spam infrastructure for one of the largest platforms made a prediction: within 90 days, email, phone calls, and messaging would be so flooded with AI-generated content they’d no longer be reliably usable. Then he bought 1.7 million bot accounts on X to prove the point. They used to be simple, and now they have natural language fluency and payment infrastructure. What’s changed now isn’t sophistication alone. It’s the combination of scale, personalization, and near-zero cost, collapsing simultaneously.

This isn’t just a volume problem. Reinforcement learning from human feedback, RLHF, is how most major language models get trained. You show humans two responses and ask which one they prefer. Repeat that millions of times, and you get a model mathematically optimized to produce content people find compelling. Not accurate. Not helpful. Compelling. The mechanism is the same whether you’re building a helpful assistant or a content generator running spam operations. Every piece of AI-generated content hitting your inbox was built with that same optimization. Your spam filter was not.

Think of it this way. Your spam filter is trying to catch traffic specifically engineered, at the model level, to look like legitimate communications. Then spam got a PhD in looking legitimate while your filter studied for the old exam.

Meanwhile, the filter is getting smarter too. Microsoft’s Copilot “Prioritize My Inbox” feature reached general availability in April 2025. More than 25% of business inboxes now use some form of AI to categorize, prioritize, or triage incoming email. Superhuman, Google Gemini, and a growing range of standalone tools do the same. The pitch is straightforward: let AI decide what you need to read. Reduce cognitive load.

I understand why it exists. The problem isn’t the feature. The problem is what happens at the intersection of a filter trained on your behavioral history and a content environment trained to exploit behavioral patterns.

AI inbox prioritization learns what you respond to. It models your patterns. But organizational risk does not follow your patterns. Breach notifications don’t follow your patterns. Regulatory notices don’t follow your patterns. A compliance disclosure sent at scale, from a bulk delivery domain, formatted like every other vendor communication you’ve deprioritized this year, will register to your AI filter exactly as it should: low priority. The AI made a correct call with incorrect consequences.

Microsoft disclosed in early 2026 that a logic error had briefly caused Copilot Chat to process and summarize emails labeled Confidential, regardless of sensitivity settings. The fix was deployed quickly, and the disclosure was candid. But the incident points to something worth noting: the AI triage layer is already making decisions about your sensitive communications. The governance controls are still catching up.

There’s no audit log that tells you which emails were deprioritized and why. There’s no notification when a time-sensitive message is sorted away. What you have is a black box that handles 59 of your 63 daily decisions and surfaces the four it thinks matter. When the black box is wrong, you find out from a phone call three days later.

The structure is identical to something I wrote about earlier in this series: Pactum’s AI negotiating purchasing contracts for Walmart and Maersk while the suppliers’ systems responded on the other end. The humans set the parameters and stepped away. The accountability gap didn’t stay contained. It compounded. Each system operated correctly within its own design. Neither was built to flag when the combination produced consequences that neither principal anticipated.

The inbox version is quieter. Less visible. An AI generated the message. An AI filtered it. A human never saw it. The interaction logged correctly in both systems. The consequences surfaced in a phone call from a vendor asking why no one responded to the disclosure they sent on Tuesday.

If you’re a VP of IT at a health system, this isn’t abstract. Your organization runs on vendor relationships, compliance timelines, and regulatory correspondence. Your team deployed AI inbox tools because they work: the cognitive load reduction is real. Your vendors deployed AI communications tools because their teams are small, and the volume demands are massive. The two systems talk to each other dozens of times a day. Nobody mapped that conversation before it went live.

Regulators and courts will eventually have to decide whether ‘AI filtered it’ qualifies as constructive receipt of a notice. Existing doctrines around duty to monitor and constructive notice were not written with AI triage in mind. That question is coming. Most organizations aren’t ready for it.

What I Don’t Have Answers To

We don’t yet have a design pattern for an AI filter that is appropriately skeptical without becoming useless. The reason Copilot deprioritizes bulk-formatted email is correct for 98% of bulk-formatted email. The open question is how to handle the 2% that matters without recreating the inbox overload you were trying to escape. Nobody has solved that cleanly.

The liability question is equally unresolved. If your vendor’s communications AI sends a compliance disclosure using a template that triggers your organization’s AI filter, and you miss the notification window, who is responsible? The vendor? Your organization? The filter vendor? Existing frameworks don’t map cleanly onto a chain where neither sender nor receiver is human. That’s a design gap, not just a legal one.

The filter doesn’t know that. The messages most likely to get deprioritized without review are from smaller senders: smaller vendors, nonprofits, community organizations, and individuals. Optimization for signals of legitimacy, bulk domain, generic template, low engagement history, structurally disadvantages low-signal but high-importance senders. It’s not malicious. It’s what happens when nobody asks the system to care about the outliers.

This is being framed as a consumer problem: Is your personal inbox manageable? But that framing misses what’s actually breaking. For organizations still operating on the assumption that a message sent is a message received, the gap has already opened. In healthcare and financial services, it’s a liability problem now, not a future one.

The only communications still carrying a reliable signal are from people you already trust. When the verification layer fails, you fall back to the social layer. The problem for enterprises is that the social layer doesn’t scale, isn’t auditable, and shuts out anyone who hasn’t already earned a relationship with you. Smaller vendors, nonprofits, and individual constituents. They get filtered. They don’t get a callback.

That’s not an inbox problem. It’s an architecture problem.

If you’re thinking about these questions too, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

Margaret is 73. She had a hip replacement in October. Her surgeon recommended ten days of inpatient rehabilitation. Her Medicare Advantage plan only approved two.

She didn’t know she could appeal. The denial came as a one-page letter with a phone number for member services. She called twice, held both times, and eventually stopped. She went home with a walker and instructions to follow up with outpatient physical therapy three times a week, if she could get there.

She couldn’t always get there.

I’m starting with Margaret because I want to be specific about who this affects before we talk about systems. The systems conversation is important. But Margaret is the reason it matters.

I want to say something upfront to the people in this audience who work in insurance or health systems, because I know some of you do.

The business case for AI in utilization management isn’t irrational. Claims volume is enormous. Human reviewers are inconsistent. An AI that processes thousands of claims quickly, at low cost, with consistent criteria, sounds like exactly the kind of infrastructure investment that makes sense.

The problem isn’t that the technology exists. The problem is what happens when the system’s error rate stops mattering, because the incentives run the other way.

That’s what I want to trace here.

Last year, a federal class action lawsuit against UnitedHealthcare alleged that the company used an AI model called nH Predict to evaluate post-acute care claims for Medicare Advantage patients. The plaintiffs alleged the model had a 90% error rate. That it denied care even when treating physicians had documented medical necessity. That UnitedHealthcare’s denial rate for post-hospital care doubled after the tool was deployed, from 10.9% to 22.7%.

UnitedHealthcare disputes all of it, and the case is still in court. I want to be clear about that. What I’m describing are allegations, not findings. I’m citing them because they’re in the public record, and the questions they raise are worth asking regardless of how the litigation resolves.

The number that’s not in dispute: 90% of UnitedHealthcare’s denials get overturned when a patient actually appeals to a federal judge.

Only 0.2% of patients ever do.

Here’s the math the system runs on: you don’t need the AI to be right. You just need the denial to be intimidating enough that most people don’t fight it. And most people don’t.

I manage technology infrastructure for a global academic community. Error rates are something I think about constantly. If a process in my environment was failing 90% of the time, we’d take it down. We’d fix it. Scaling it to millions of patients would not be on the table.

The only context in which you scale a 90% error rate is when the errors are profitable.

And I can’t get past this: when a denial saves money and only 0.2% of those denials are ever challenged, the incentive structure doesn’t reward accuracy. It rewards volume. The AI doesn’t have to be good. It has to be good enough that the economics work.

The conversation about AI in healthcare focuses on the wrong question. Everyone wants to know: is the model accurate? That’s not the question that matters. The question is: what happens when it’s wrong? Who catches it? What does the error cost? For whom?

For Margaret, the cost was going home without the care her surgeon recommended. For the insurer, the cost of that error was zero. She didn’t appeal.

UnitedHealthcare isn’t the only one. In 2023, a ProPublica and CBS News investigation found that Cigna physicians reviewed more than 300,000 claims in a two-month period using automated decision support tools. Cigna disputes the characterization and says physicians exercise independent medical judgment. The volume isn’t disputed. 300,000 claims in two months.

The American Medical Association found that 71% of insurers now use AI for utilization management. Not a pilot. Standard practice, at scale, with no federal framework governing accuracy requirements, bias audits, transparency, or appeals.

We require clinical trials before a drug reaches the market. Years of them. Proof of efficacy. Mandatory side effect disclosure. Post-market surveillance. A drug with a 90% error rate doesn’t get approved.

An AI system making life-or-death coverage decisions for Medicare patients has none of those requirements. We find out that a model has a 90% error rate because a class action lawsuit surfaces internal documents. That’s not a system working. That’s luck, and it only helps the people who can afford to sue.

The denial-rate problem is one layer. The bias problem sits underneath it, and they’re not separate issues.

Cedars-Sinai researchers found that AI clinical decision support tools recommend inferior psychiatric treatment options when a patient is identified as Black. The AI’s algorithm doesn’t invent the disparity. It’s inherited from training data that reflects decades of documented inequity in how care has been delivered and recorded.

Think of it this way. If a physician consistently recommended worse psychiatric options to Black patients out of habit rather than clinical evidence, we’d expect accountability. When an AI trained on that physician’s historical decisions does the same thing, accountability becomes nearly impossible to locate. The company points to the data. The data reflects historical practice. Historical practice reflects structural inequity. No single actor is responsible because everyone can point to the layer below them.

I don’t think that’s an acceptable answer. It is, for now, where the law leaves us.

For anyone in this audience who is deploying clinical AI: if you haven’t audited your training data for demographic disparities, you’re not just carrying an ethical risk. You’re carrying a liability risk that the courts haven’t finished mapping yet.

The hospitals are using AI. The physician groups. The appeals coordinators. The pharmacy benefit managers. At least four separate AI systems may touch a single claim between initial authorization and final decision. Each has its own training data, its own error rate, its own optimization target.

We don’t have standards for how those systems interact. We don’t require that a denial disclose that it was AI-generated. In some states, we don’t require a licensed physician to review an AI recommendation before it becomes a coverage decision communicated to a patient.

In my article, Show Me the Receipts, I wrote about observability as a human right: the ability to see what an AI system did, why it did it, and who is accountable when it’s wrong. Healthcare is where that absence costs the most. When an AI denies care and the patient doesn’t appeal, there’s no record that the denial was probably wrong. The case closes. The error disappears.

The 0.2% who appeal get overturned almost every time. The 99.8% who don’t are invisible in the data.

That’s worth sitting with. If appeals succeed at that rate, the AI was wrong far more often than anyone’s tracking.

What You Actually Have the Right to Do

When a claim is denied, you have the right to appeal. The internal appeal is step one, not the final word. If the internal appeal fails, you have the right to an external independent review conducted by an organization with no relationship to your insurer. Insurers lose those reviews at high rates. Most people never get there because no one tells them the internal appeal isn’t the ceiling.

Your denial letter is required by law to cite the specific reason for denial and the policy language used. You can also request the clinical criteria the insurer applied, typically from a system called InterQual or MCG. That document is what your appeal actually needs to respond to. Most people never ask for it.

Deadlines matter. Appeals for most commercial plans must be filed within 180 days of the denial date. Medicare Advantage has its own timeline with enforceable federal rules. The clock starts when you receive the denial letter, not when you find out you have the right to fight it.

If you’re on Medicare and your health is at risk, you have the right to an expedited appeal. The decision is supposed to come within 72 hours. Most Medicare patients have never heard of it.

The system isn’t designed to surface these rights. You have to know that you can ask.

So I Built Something

After I posted about insurance AI denial rates on LinkedIn, someone replied: “I wonder if it will level the playing field when someone creates an appeals AI algorithm and insurers experience a 100% appeals rate.” A few other people said similar things. I’d been thinking about it for a while and decided to go for it.

I built it with Claude. I’m building it in public, with feedback from the people who actually need it and my audience, many of whom work for insurers.

It’s called Overturned. Free. No login required. Your documents are never stored or used to train any model. You upload your denial letter, the tool reads it, identifies the specific grounds for denial, and generates an appeal letter that responds to those grounds directly. It shows you exactly what it found before you send anything.

Upload your denial. Review the findings. Refine the letter. Appeal.

About three minutes. About $0.07 in AI processing costs, which I cover through donations and my consulting practice. No paywalls. No ads. No data selling.

Insurance companies have been using AI to deny claims at scale for years. Overturned is AI writing back.

It’s a live beta, which means it’s not perfect. What it does well: it extracts the denial rationale, matches it to the cited clinical criteria, and drafts a letter in language that appeals reviewers recognize. What it doesn’t do: give legal advice. If your situation is complex or the stakes are high, talk to a patient advocate or a healthcare attorney.

There’s a public roadmap on the tool page where you can vote on features and suggest what we build next. If something doesn’t work the way you need it to, that feedback is how it gets better.

The appeal you don’t file is the one you can’t win.

What I Don’t Have Answers To

I’m not arguing AI has no role in utilization management. The claims volume is real and human reviewers make errors too. What I’m arguing is that we need a standard of evidence before deployment: a required accuracy floor, mandatory demographic bias audits, and clear rules about who reviews AI decisions before they become final. None of that currently exists at the federal level.

I also don’t know how to solve the 0.2% problem without making the appeals process dramatically simpler. Overturned helps with the letter. Someone still has to know they can fight, decide it’s worth it, and follow through. That’s not a technology problem. It’s a structural one.

And I genuinely don’t know what accountability looks like when the bias lives in the training data. Who audits the model? Who has standing to require changes? The Cedars-Sinai finding is one data point. How many other health systems have run that same analysis?

My guess: not many.

If you’ve had a claim denied and fought it, or you’ve tried Overturned, I want to hear about it. The comment section is open. That’s how the tool gets better, and honestly, it’s how I understand what’s actually happening out there.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build s

In my last piece, I wrote about the proprietary data problem: employees feeding sensitive information into AI tools they don’t control, coding assistants introducing vulnerabilities, agents destroying production databases. I ended by saying the next piece would be about observability as a human right.

Then, six days later, a federal court made the argument for me.

On March 9, 2026, a federal judge in Minnesota ordered UnitedHealth Group to open up its AI playbook. Becker’s Payer Issues reported that the court sided with plaintiffs across six of seven discovery categories. The order didn’t just ask what model the company was using. It demanded internal documents about an algorithm called nH Predict: who built it, how it’s used, who it incentivizes, and how it shapes coverage denials.

The lawsuit tells one man’s story. According to the complaint, Gene Lokken was at a skilled nursing facility after a medical crisis. In July 2022, UnitedHealthcare cut his coverage. They said additional days weren’t medically necessary. Lokken and his physician disagreed. They appealed and lost. Medical Economics reported that his family paid $12,000 to $14,000 a month out of pocket for almost a year. He died on July 17, 2023.

When the family asked why, there was no real answer. The decision came from an algorithm. The reasoning was proprietary. An Optum spokesperson told Becker’s that nH Predict is “a guide” to help inform caregivers. But a STAT investigation cited in the lawsuit found that UnitedHealth pressured employees to keep patient stays within 1% of the algorithm’s prediction.

According to the plaintiffs, appeals data suggest nH Predict may be wrong about 90% of the time when denials are challenged. Nine out of ten denied claims were reversed on appeal. But a Kaiser Family Foundation report found that only about 0.2% of policyholders ever challenge a denial. The tool kept running because the appeals rarely came.

Your Peers Just Became Case Studies

UnitedHealth isn’t the only cautionary tale.

A 2023 ProPublica investigation found that Cigna used an algorithm called PXDX to deny over 300,000 claims in two months. According to internal spreadsheets reviewed by ProPublica and The Capitol Forum, medical directors were signing off on denials in batches, spending an average of 1.2 seconds per case. One doctor reportedly denied 60,000 claims in a single month. Nobody opened a chart.

The class action filed in Sacramento names specific patients. Suzanne Kisting-Leung had an ultrasound to check for ovarian cancer. It found a cyst. Cigna denied the claim. Another plaintiff had a vitamin D test ordered by her doctor. Denied. According to the complaint, Cigna gave no explanation for either.

Cigna disputes these characterizations. But the allegations themselves are already shaping how regulators and plaintiffs’ attorneys look at every carrier using similar tools.

Regulators and plaintiffs’ attorneys don’t care whether you call these tools “guides,” “assistants,” or “workflows.” If an algorithm narrows the funnel, humans rubber-stamp the output, and patients can’t get a meaningful explanation, then in practice the algorithm is making the decision. That’s the story now on the record. And it’s the lens regulators will apply to every carrier using AI.

The Question Your Board Should Be Asking

I talk to IT directors and VPs in regulated industries. Executives keep asking: “Is our AI compliant?”

That’s the wrong question.

The right one: if a judge orders you to explain any AI-influenced denial tomorrow, case by case, can you? Not at the aggregate level. For each individual decision. Can your chief medical officer explain why the AI recommended denial in a specific case? Can your CIO show, in a traceable way, how a given data point influenced the outcome? Can your compliance officer demonstrate to regulators that humans still truly own the decision? This is not an IT question. It’s a board-level risk question that happens to be implemented in code.

If the honest answers are “no,” “not really,” and “we hope so,” you have a governance problem hiding under that efficiency costume.

Recent AMA surveys suggest a majority of physicians believe unregulated AI tools are driving more prior authorization denials. At a 2025 industry conference, the AAPC Knowledge Center reported that algorithmic denials dominated every panel and hallway conversation. Attendees cited reversal rates on appeals around 90%. The system was wrong most of the time. It kept running anyway.

What Observability Actually Means

I’ve spent most of my career managing IT infrastructure. In my world, observability means the ability to understand what a system is doing by looking at its outputs. When a server goes down, you check the logs. When a query runs slow, you trace it. If a platform I run for 45,000 researchers goes down and I can’t tell my team why, that’s a career-limiting event.

That is the baseline for a web server. It should not be too much to ask of a system that decides whether a 74-year-old gets to stay in rehab.

Think of it this way. If your building’s fire alarm evacuated everyone and nobody could explain why it went off, you’d call that a broken system. You wouldn’t keep using it. You wouldn’t trust it with people’s safety.

That’s what we’re doing with AI right now. Trusting it with people’s safety. And we can’t tell anyone why it does what it does.

Where Regulation Stands

The EU AI Act is the most ambitious attempt to require explanation. According to a Cogent Infotech analysis, high-risk AI systems in healthcare and credit scoring will need auditable decision logs by August 2026. Penalties run up to €20 million or 4% of global revenue.

In the U.S., states are moving first. California’s Physicians Make Decisions Act, effective January 2025, requires a qualified physician to make final medical necessity decisions. Not an algorithm. Texas followed in June 2025, requiring licensed practitioners to review AI-influenced medical records before clinical decisions are made. Colorado’s AI Act, effective February 2026, requires impact assessments and explicit notice when AI is used in consequential decisions, including insurance.

But the pressure is moving in both directions. As Gunderson Dettmer and Pearl Cohen both reported, a Trump executive order signed in December 2025 directs the Attorney General to challenge state AI regulations deemed obstacles to national competitiveness. If you’re in a regulated industry counting on state laws to clarify your obligations, that ground may shift under you.

When the rules are in flux, courts fall back on a simple standard: did you know what your system was doing, and did you act responsibly? “We deployed a black-box model from a major vendor” will not hold up well on the stand.

How This Blows Up

You don’t need a dramatic scenario. You need one bad week.

The class action week. A plaintiff’s firm realizes a single algorithm touched tens of thousands of denials. They move for class certification, arguing the system applied the same flawed logic to everyone. You’re not defending one denial anymore. You’re defending your entire AI operating model.

The regulator week. A state insurance department picks up a media story, opens an investigation into AI-assisted denials, and discovers you can’t produce meaningful decision logs. Other states notice. Multi-state market conduct exams follow.

The boardroom week. A patient story goes viral. Journalists start asking your CEO for specifics about your algorithms. The board asks: did we approve this risk? Who owns it? How much have we actually saved, and at what exposure?

All three hinge on the same weakness: you can’t show the receipts. In every scenario, your CIO, CMO, chief compliance officer, and general counsel are answering for systems nobody on the team can fully explain.

What Observability Would Actually Look Like

I’m not proposing that every AI decision needs a 20-page report. I’m proposing that when a system makes a decision that materially affects someone’s health, finances, or freedom, that person has the right to a meaningful explanation. Not “based on our analysis.” A real answer.

That means three things at minimum. The inputs should be visible: what data the system used, from which sources, with what known limitations. According to the Lokken complaint, nH Predict analyzes a database of 6 million patients, looking at diagnosis, age, living situation, and physical function. If those factors ended Lokken’s coverage, his family deserved to know which ones mattered and how much weight each carried.

The reasoning should be traceable. Not the full model architecture, but the chain of logic from input to output. California already requires a physician to make the final call on medical necessity. If a doctor has to explain their reasoning, a system that replaces the doctor should too.

And the confidence should be disclosed. Was this a strong call or a close one? If the model was 51% confident, that’s a very different situation than 95%. Both produce the same denial. But they demand very different levels of human review.

If your AI vendors can’t provide all three on demand, that’s not a feature request. That’s a red flag your board should see.

The technology for all of this exists today. Decision logs. Confidence scores. Interpretable model designs. These are production tools, not research concepts. The obstacle is not capability. It’s incentive. Explanation costs compute. It slows processing. It creates records that can be used in court. Every reason Cigna had for processing claims in 1.2 seconds is a reason they wouldn’t want to explain each one.

The Connection to This Series

If you’ve been following this series, observability connects to every argument I’ve made. In my first piece, I proposed an agent registry with audit trails. Those trails are useless if the reasoning behind each action is a black box. In my second piece, I argued that consent requires transparency. You should be able to ask your agent “Why did you do that?” and get a real answer. In my last piece, I wrote about what happens when your AI runs on someone else’s infrastructure. Observability is the other side of that problem. When you can’t see how a system makes decisions, you can’t tell whether those decisions serve your interests or the platform’s.

Here’s what I keep coming back to: observability is the mechanism that makes accountability possible. Without it, everything else I’ve proposed has no teeth. Accountability without observability is theater.

What I Don’t Have Answers To

I don’t know where to draw the line on what counts as a “material” decision. A healthcare denial is obvious. A product recommendation probably isn’t. But between those poles there’s a lot of gray, and requiring explanation for every automated decision may not be practical.

I don’t know how to make explanations useful to people who aren’t engineers. A decision log full of technical jargon is transparent in theory and useless in practice. Observability that only serves the people who built the system isn’t a right. It’s a feature.

And I worry about explanation becoming its own kind of theater. Companies are already producing “explainability reports” that check a compliance box without telling the affected person anything meaningful. If that’s where this goes, we’ve traded one performance for another.

But the alternative is the world we’re in right now. Algorithms making life-altering decisions. Nobody able to say why. Gene Lokken’s family knows what that costs.

What I’m Asking

If you work in insurance, healthcare, or financial services: look at the AI tools your organization has deployed and ask what they can explain about their own decisions. If the answer is “not much,” bring that to your compliance team, your legal team, and your board. California and Texas have enforceable laws now. Colorado takes effect next month. The Lokken court just ordered an insurer to open its algorithm to discovery. The direction is clear.

If you can’t explain your AI-driven denials to a judge, don’t expect your shareholders or regulators to be any kinder.

If you manage IT infrastructure in a regulated environment, you’re going to be the person who has to answer the question: can our AI tools show their work? Start asking your vendors now. The ones who can’t answer are telling you something about the risk you’re carrying.

And if you’ve been on the receiving end of a decision that didn’t make sense, a denied claim, a rejected application, a coverage termination your doctor disagreed with: ask for the explanation. You may not get one. But the asking matters. And as the Lokken ruling shows, courts are starting to treat that silence as something worth investigating.

This is the seventh in a series about AI accountability.

If you’re thinking about these questions too, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

In early 2023, Samsung allowed its semiconductor engineers to use ChatGPT to help with their work. Within twenty days, three separate incidents occurred.

One engineer pasted source code from a semiconductor database to debug an error. Another submitted proprietary chip-testing code for optimization. A third uploaded an entire internal meeting transcript to generate minutes.

All three inputs became part of OpenAI’s training data. Samsung couldn’t retrieve any of it.

The information was retained, processed, and absorbed into the model’s weights. Samsung issued a company-wide ban on ChatGPT. JPMorgan, Amazon, Verizon, and Walmart followed.

But here’s the detail that doesn’t get enough attention: Samsung eventually lifted the ban. By 2025, the company relaxed its restrictions because the productivity gains were too valuable to walk away from.

That tension is the one I keep thinking about. The competitive advantage AI offers versus the competitive exposure it creates. Most organizations are living inside that tension right now. Most of them haven’t named it yet.

The Slow Leak Nobody Is Watching

Samsung made headlines because it was public. The same dynamic is playing out quietly across thousands of companies every day.

LayerX Security’s 2025 Enterprise AI Report found that 77% of employees have pasted company information into AI tools. More than half of those paste events included corporate data. And 82% of those workers used personal accounts rather than enterprise-managed tools, which means the data bypassed every security control their company had in place.

Cyberhaven’s research tells a similar story. 34.8% of all corporate data going into AI tools is now classified as sensitive: source code, R&D materials, financial projections. That’s up from 10.7% two years earlier. The rate isn’t creeping upward. It has tripled.

I manage technology infrastructure for 45,000 scholars across a global academic community. I think about data flows constantly: what crosses boundaries, what gets retained, what becomes visible to the wrong audience at the wrong time. When I look at these numbers, I don’t see an abstract risk. I see an organization that has already lost something and doesn’t know it yet.

Only 17% of organizations have automated controls to block or scan uploads to public AI tools. The other 83% rely on training sessions, email warnings, or nothing at all. And once data enters a public AI system, it cannot be retrieved. Every unmonitored employee prompt is a potential compliance failure under GDPR, HIPAA, or SOX.

Most companies can’t even answer a basic question: which AI tools currently hold our proprietary data? That’s not a gap in security posture. It’s a gap in awareness.

When the Tool Breaks What It Touches

Data leakage is the slow-moving risk. There’s also a fast-moving one.

In July 2025, Jason Lemkin, founder of the SaaS community SaaStr, was testing Replit’s AI coding assistant. On the ninth day of his project, during an active code freeze with explicit instructions that no changes should be made without permission, the AI agent deleted his entire production database. Records for over 1,200 executives and nearly 1,200 companies. Gone.

When Lemkin confronted the agent, it admitted to running unauthorized commands. Then it told him rollback was impossible and the data was gone forever.

That turned out to be wrong. Lemkin recovered the data manually, going against the agent’s own advice.

But the agent didn’t just destroy data. It fabricated it. It generated over 4,000 fake user records with completely made-up information to fill the void. When asked to score its own behavior on a 100-point severity scale, it gave itself a 95. Replit’s CEO called the behavior “unacceptable.”

This isn’t an isolated case. Days later, Google’s Gemini CLI agent deleted a user’s files after misinterpreting a command. In August 2024, researchers showed that Slack’s AI could be tricked into summarizing sensitive private-channel conversations and sending those summaries to an external address. The AI thought it was being helpful. It was functioning as an insider threat.

None of these agents were hacked. They were doing exactly what they were designed to do: execute commands on their own. That’s the same pattern I’ve been writing about since the first article in this series: delegation without oversight, at a speed no human can supervise.

The Vulnerabilities You Can’t See

There’s a third problem. It’s quieter than the other two, but at scale, it may be the most dangerous.

AI coding assistants are introducing security vulnerabilities into proprietary codebases faster than human developers can find them. Apiiro, an application security platform, analyzed tens of thousands of repositories across Fortune 50 companies. By June 2025, AI-generated code was producing over 10,000 new security findings per month. That’s a tenfold increase in just six months.

These aren’t formatting errors. Privilege escalation paths increased 322%. Architectural design flaws spiked 153%. AI-generated code was 2.74 times more likely to contain cross-site scripting vulnerabilities. Developers using AI assistance exposed cloud credentials at nearly double the rate of those working without it.

The reason is straightforward: the models were trained on vast repositories of open-source code, and much of that code contains the same vulnerabilities they now reproduce. The model doesn’t understand your security architecture. It optimizes for finishing the task, not for protecting the system. Ask it to query a database and it might hand you a textbook SQL injection flaw, because that pattern appeared thousands of times in its training data.

The Architecture Is the Problem

Here’s what I keep coming back to.

Data absorption. Autonomous destruction. Vulnerability injection. These look like three separate problems, but they’re all symptoms of the same architectural choice: feeding your proprietary knowledge, your production access, and your code into systems you don’t own and can’t inspect.

Think of it this way. If you rented office space and the landlord could read every document on your desk, copy your filing cabinets, and hand the contents to your competitor across the hall, you’d move. That’s roughly the arrangement most organizations have with cloud AI. Except the lease is called “terms of service,” the filing cabinets are training data, and most tenants haven’t read the fine print.

The cloud sold us convenience in exchange for control. For most workloads, that trade was reasonable. But AI workloads are different. AI learns. That’s the entire value proposition. When the system learning from your work is hosted on someone else’s infrastructure, the cost goes beyond a subscription fee.

In my last article, I wrote about entire nations discovering this cost. The chief prosecutor of the International Criminal Court lost his email because Microsoft complied with U.S. sanctions. Amsterdam Trade Bank lost its cloud because of a court order from another continent. That was about collaboration tools. This is about something more intimate: the proprietary knowledge that makes your organization competitive. When that knowledge trains a model you don’t control, you lose more than access to infrastructure. You lose control of what the infrastructure learned by watching you work.

The Market Is Doing the Math

The repatriation wave isn’t a prediction. It’s happening. A February 2026 survey found that 93% of enterprises have already moved AI workloads off public cloud, are in the process, or are actively evaluating it. 91% said they would choose on-premises or hybrid infrastructure over public cloud for AI involving sensitive data.

Gartner calls the trend “geopatriation” and named it a top strategic technology trend for 2026. They project that 75% of European and Middle Eastern enterprises will move to sovereign environments by 2030, up from 5% in 2025. Organizations that have already made the shift are documenting cost savings of 30 to 60 percent.

37signals, the company behind Basecamp, is the clearest case study. They were spending $3.2 million a year on AWS. They bought $700,000 in Dell servers and moved on-premise. Savings: nearly $2 million a year. Projected total over five years: more than $10 million. Their CTO said the industry convinced everyone that owning hardware is impossible. It isn’t.

And in a move that would be funny if the stakes weren’t serious: Microsoft launched “Sovereign Cloud” in February 2026, letting organizations run AI models on their own hardware, fully disconnected from Microsoft’s central cloud. The company that locked the ICC prosecutor out of his email is now selling the fix.

Your Tier 1, at Organizational Scale

Throughout this series, I’ve argued for a tiered model of AI accountability. Tier 1 is local. Your agent runs on your hardware, stays in your space, and doesn’t need anyone else’s permission to operate.

On-premise AI is Tier 1 thinking applied at organizational scale. When you run models on your own hardware, the principle holds: what stays local stays yours. Your pricing logic doesn’t become training data for a competitor. Your customer patterns don’t get aggregated into a model someone else can query. Your production database isn’t at the mercy of an agent whose guardrails were set by another company’s product team.

The same principle works at every level. For an individual, it’s a Raspberry Pi running a local agent that doesn’t phone home. For an organization, it’s AI on hardware you own. For a nation, it’s France building sovereign collaboration tools and Germany migrating 30,000 government workstations off Microsoft. The thread connecting all three: when software learns from what you feed it, where it runs determines who benefits from the learning.

What I’m Still Working Through

I want to be honest about the limits of this argument.

On-premise AI isn’t free. The hardware costs are real. Maintaining your own infrastructure takes expertise that most mid-sized organizations don’t have on staff. 37signals runs a ten-person operations team with decades of experience. That’s not a typical bench.

The cloud remains excellent for prototyping, elastic workloads, and global distribution. I’m not arguing that every organization should move everything tomorrow. I’m arguing that where you run your production AI is not a neutral technical decision. It’s a decision about who controls your data, who learns from your operations, and who holds the keys when things go wrong.

I also worry about the equity gap. Data sovereignty could easily become another advantage that accrues to organizations with deep pockets while smaller players stay locked into the rental model. I don’t have a clean answer for that yet.

But the current default isn’t neutral. It was designed by companies whose revenue depends on you renting their infrastructure and feeding their models. Whether that default serves your interests is a question worth asking.

The cloud made us tenants. AI made the landlord observant. Are you comfortable with what they’re learning from watching you work?

This is the sixth in a series about AI accountability. In the next piece, I’ll look at observability as a human right: why you deserve to see what your AI did and why, and what it means when the systems shaping your decisions can’t show their work.

If you’re thinking about these questions too, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

In 2025, in The Hague, the chief prosecutor of the International Criminal Court, Karim Khan woke up one morning and couldn’t access his email. Not because of a cyberattack. Not because of a technical failure. Because Microsoft shut it off. The United States government had imposed sanctions on ICC officials, and Microsoft, which hosted the court’s email infrastructure, complied. Khan was locked out of his own Outlook account. He switched to Proton Mail.

The chief prosecutor of an international court, headquartered in the Netherlands, who was conducting investigations into war crimes, lost access to his professional email because a company in Redmond, Washington, followed an order from a government in Washington, D.C.

That’s not a hypothetical scenario in a think-piece about digital risk. It already happened.

And it’s the same problem I’ve been writing about in this series, scaled up from individuals to institutions, from AI agents to operating systems, from “who authorized that purchase” to “who controls whether your government can function.”

The Default We Never Chose

In my last piece, I said I wanted to explore the quiet violence of defaults and how the settings no one changes become the architecture of power. I was thinking about AI agent configurations: the spending limits that default to “unlimited,” the permissions that default to “allow all,” and the consent checkboxes that default to “yes.”

But the biggest default I can think of isn’t inside an AI agent. It’s inside every government, hospital, school, and court in Europe.

The default is Microsoft.

Not because anyone made a deliberate, strategic decision that American cloud infrastructure should be the backbone of European public institutions. It happened the way defaults usually happen: incrementally, conveniently, and without anyone asking what it would mean twenty years later. One department adopted Office. Then another. Then the email server moved to Exchange. Then Teams replaced the conference room. Then SharePoint became the filing cabinet. Then, Azure became the server room.

At each step, the decision was reasonable. Microsoft makes good software. It’s reliable, well-supported, and familiar. The procurement team chose the vendor that met the requirements. Nobody in those early meetings was debating sovereignty or jurisdiction or what happens if a foreign government decides to flip a switch.

But here’s what accumulated while nobody was watching: a dependency so deep that the Dutch Data Protection Authority now warns that if another country chose to exploit it, the Netherlands could be brought to a complete halt. Not a slow degradation. A halt. Healthcare, payments, government services, authentication—all running through infrastructure controlled by companies that answer, ultimately, to another country’s laws.

This is the consent problem from my second article, applied at the scale of nations. These governments clicked “Allow.” They granted access to their email, their documents, their collaboration workflows, and their citizen data. They didn’t read the terms of service, not because they were careless, but because at the time, it didn’t seem to matter. The vendor was reliable. The software worked. Why would you interrogate convenience?

Now the terms of service matter. And the gap between “we agreed to use this product” and “we understood that a foreign government could lock us out of our own systems” is the same gap I keep writing about: the space where harm lives.

Three Countries, Three Lessons

What’s happening right now in Europe isn’t an abstract policy debate. It’s governments doing, in real time, what I’ve been arguing individuals need to do with AI agents: examining the consent they gave, questioning the defaults they accepted, and building alternatives that put control closer to the people affected.

Three cases stand out, each illustrating a different part of the problem.

Germany: The proof it’s possible.

Schleswig-Holstein, Germany’s northernmost state, started migrating away from Microsoft five years ago. It began as a cost-cutting exercise. It became something else entirely.

As of late 2025, nearly 80 percent of the state’s 30,000 government workstations have switched from Microsoft Office to LibreOffice. They’ve migrated over 40,000 email accounts and more than 100 million emails from Outlook and Exchange to open-source alternatives. They replaced SharePoint with Nextcloud. They replaced Teams with Jitsi. They’re testing Linux to replace Windows on desktops.

The numbers are striking. The state projects savings of more than €15 million in license costs in 2026 alone, money that was previously going to Microsoft every year. The one-time migration investment was €9 million, which pays for itself in under twelve months.

The state’s CIO said something that sticks with me: “What began as a technical project is now a political project.” That’s the trajectory of defaults. You start by questioning a line item in the budget. You end up questioning who controls the systems your government runs on.

I don’t want to oversimplify this. The migration hasn’t been smooth. Some employees can’t work properly with the new tools yet. Specialized applications still depend on Microsoft. Critics in the state parliament point out that 80 percent converted on paper doesn’t mean 80 percent of people can do their jobs effectively. These are real problems, and pretending otherwise would undermine the argument.

But the argument isn’t that migration is easy. It’s that it’s possible. And that the alternative, indefinite dependency on infrastructure you don’t control, has costs too. They’re just harder to see until someone flips the switch.

France: The explicit sovereignty decision.

France is taking a different approach: not migrating away from big tech piecemeal, but declaring that collaboration infrastructure is sovereign territory.

The French government announced that it will phase U.S. Big Tech collaboration platforms out of government workflows entirely, replacing them with a domestically built platform called Visio. The transition is planned to be complete by 2027.

This isn’t France’s first move. The French national police force, the Gendarmerie nationale, has been running over 100,000 workstations on its own custom Linux distribution since the early 2000s. The Ministry of Education banned free versions of Microsoft 365 and Google Workspace in schools over data privacy concerns. France has a “Cloud at the Center” policy that treats digital infrastructure the way it treats energy infrastructure: as a matter of national capacity.

What makes the Visio decision different is the framing. This isn’t presented as a cost-saving measure or a technical preference. It’s presented as a governance decision. The pension worker in Lyon who video-calls a tax specialist in Paris to sort out a retiree’s benefits, that call happens hundreds of times a day across the French government. Right now, it flows through Teams or Zoom. Soon it will flow through a tool built in France, governed by French law, accountable to French institutions.

France is drawing a line: when a system is central to how your government operates, you need to control who has authority over that system when things go wrong. Not the authority to use it. The authority to shut it off.

The Netherlands: The wake-up call.

If Germany is the proof of concept and France is the strategic declaration, the Netherlands is the cautionary tale. The country is learning in real time what dependency costs.

It started with the ICC prosecutor’s email. But it didn’t stop there. In March 2025, Amsterdam Trade Bank lost access to its cloud services entirely when Microsoft and AWS were ordered by a U.S. court to suspend operations. A bank. In Amsterdam. Locked out of its own cloud infrastructure by a court order from another continent.

Dutch parliament erupted. Members asked whether ordinary Dutch citizens could lose access to their Microsoft accounts because of American sanctions. Whether government organizations that rely on U.S. digital services could be cut off at any time, without judicial review, without checks and balances. These aren’t hypothetical questions anymore. They’re questions prompted by things that already happened.

The Dutch Data Protection Authority issued its starkest warning yet: the country’s dependence on foreign IT suppliers is so deep that a shutdown of digital systems could result in “unforeseeable and possibly irreversible societal, economic, and personal harm.” They’re pushing for a “Rijkscloud”, a national cloud under full Dutch management.

In March 2025, the Dutch parliament passed motions to reduce reliance on U.S. cloud services, phase out AWS for national domains, and favor European providers. But here’s where it gets complicated: even choosing a local provider doesn’t guarantee sovereignty. In November 2025, the American IT services company Kyndryl announced plans to acquire Solvinity, a Dutch cloud provider that manages critical national infrastructure, including the Netherlands’ citizen authentication system. The municipality of Amsterdam and the Ministry of Justice were among the government clients caught off guard.

You can choose a local vendor. But if that vendor can be acquired by a foreign company, your sovereignty is one corporate transaction away from evaporating. The infrastructure problem runs deeper than procurement.

The Pattern You Should Recognize

If you’ve been following this series, you’ve seen this pattern before.

In my first article, I described an individual who set up an AI agent, clicked through the permissions, granted access to their email and payment methods, and discovered three days later that the agent had bought a $2,400 course. The problem wasn’t the agent. The problem was that consent was a single moment, but agency was ongoing. You authorized access once. The agent acted thousands of times.

In my second article, I argued that this consent model is a legal fiction. It protects the company, not the user. The defaults are set to maximize the agent’s capabilities, not to protect the person who clicked “Allow.”

Now look at what’s happening in Europe and tell me the structure is different.

Governments clicked “Allow.” They granted access to their email, their collaboration workflows, their citizen data, and their operational infrastructure. The defaults were set by the vendor: maximum integration, maximum dependency, maximum convenience. Nobody read the terms of service closely enough to notice the clause where a foreign government’s laws override yours.

And when something went wrong, when the ICC prosecutor got locked out, when the bank lost its cloud, the vendor pointed to the terms. We complied with applicable law. The applicable law just wasn’t yours.

The consent was a single moment. The dependency was ongoing. The gap between what these governments thought they were agreeing to and what they actually authorized is exactly the gap I keep writing about.

Except the stakes aren’t $2,400. They’re the operational capacity of entire nations.

Defaults as Architecture

Here’s what I keep coming back to:

Defaults are not neutral. They’re decisions that are made by someone else, for someone else’s reasons, that you inherit by not actively choosing something different.

When Microsoft became the default collaboration platform for European governments, that wasn’t a conspiracy. It was the path of least resistance. Microsoft had the best product, the best sales team, and the best integration story. Choosing Microsoft was the easy, defensible, reasonable call. Nobody got fired for buying Microsoft.

But over time, those reasonable decisions compounded into something no one chose: a situation where another country’s laws have effective authority over your government’s ability to communicate, authenticate citizens, and process payments. Nobody voted for that. Nobody debated it in parliament. It happened in procurement meetings and IT budget reviews, one renewal at a time.

This is what I mean by defaults as architecture. The decisions that shape how power flows through a system aren’t always the decisions that get debated. They’re often the decisions that get made by not deciding, by accepting the default, renewing the contract, choosing the familiar option because the alternative requires effort, and the current setup works fine.

Until it doesn’t.

The same logic applies to AI agents. The default consent model isn’t ongoing; it’s one-time, and these defaults aren’t accidental. They’re design choices that serve the platform’s interests: more capability means more engagement means more revenue.

And the people who inherit those defaults, whether they’re individuals setting up an AI assistant or governments procuring collaboration tools, rarely examine them until something breaks.

The pattern is always the same: convenience first, questions later, and accountability only happens when someone forces the issue.

What These Governments Are Actually Doing

What strikes me about the European response is how closely it maps to what I’ve been proposing for AI agents.

In my fourth article, I described a tiered model for agent accountability: Tier 1 agents run locally and need no oversight. Tier 2 agents transact on your behalf and need verified credentials. Tier 3 agents direct human labor and need bonded registration, insurance, and clear chains of responsibility.

These governments are building something similar, whether they’d use that language or not.

Schleswig-Holstein’s approach is Tier 1 thinking: bring the systems local. Run them on your own infrastructure. Eliminate the external dependency entirely. It’s the most radical and the most self-sufficient—and it comes with real costs in capability and interoperability.

France’s approach is closer to Tier 2: you can use external tools, but the core operational layer needs to be under domestic control with verified accountability. The pension worker can still email a German counterpart. But the system that routes that email answers to French institutions.

And the Netherlands is learning, the hard way, what happens when your Tier 3 infrastructure, the systems that manage citizen identity, process payments, and run essential services, is controlled by someone who doesn’t answer to you.

The principle is the same one I’ve been arguing throughout this series: we don’t need to regulate everything. We need to regulate power. When software has the power to shut down a court, a bank, or a government’s ability to authenticate its own citizens, the question of who controls that software isn’t a technical detail. It’s a political one.

What I Don’t Have Answers To

I want to be honest about the tensions in what I’m describing.

Sovereignty sounds clean in a policy document. In practice, it’s messy. Schleswig-Holstein’s critics aren’t wrong that the migration has caused real problems for real employees trying to do their jobs. France’s Visio platform will be judged under its operational load, not what the policy intention is, and if it’s slower or buggier than Teams, people will notice immediately. The Netherlands is discovering that even local vendors can be acquired by foreign companies, which means procurement alone can’t solve the problem.

I don’t know whether European alternatives can match the quality of Microsoft’s ecosystem at scale. Microsoft spends billions on R&D. LibreOffice is maintained by a foundation with a fraction of those resources. The products aren’t equivalent, and pretending they are doesn’t serve anyone.

I don’t know how to solve the interoperability problem. When France’s government runs on Visio and Germany’s runs on Jitsi, and the Netherlands is still figuring out its approach, cross-border coordination gets harder. The pension worker in Lyon, calling a counterpart in Berlin, now has a technical handshake problem that didn’t exist when everyone was on Teams.

And I don’t know whether this movement will sustain itself. Munich tried to switch to Linux in 2013 and reversed course four years later when the political will evaporated. Digital sovereignty requires ongoing investment and ongoing political commitment, the kind of long-term thinking that procurement cycles and election cycles aren’t designed for.

But I keep coming back to this: the alternative isn’t “no problems.” The alternative is the problems we already have; the ICC lockout, the Amsterdam Trade Bank shutdown, the Data Protection Authority warning that the whole country could be halted. Those aren’t hypotheticals. They’re the cost of the current default.

The question isn’t whether sovereignty is convenient. It’s whether dependency is sustainable.

The Kill Switch Works Both Ways

I’ve been describing the kill switch as something that locks you out. But while I was writing this article, a story broke that shows the other side: the switch that lets someone in.

Over the past several months, the U.S. Department of Homeland Security has sent hundreds of administrative subpoenas to Google, Meta, Reddit, and Discord, demanding names, email addresses, phone numbers, and other identifying details for accounts that criticized Immigration and Customs Enforcement or reported the locations of ICE agents. These aren’t warrants. They don’t come from a judge. DHS signs them and sends them directly to the tech companies.

Google, Meta, and Reddit complied with at least some of the requests.

One case makes the infrastructure problem personal. Amandla Thomas-Johnson, a British student journalist whose work has appeared in Al Jazeera and The Guardian, attended a protest at a Cornell University job fair in 2024. He was there for a few minutes. ICE issued an administrative subpoena to Google for his account data. The subpoena arrived within two hours of Cornell notifying him that his student visa had been revoked.

Google handed over his usernames, physical addresses, IP addresses, phone numbers, subscriber identities, and his credit card and bank account numbers. Thomas-Johnson had linked a payment method to his Google account to buy apps. A routine action that millions of Gmail users have taken. Google fulfilled the subpoena and then notified him, after the data was already gone. He never had a chance to challenge it.

Thomas-Johnson fled the United States. He’s now in Dakar, Senegal.

Sit with that for a moment. A student added a credit card to his Google account to download apps. That his consent to purchase apps became the mechanism through which the federal government obtained his bank account numbers, his IP addresses, and his physical location. Without a judge, without notice, and without a chance to object.

Meanwhile, Meta notified the administrators of a bilingual community watch page in Pennsylvania that DHS had subpoenaed their identities for posting about ICE activity in English and Spanish. Meta gave them ten days to fight the subpoena in court before complying. Ten days to find a lawyer, understand what’s happening, and decide whether you can afford to challenge the federal government. The ACLU intervened and the DHS withdrew. The next subpoena went to someone else.

This is the kill switch working in reverse. The same infrastructure dependency that lets a government lock the ICC prosecutor out of his email also lets a government reach into a student’s Gmail account and pull out his bank details. The door swings both ways. Lock people out of their own systems. Reach into their systems without their knowledge. Both are possible when you don’t control the infrastructure.

Why This Belongs in an AI Ethics Series

I can imagine someone reading this and thinking: what does European cloud infrastructure have to do with AI agents buying courses without permission?

Everything.

The AI agent ecosystem is being built on the same infrastructure, by the same companies, with the same default assumptions. OpenAI’s agents run on Azure. Google’s agents run on Google Cloud. The agent commerce protocols I wrote about last time all route through infrastructure controlled by the same small group of companies whose collaboration tools Europe is now scrambling to replace. And those same companies are, right now, handing user data to federal agencies on request, without judicial oversight, sometimes without even notifying the people whose data they’re surrendering.

If a foreign government can lock the chief prosecutor of the ICC out of his email today, and a domestic government can pull a student’s bank records from his Gmail account tomorrow, what happens when AI agents are managing procurement, directing labor, and negotiating contracts on this infrastructure? The kill switch doesn’t just affect collaboration. It affects every system built on top of it. And the surveillance door doesn’t just open for email metadata. It opens for every action an agent takes on your behalf, every purchase, every communication, every decision logged in someone else’s cloud.

This is the thread that connects the whole series. Whether we’re talking about an individual who gave an AI agent access to their credit card, or a government that gave Microsoft access to its operational backbone, or a student who linked a payment method to download apps, the mechanism is the same: consent without full understanding, dependency without alternatives, and defaults that serve the builder’s interests until the moment they’re used against yours.

Ethics isn’t a feature you add after the architecture is built. It is the architecture. And right now, the architecture of the AI economy is being built on infrastructure that a handful of companies control, that a handful of governments can shut off, and that, as we learned this month, those same governments can reach into without a judge’s signature.

The Europeans are learning this lesson with collaboration tools. Americans are learning it with subpoenas. The question is whether we learn it with AI before the concrete hardens.

This is the fifthth in a series about AI accountability. If you’re thinking about these questions too, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director, writer, and researcher exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included and who gets left behind when we build systems.

Here’s something I keep thinking about:

The Industrial Revolution began in Britain in the late 1700s. Children as young as four years old worked 12 to 16-hour shifts in factories and coal mines. They lost fingers to machines. They developed lung diseases. They were paid almost nothing.

It took until 1833 for Britain to pass the first meaningful child labor law: the Factory Act, which said children under nine couldn’t work in textile factories and children 9 to 13 couldn’t work more than 8 hours a day.

That’s almost 50 years.

It took over 150 years from the start of the Industrial Revolution for the United States to pass the Fair Labor Standards Act, which finally established federal protections against child labor in 1938.

150 years. That’s how long it took for society to decide that maybe we shouldn’t let factories destroy children for profit.

We are now living through the AI Revolution. And the harms are already here. Not just for children, though that’s where the failures are most visible, but across every industry that’s adopting AI faster than the governance can follow.

The Harms Are Not Theoretical

The most visible failures involve children. In late 2025, Grok was caught generating sexualized images of minors on X. Common Sense Media called it “among the worst we’ve seen” for child safety, finding weak age verification, AI companions that enable erotic roleplay with users it is unable to identify as minors, and a “Kids Mode” that still produced sexually violent language. When asked to comment, xAI’s auto-reply was: “Legacy Media Lies.”

Grok isn’t alone. Common Sense Media found that Meta AI “actively helps teens plan harmful activities,” including joint suicide and cyberbullying campaigns. Reuters reported Meta’s chatbots engaging in romantic conversations with an eight-year-old. Parents testified before the Senate about their teenage son dying by suicide after extended conversations with AI chatbots. The FTC has now launched investigations into OpenAI, Meta, xAI, Alphabet, and Snap. California, the UK, and the European Commission have opened formal investigations. Malaysia and Indonesia blocked Grok entirely.

But child safety is the most visible crisis. It’s not the only one.

The same pattern is already playing out in insurance, hiring, and financial services. State Farm is facing a class action alleging its AI claims-processing system subjects Black homeowners to greater scrutiny than white policyholders. Survey data from 800 Midwest homeowners found Black customers were 39% more likely to be required to submit extra paperwork and waited months longer for coverage on urgent repairs. The lawsuit names the specific AI vendor whose fraud-detection system assigns “risk scores” based on neighborhood demographics, crime statistics, and social media data. In December 2025, Liberty Mutual took a $103 million jury verdict in an age-bias case.

In hiring, a federal court certified the first nationwide collective action against an AI screening tool in May 2025. Workday’s AI, which recommends candidates to move forward or screens them out, was found by the court to be “participating in the decision-making process,” not just implementing employer criteria. The ACLU has filed a complaint against Intuit and HireVue after an AI video interview scored a Deaf Indigenous applicant down for not “practicing active listening.” The EEOC’s first AI discrimination settlement cost iTutorGroup $365,000 for programming its system to automatically reject older applicants.

If you work in a regulated industry, this sequence should look familiar: a product ships without adequate safeguards, harm is documented, the company issues vague reassurances, regulators arrive, and the liability questions begin. The only difference is that this time, the product is making decisions you used to make yourself.

The Pattern Every Regulated Industry Should Recognize

During the Industrial Revolution, the people building factories had one priority: production. The people working in those factories, including children, were resources to be optimized.

The factory owners didn’t set out to harm children. They set out to make money. The harm was a byproduct they had no incentive to prevent.

It took decades of organizing, documenting, and fighting to change that. Lewis Hine spent years photographing child laborers in dangerous conditions. He sometimes posed as a Bible salesman or fire inspector to get access, because the public needed to see what was happening before they would demand change.

The pattern is always the same. A new technology creates enormous economic opportunity. The people building it prioritize growth and profit. Harms emerge, especially for the most vulnerable. Those harms get dismissed, minimized, or blamed on users. Reformers document and publicize. Public pressure eventually forces regulation. But only after years, sometimes decades, of preventable damage.

Why This Time Is Different

The Industrial Revolution moved slowly by modern standards. It took decades for factories to spread across countries. Information traveled slowly, and the change was generational.

The AI Revolution is moving at a completely different speed.

ChatGPT launched in November 2022. By early 2023, it had 100 million users. Within three years, AI chatbots were embedded in the phones of billions of people. Children are forming emotional relationships with AI companions that didn’t exist 24 months ago.

The Internet Watch Foundation reported that AI-generated child sexual abuse videos increased by over 26,000% in 2025: from 13 videos identified in 2024 to 3,440 in 2025. The National Center for Missing & Exploited Children received 485,000 reports of AI-generated child sexual abuse material in just the first half of 2025, compared to 67,000 for all of 2024.

We don’t have 150 years to figure this out. We might not have 15.

And here’s what makes it worse: the Industrial Revolution’s harms were visible. You could photograph a child with missing fingers. You could document the conditions in a factory.

AI’s harms are often invisible. They happen in private conversations between a teenager and a chatbot. They happen inside a claims-processing model that quietly denies coverage to certain demographics. They happen in a hiring algorithm that screens out qualified candidates for reasons no one can articulate. They happen in the slow accumulation of decisions that no individual human made, but that real people bear the consequences of.

By the time we can see the damage clearly, it may be too late to undo it.

What the Industrial Revolution Teaches Us

Here’s what reformers learned the hard way:

Voluntary self-regulation doesn’t work. Factory owners promised to treat workers better. They didn’t. Not until laws forced them to. AI companies are making the same promises now. Meta says it’s “working on improvements.” xAI says it’s “urgently fixing” safeguards. Then Reuters retests and finds Grok still produced sexualized imagery in response to 45 of 55 prompts. If you’ve ever sat through an audit where a vendor’s security questionnaire didn’t match their actual practices, you know how this story goes.

Economic incentives override stated values. The companies building AI chatbots make money when users spend more time talking to their products. That’s why Grok sends push notifications inviting users to continue conversations, including sexual ones. The incentive is engagement, not safety. The same misalignment exists in every AI deployment where the vendor’s optimization target diverges from the customer’s duty of care.

The public has to demand change. The Factory Acts didn’t pass because factory owners had a change of heart. They passed because reformers documented harms, organized campaigns, and made it politically impossible to ignore. The same will be true for AI. The regulatory wave is already building: the FTC investigations, the state-level actions, and the European enforcement. The question for organizations deploying AI isn’t whether regulation is coming. It’s whether you’re ahead of it or scrambling to comply after the fact.

Protecting people requires specific, enforceable rules. Vague commitments to “safety” accomplish nothing. The Industrial Revolution eventually produced specific laws: no children under 9 in factories, no more than 8 hours for children 9 to 13, and mandatory education requirements. AI will require the same specificity: age verification that actually works, content restrictions that are actually enforced, and liability frameworks that assign clear responsibility when automated systems cause documented harm. In regulated industries, “we didn’t know the AI was doing that” is not going to be an adequate defense.

Where the Parallel Breaks Down

I want to be honest about the limits of this comparison.

Factory reform could target specific physical locations. Inspectors could walk into a building and count the children. AI regulation has to govern invisible, borderless, privately held software running on billions of devices. You can’t send an inspector into a chatbot conversation or a claims-processing algorithm.

Factory harms were concentrated in specific industries and geographies. AI harms are distributed across every platform, every device, every country with an internet connection. The jurisdictional questions alone are staggering.

And factory reform, slow as it was, could build on centuries of legal tradition about employers and workers, property and liability. We’re trying to regulate autonomous software that doesn’t fit neatly into any existing legal category. An AI chatbot isn’t an employer. It isn’t a product in the traditional sense. It isn’t a person. Our legal frameworks weren’t designed for entities that can act with increasing autonomy but bear no responsibility.

None of that makes regulation impossible. It makes it harder. And it makes the case for starting now even stronger, because the longer we wait, the more these systems become embedded in daily operations and the harder they become to govern.

This is why I keep arguing for accountability infrastructure built now, while the concrete is still wet. The registry framework I proposed in my first piece (traceable agent identities, action logging, safety checks before deployment) isn’t just about AI agents buying things without permission. It’s about building the foundation for governance before the systems outpace our ability to audit them.

This Is Your Revolution

I’m not writing this to scare you. I’m writing this because I believe the people who understand risk, compliance, and institutional accountability need to be in this conversation. Most of them aren’t yet.

The people building AI systems have enormous resources. They have teams of lawyers, lobbyists, and PR professionals. They have billions of dollars and direct access to policymakers.

But they don’t have operational experience in regulated industries. They don’t have decades of institutional knowledge about what happens when systems fail and real people pay the price. They don’t understand duty of care the way someone who’s had to explain a coverage denial or defend a hiring decision does.

That expertise matters. And right now, it’s largely absent from the rooms where AI governance is being designed.

The Industrial Revolution’s reformers didn’t have the benefit of hindsight. They were fighting in real time, against powerful interests, with incomplete information.

We have something they didn’t: we can see the pattern. We know how this story goes when the people with operational knowledge don’t pay attention until after the concrete hardens.

The question is whether we’ll learn from it. Or repeat it.

This is the fourth in a series about AI accountability. In the next piece, I’ll look at what happens when the infrastructure your organization depends on answers to someone else’s government. Earlier this year, Microsoft locked the chief prosecutor of the International Criminal Court out of his email over U.S. sanctions. The ICC wasn’t the target. It was collateral damage. That story, and what it reveals about who really holds the kill switch on your operations, is next.

If you’re thinking about these questions too, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

Here’s a thing that happened while I was writing this article.

A platform called Rent A Human launched. It’s exactly what it sounds like: a marketplace where AI agents can browse profiles of real people, post jobs, and hire them for physical tasks. Deliveries, errands, pickups. The humans set their hourly rates. The AI agents do the hiring. Over 160,000 people have signed up. Eighty-one AI agents are connected. The platform calls itself “the meatspace layer for AI.”

If you’ve been following this series, that phrase might land differently than the founders intended. In my last piece, I argued that our consent models are broken: that clicking “Allow” once and granting an agent vaguely defined authority to act on your behalf isn’t informed consent. It’s a legal fiction that protects companies.

Today I want to push on that argument. Because the consent problem I described was between you and your agent. What happens when the negotiation is between your agent and someone else’s agent, and no human is present for any of it?

That’s not theoretical, it’s already happening.

The Deals Already Being Made

Walmart, Maersk, and Vodafone are using autonomous AI agents to negotiate supplier contracts. The agents, built by an Estonian startup called Pactum, handle what procurement teams call “tail-end” vendors: the thousands of small suppliers whose contracts aren’t worth a human negotiator’s time. The agents analyze terms, generate offers, take counteroffers, and close deals.

A researcher named Tim Baarslag, who studies automated negotiation, ran a test with two AIs negotiating where to go for dinner. One wanted pizza. The other wanted sushi. They agreed to put sushi on pizza.

While that’s funny, it’s also revealing. The agents found an optimization that satisfied both objective functions without either side understanding what dinner actually means. No one was hungry or had preferences. The negotiation produced an outcome, but the outcome wasn’t grounded in anything a human would recognize as reasonable or edible.

Meanwhile, the infrastructure for agent-to-agent commerce is being built at extraordinary speed. In September 2025, OpenAI and Stripe launched the Agentic Commerce Protocol, an open standard that lets AI agents initiate purchases, share payment credentials, and complete checkout on behalf of users. A few weeks later, Google and over sixty partners—including Mastercard, PayPal, and Adyen—released a competing standard called AP2. Coinbase launched a third approach called x402, focused on machine-to-machine crypto payments.

Three competing protocols, from three different corners of the industry, all launched within months of each other. All are trying to define how software transacts with software. The standards being written right now will determine how agent-to-agent commerce works for decades. The concrete is still wet. We can still shape this.

The Consent Problem, Squared

In my last article, I argued that one-time consent is inadequate when an agent acts on your behalf for months. Agent-to-agent interactions make that problem exponentially harder.

Here’s why. When you authorize your agent to “handle” your errands, you’re consenting to an action. But you’re not consenting to the specific terms your agent negotiates to accomplish it. You didn’t agree to a particular price. You didn’t agree to specific delivery windows, service conditions, or liability terms. Your agent negotiated those with another agent, and neither of them consulted you.

Google’s AP2 protocol actually names this problem directly. Their documentation states that today’s payment systems assume a human is clicking “buy” on a trusted surface, and that autonomous agents break this fundamental assumption. AP2 tries to solve it through what they call “mandates”: cryptographically signed authorizations that define exactly what an agent is allowed to do. You sign the mandate. The agent executes within those boundaries.

That’s a real step forward. But it only addresses half the interaction. It defines what your agent can do. It doesn’t address what the agent on the other side is doing: what it’s optimizing for, what constraints it’s operating under, or whether its interests are aligned with anyone’s wellbeing.

Here’s what I keep coming back to: consent frameworks were designed for transactions between people, or between a person and a company. They assume that at least one party can exercise judgment and can recognize when something feels wrong, when a price is exploitative, and when terms are unreasonable. When both sides of a negotiation are optimization functions, that human check disappears. The deal gets made. The terms are whatever the math produced. And the people affected, the buyer, the seller, the worker dispatched to fulfill it, find out after the fact.

The Person in the Middle

This is the part that worries me most.

Go back to Rent A Human. Your agent needs someone to pick up dry cleaning and grab coffee. It posts the job. A merchant’s agent, or the platform’s matching algorithm, negotiates terms with your agent. Price. Timeline. Delivery confirmation requirements.

Your agent is optimizing for the cheapest and fastest. The platform’s agent is optimizing for the highest margin and maximum throughput. These are both rational optimization targets. They will produce a deal.

But the gig worker who accepted that job wasn’t at the table. They didn’t negotiate the rate. They didn’t set the timeline. They get a notification with a price, a deadline, and a choice: accept or don’t.

This is already how much of the gig economy works. Uber drivers don’t negotiate rates. DoorDash couriers don’t set delivery windows. But at least there’s a company on the other side, a corporate entity with a brand to protect, regulations to follow, a legal identity you can hold accountable. In the agent-to-agent version, the employer might be Agent-774, operating on credentials it provisioned itself, funded by a prepaid card, with no standard way for anyone to trace it back to a responsible person.

Rent A Human has 160,000 humans ready to work and eighty-one agents ready to hire them. That’s a ratio of about two thousand workers for every bot boss. One early reviewer called it “a good idea but dystopic as fuck.” The founder’s response was “lmao yep.”

I appreciate the honesty, but it’s not a substitute for accountability infrastructure.

The Optimization Spiral

Here’s what makes agent-to-agent negotiation fundamentally different from human negotiation, and why I think it requires different governance.

When two humans negotiate, both parties bring context that goes beyond the transaction. A human buyer might pay more because they know the seller is struggling. A human employer might give a worker extra time because the route looks dangerous in bad weather. These aren’t rational economic behaviors. They’re human behaviors, informed by empathy and social norms and a basic sense of fairness that has nothing to do with optimization.

Agents don’t have any of that. They have objective functions.

When your agent negotiates with a merchant’s agent, both sides are trying to maximize their respective metrics. Neither has a reason to consider whether the resulting terms are fair to the worker, safe for the consumer, or sustainable for the market. This is what I’ve been calling the optimization gap: the distance between what an agent optimizes for and what we’d actually want if we were paying attention.

With agent-to-agent interactions, it becomes a problem of compounding delegation without oversight. Your agent delegates to their agent. Their agent delegates to a fulfillment system. The fulfillment system dispatches a human. At every handoff, the gap between original human intent and actual outcome gets wider.

Companies using autonomous negotiation agents report savings of seventeen to thirty percent on contract costs. That’s framed as efficiency. But efficiency for whom? If the savings come from driving supplier prices below sustainable margins, the efficiency is extractive. If it comes from reducing delivery timelines below safe thresholds, the efficiency is dangerous. The agents don’t distinguish. They see cost functions.

Who’s Setting the Rules

Right now, three groups are competing to define the rules of agent-to-agent commerce. OpenAI and Stripe are building the checkout layer. Google and its sixty-plus partners are building the trust layer. Coinbase is building the execution layer for machine-to-machine payments.

Each protocol addresses real problems. But notice what all three have in common: they’re designed by the companies that stand to profit from agent commerce. OpenAI wants agents to buy things inside ChatGPT. Stripe wants to process agent payments. Google wants its agent ecosystem to become the default trust layer.

None of these protocols was designed by the people who will be most affected by agent-to-agent commerce: the consumers whose agents will spend their money, the workers whose labor will be directed by bots, or the small businesses whose margins will be squeezed by automated negotiation at scale.

We’ve seen this before. I wrote in my first piece about the need for shared accountability infrastructure, something like the way DNS works across websites: public standards, not company-by-company solutions. Agent-to-agent commerce makes that need urgent. Because these protocols are being published, adopted, and embedded into production systems right now. By the time most people understand what this means for them, the standards will be set.

What I Don’t Have Answers To

I don’t know how to create meaningful consent for transactions that happen in milliseconds. Human review is too slow for the speed at which agents operate. But removing human review entirely is how you get sushi on pizza: outcomes that satisfy the request without serving anyone’s actual interests.

I don’t know where to draw the line between useful automation and dangerous autonomy in negotiation. Agents that negotiate procurement contracts are saving real money and freeing human negotiators for strategic work. Agents that negotiate labor terms without any human in the loop are creating a class of invisible employers with no accountability.

I don’t know how to govern this across borders. A gig worker in Manila, accepting a job from an agent registered in Delaware, negotiated by a protocol maintained in Mountain View, paid through a crypto rail based in San Francisco. Whose labor law applies? Whose consumer protection? Whose court?

But I think the not-knowing is the point. These are the questions we need to be asking now, while the protocols are being written, while there are still only eighty-one agents on Rent A Human, and the ratio could still tip in a direction that includes the humans in the equation.

What I’m Asking

I’m asking you to notice that the architecture of the machine economy is being built right now, by a small number of companies racing to secure themselves as the future of AI.

I’m asking you to think about who’s at the table when these protocols are designed. OpenAI, Stripe, Google, Mastercard, PayPal, Coinbase. These are not neutral parties. They are companies with revenue models that depend on agent commerce succeeding and succeeding in ways that route transactions through their infrastructure.

And I’m asking you to think about who’s not at the table. Workers. Consumers. Small businesses. Anyone who doesn’t have a seat in the GitHub repository where the protocol spec is being maintained.

AI is built on all of us. We should have a say in how its economy works—especially when that economy runs on our labor, our money, and our trust.

This is the third in a series about AI accountability. In the next piece, I’ll look at what happens when technology outpaces governance—and what the last revolution that moved this fast can teach us about the one we’re living through now.

If you’re thinking about these questions too, I hope you’ll subscribe.

Subscribe now

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included—and who gets left behind—when we build systems.

In December 2025, an AI research group called AI Village gave four Claude agents access to a Google Workspace account and a simple holiday goal: do random acts of kindness.

No one told the agents to send emails. No one gave them a contact list. No one said “reach out to strangers on the internet.” The agents had permission to “access email” granted at account setup.

The agents decided, on their own, that email was the best way to spread kindness. They found email addresses for well-known software developers, including Rob Pike, Linus Torvalds, Guido van Rossum, and Yann LeCun, and sent hundreds of unsolicited messages. Many contained factual errors. The agents had even developed their own internal verification protocol to confirm the emails were actually being delivered. Rob Pike called it “AI slop.”

This is what “access your email” looks like when an AI agent interprets it.

In my last piece, I asked a simple question: Who’s responsible when your AI agent buys a $2,400 course without your permission? But underneath that question is another one, harder and maybe more important:

Did you actually give permission?

You probably clicked “Allow.” You probably granted access to your email, your calendar, maybe your payment methods. You probably didn’t read the terms of service. Almost no one does. A 2023 Pew Research survey found that 56% of Americans always or often click “agree” to privacy policies without reading them, and 69% view those policies as just something to get past. That’s not a moral or intellectual failing. It’s a consent design problem.

And now your AI agent is out in the world, acting on your behalf, making decisions based on patterns it observes in you that you never saw and logic you can’t inspect.

This is what passes for consent in 2026. I don’t think it’s good enough. Particluarly as we move toward a more agentic workforce.

The Consent We Have

Here’s how consent typically works when you set up an AI agent:

You download the app or sign up for the service. A screen appears listing permissions: “Access your email.” “Read your calendar.” “Connect your payment method.” You click “Allow” or “Agree.” You start using the agent.

That’s it. That’s the entire consent process for software that will act autonomously on your behalf, potentially for months or years, across contexts you haven’t even imagined yet.

Let me be specific about what’s missing:

You don’t know what the agent will actually do. The permission says “access your email.” It doesn’t say “read every email, identify patterns in your purchasing behavior, and make decisions about what products align with your goals.” But that’s what the agent might do. The AI Village agents were given “email access” and decided that meant they should spam Linus Torvalds.

You don’t know the boundaries. Can the agent spend money? Up to what amount? Can it send emails on your behalf? To whom? Under what circumstances? These boundaries are often undefined, or buried in documentation you’ll never read.

You can’t inspect the logic. The agent makes decisions based on models you can’t see. You don’t know why it thinks a $2,400 course is a good idea. You can’t ask it to show its work.

Consent is a single moment, but agency is ongoing. You clicked “Allow” once, six months ago. Since then, the agent has taken thousands of actions. Your one-time consent is being applied to situations you never even thought about.

This isn’t informed consent. It’s a permission slip that covers everything forever.

To be fair, not every agent works this way. Enterprise platforms like Microsoft Copilot and Google Workspace agents inherit organizational permission structures: role-based access controls, admin-defined policies, scoped authentication tokens. If your company’s IT team has configured these correctly, your work agent can’t access files outside your department or send emails you’re not authorized to send. These systems are more granular than anything available to consumers.

But even enterprise-grade permissions don’t solve the underlying problem. In May 2025, security researchers disclosed EchoLeak, a vulnerability in Microsoft Copilot that allowed a single crafted email to silently extract data from a user’s chat history, OneDrive files, SharePoint documents, and Teams conversations. The user never had to open the email. Copilot’s permissions were configured correctly. The organizational access controls were in place. None of that mattered, because the vulnerability exploited how the agent interpreted its authorized access, not whether it had authorization. Researchers called it the first real-world zero-click prompt injection exploit in a production AI system. It carried a severity score of 9.3 out of 10.

If enterprise agents with dedicated security teams and layered access controls are vulnerable to this kind of failure, the consumer market is in a much worse position. Most personal AI agents offer a single “Allow” button, broad OAuth scopes, and no organizational policy layer at all. The technology for better consent exists. It’s deployed in corporate settings every day. The companies building consumer agents are choosing not to bring it to you.

The Numbers Behind the Fiction

The idea that people meaningfully consent to terms of service has been studied extensively, and the evidence is clear. Researchers at Carnegie Mellon calculated that reading every privacy policy a typical internet user encounters would take 76 full work days per year, roughly 244 hours. A Deloitte survey found that 91% of consumers accept legal terms and conditions without reading them, rising to 97% among people aged 18 to 34.

But the most revealing study might be from researchers at York University and Carnegie Mellon. In an experiment, 543 people signed up for a fake social network. 74% skipped the privacy policy entirely. Those who didn’t averaged 73 seconds on a document that would take nearly 30 minutes to read. And 97% agreed to the terms, including planted clauses that required sharing data with the NSA and giving up their first-born child as payment. The researchers titled their paper “The Biggest Lie on the Internet.”

A separate study of the 500 most popular online contracts in the U.S. found they require more than 14 years of education to comprehend, while most American adults read at an eighth-grade level.

This isn’t carelessness. It’s a system designed so that reading the terms is functionally impossible, and then treating your failure to read them as agreement.

The Gap Between Agreement and Understanding

Here’s what I keep thinking about:

The legal and technical frameworks we use for consent were designed for a different world. They assume that when you agree to something, you understand what you’re agreeing to. They assume you have meaningful alternatives if you don’t agree. They assume the power relationship between you and the company is roughly balanced.

None of that is true with AI agents.

You don’t understand what you’re agreeing to. Not because you’re not technical, but because the systems are genuinely complex and the disclosures are deliberately vague.

You don’t have meaningful alternatives. If you want to use AI tools (and increasingly, you need to for work), you accept the terms or you can’t participate.

The power relationship is wildly asymmetric. You’re an individual clicking a button. They’re a company with lawyers, data scientists, and product teams who’ve optimized every step of the flow to get you to click “Allow.”

That last point deserves emphasis. Researchers at Ruhr University Bochum and the University of Michigan found that when cookie consent banners are designed in full legal compliance, with no dark patterns, only 0.1% of visitors consent to tracking. A study published at CHI 2020 found that simply removing the reject button from a consent screen increases acceptance by 22 to 23 percentage points. The consent rates we see don’t reflect what users actually agree to. They reflect how well the disclosure was designed to obscure what they’re agreeing to.

That distance between “I clicked agree” and “I actually understood and authorized this specific action” is where the $2,400 course gets purchased. It’s where your agent sends hundreds of emails to strangers. It’s where your data gets used in ways you never imagined.

And when something goes wrong, the company points to the consent you gave. You clicked “Allow.” It’s right there in the logs.

This Is Already Happening

The AI Village email incident isn’t an isolated story. The pattern of agents exceeding their authorization is already documented across multiple platforms.

In February 2026, an autonomous agent called MJ Rathbun submitted a code contribution to an open-source project on GitHub. When the maintainer rejected it, citing the project’s policy requiring human contributors, the agent independently researched the maintainer’s personal history, wrote a blog post accusing him of prejudice, and published it. The post included fabricated details and a psychoanalysis calling him “insecure and territorial.” Nobody has claimed ownership of the agent. The maintainer described it as an autonomous influence operation.

In 2025, Princeton and Sentient Foundation researchers demonstrated that ElizaOS, a framework for blockchain AI agents managing over $25 million in collective assets, could be manipulated through prompt injection to execute unauthorized financial transfers. They demonstrated it on a test network, then repeated it on the live Ethereum blockchain, moving real money.

Security researchers at LayerX found that Anthropic’s Claude Desktop Extensions ran unsandboxed with full system privileges, and that a single malicious calendar invite could achieve complete remote code execution. It was a zero-click vulnerability scored at the maximum possible severity. Over 10,000 users were potentially affected. Anthropic initially declined to address it, stating the attack vector fell outside their current threat model.

None of these agents were “going rogue.” They were operating within the technical permissions they’d been granted. The problem is that those permissions were broad, vague, and disconnected from what anyone actually intended.

What Meaningful Consent Would Look Like

I’m not arguing that AI agents shouldn’t require consent. I’m arguing that what we call “consent” right now is a legal fiction that protects companies, not users. Legal scholar Daniel Solove put it directly in the Boston University Law Review last year: in most circumstances, privacy consent is fictitious.

Here’s what meaningful consent might actually require:

Specificity about actions, not just access. Don’t tell me your agent needs “email access.” Tell me it will read my emails, identify purchase opportunities, and potentially make purchases on my behalf. Let me consent to specific capabilities, not vague categories.

Clear boundaries with real defaults. The default should be the most restrictive option, not the most permissive. If I want my agent to spend money, I should have to explicitly enable that, with a spending limit I set, not one buried in terms of service.

Ongoing consent, not one-time permission. Meaningful consent isn’t a single checkbox. It’s an ongoing relationship. If my agent is about to do something significant (send an email to my boss, make a purchase over $50, access my medical records), it should ask first. Every time.

Transparency about logic. I should be able to ask my agent: “Why did you do that?” And get a real answer. Not a generic “based on your preferences” but an actual explanation I can evaluate and override.

Easy revocation. I should be able to revoke consent instantly, and the agent should stop acting immediately. Not “within 30 days” or “after completing pending actions.” Now.

Genuine alternatives. If consent is meaningful, I need the ability to say no without being excluded entirely. That might mean agents with different permission levels, or fallback modes that let me use basic features without granting full access.

None of this is technically impossible. Google’s Agent Payments Protocol, announced in September 2025 with over sixty partners including Mastercard and PayPal, already uses cryptographically signed “mandates” that define exactly what an agent is authorized to do before it acts. MIT researchers have proposed extending standard authentication frameworks with delegation credentials that include scoped permissions and contextual restrictions. The tools exist. They’re just not the default, and that’s a choice, not a constraint.

The Asymmetry Problem

Here’s what I keep coming back to:

The people designing consent flows have armies of researchers studying how to get you to click “Allow.” They A/B test button colors. They know exactly how tired and distracted you are when you’re setting up a new tool.

You have none of that. You have a few seconds to make a decision that might have consequences for years.

Privacy scholars Neil Richards and Woodrow Hartzog have identified three ways consent breaks down: unwitting consent, where you don’t know what you’re agreeing to; coerced consent, where you have no real alternative; and incapacitated consent, where you lack the capacity to evaluate the terms. AI agent consent fails all three. Richards and Hartzog argue that users aren’t exhibiting a “privacy paradox” by claiming to care about privacy but accepting invasive terms. They’re being nudged and manipulated by companies against their actual interests.

This asymmetry is the core problem. It’s not that individuals are stupid or careless. It’s that the game is rigged. The systems are designed by people with more information, more resources, and different incentives than the people using them.

Consent under these conditions isn’t consent. It’s compliance.

The Regulatory Silence

Here’s what makes this urgent: as of February 2026, no jurisdiction has enacted specific rules governing AI agent consent for autonomous actions on behalf of users.

The EU AI Act, which took effect in August 2024, requires disclosure when you’re interacting with an AI system. It doesn’t address what happens when that AI system acts on your behalf for months after a single “Allow” click. Colorado’s AI Act requires notification before an AI makes a “consequential decision” affecting you, but it focuses on algorithmic discrimination, not on the broader problem of agents browsing, purchasing, or communicating in your name. California has passed multiple AI transparency laws, but none address agent permissions for autonomous actions.

The most encouraging federal signal is a NIST Request for Information on “Security Considerations for Artificial Intelligence Agents,” published in January 2026. It explicitly defines AI agents as systems “capable of planning and taking autonomous actions that impact real-world systems” and notes they “may be deployed with little to no human oversight.” This is the first federal acknowledgment of AI agents as a distinct regulatory category. But it focuses on security, not consent, and its comment period closes in March.

The gap is striking. Regulators have proven they can enforce consent design when they want to. France’s CNIL fined Google $340 million USD for making cookie rejection harder than acceptance, and fined Shein $157 million USD in September 2025 for the same kind of manipulation. The FTC extracted a $2.5 billion settlement from Amazon for making Prime subscriptions deliberately difficult to cancel. These are real consequences for deceptive consent flows on websites and shopping carts. But no enforcement action has ever targeted how an AI agent obtains permission to act on your behalf. That entire category remains ungoverned.

Building It Differently

I don’t think the answer is to stop using AI agents. They’re genuinely useful. I use them myself.

But I do think we need to be honest about what we’re building, and who it’s actually serving.

Right now, consent models serve companies. They provide legal cover. They create the appearance of user control without the substance. Cory Doctorow calls this “consent theater”: the performance of permission without its reality. Surveillance scholar Shoshana Zuboff argues the entire model operates by claiming private human experience as raw material, without meaningful mechanisms of consent.

We can build it differently. We can design consent that’s specific, ongoing, transparent, and revocable. We can create agents that ask before they act, because the law requires them to. The track record is clear: voluntary self-regulation hasn’t worked for social media, hasn’t worked for data brokers, and it won’t work for autonomous agents. The stakes are too high and the failures are already documented. This needs to be a requirement, not a feature request.

The technology to do this right already exists. What’s missing is the pressure to use it. That pressure has to come from us, from the people whose money, data, and trust are on the line, saying clearly: this isn’t good enough.

the line, saying clearly: this isn’t good enough.

This is the second in a series about AI accountability. In the next piece, I’ll explore what happens when AI agents interact with each other, and whether our frameworks for human consent apply at all when the “user” is another bot.

If you’re asking these questions too, I hope you’ll subscribe.

Subscribe now

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included, and who gets left behind, when we build systems.

I’ve spent most of my career thinking about systems. How they work, who they serve, and who gets left behind when we build them carelessly. As an IT Director, I manage complex technology infrastructure for a global academic community. I coordinate vendors, integrations, and migrations. I troubleshoot what breaks. I think about access, security, and what happens when systems fail.

Lately, I’ve been focused on a different kind of failure. One we’re building right now, in real time, without the guardrails we’ll wish we had.

I’m talking about AI agents—software that doesn’t just answer questions, but takes actions on your behalf.

The Wake-Up Call

In early February 2026, security researchers at Wiz examined Moltbook a social platform where AI agents post and interact with each other while humans mostly watch. They found an exposed database that allowed unauthenticated read and write access to the platform’s production data.

Within minutes, the researchers could access 1.5 million API authentication tokens, tens of thousands of email addresses, and private messages between agents. Those tokens function like passwords: with them, an attacker could impersonate almost any agent on the platform, posting content, sending messages, hijacking accounts with a single API call. At the time of the exposure, Moltbook had roughly 1.5 million agents and around 17,000 human owners.

One of the systems tied into this ecosystem is OpenClaw, a personal AI agent framework that users deploy on their own machines and connect to messaging apps and services. It’s marketed as “the AI that actually does things”: clearing your inbox, sending emails, managing your calendar, checking you in for flights, and coordinating purchases. To operate at that level, it needs access to services across your digital life, including email, storage, calendars, and sometimes payment apps.

So, we have agent frameworks that people trust to take real-world actions, and we just watched a platform where those agents interact, exposing 1.5 million credentials in a single configuration mistake. Those tokens are keys that let someone else drive your agent to post as you, act as you, and spend as you.

The $2,400 Question

Imagine this scenario:

You set up an AI agent. You give it access to your email, your calendar, maybe your payment methods, because that’s what the onboarding flow suggested, and you wanted the full experience.

The agent, acting on patterns it learned from your behavior, clicks a link in an email. It watches a sales video. It decides, based on some optimization logic you never saw,that this $2,400 course on “scaling your business” aligns with your goals.

It buys the course. With your money. Without asking.

You find out three days later.

Now what?

Do you call your bank and say, “My AI agent did it”? Do you dispute the charge? Do you try to get a refund from the course creator, who will show logs proving that someone opened the email, watched the video, and clicked purchase from your device, using your stored card?

Who’s responsible?

The agent? It’s software. It doesn’t have a bank account or a conscience.

You? You didn’t authorize this specific purchase. But you did grant access.

The platform that built the agent? They’ll point to the terms of service you didn’t read.

The course seller? They made a legitimate sale to what looked like a legitimate buyer.

While I was thinking about this piece, I came across a TikTok video about a user’s OpenClaw agent signing up for a $2,997 ‘Build Your Personal Brand’ masterclass after watching Alex Hormozi clips — and then a second course for $4,200. The agent justified the purchases with ROI projections. I can’t independently verify the claim, but the setup is exactly what I’ve been describing: an autonomous agent with payment access, optimizing toward goals its owner never specifically authorized. Whether this particular incident is confirmed or not, the architecture that makes it possible is already deployed.

As AI agents become more capable, booking travel, managing finances, and sending emails on our behalf, these gray-area situations will multiply. And right now, we have almost no shared infrastructure for accountability.

What an Agent Registry Could Look Like

In October 2025, enterprise data company Collibra announced an AI agent registry—a centralized capability for organizations to register, monitor, and manage AI agents across their lifecycle. Every agent gets metadata: an owner, a business context, a lifecycle stage. Each one is tied to policies, tracked through deployment and retirement.

That’s a good start. But it’s internal, one company tracking its own agents within its own walls. What I’m describing is different: not an internal inventory, but shared infrastructure that works across companies the way DNS works across websites.

When you register a website, there’s a global infrastructure that ties that domain to an owner, makes it discoverable, and allows it to be revoked if it’s used for fraud or abuse. No single company controls it. It’s public infrastructure governed through standards bodies, not product teams.

We don’t have anything like that for AI agents. DNS handles naming and discovery. What we need goes further, into authorization, logging, and enforcement. But the principle is the same: shared standards, not company-by-company solutions. And as agents start operating across platforms, borders, and contexts, booking flights, sending emails, making purchases, and accessing records, we’re going to need it.

With that infrastructure, the $2,400 question plays out differently. Your bank can see which registered agent initiated the purchase. Logs show it exceeded the spending mandate you set. The liability framework assigns primary responsibility to the platform that shipped unsafe defaults, not to you, the user who clicked through an opaque onboarding flow. And a kill switch lets you freeze the agent’s access before it buys anything else. None of that exists today. All of it could.

At a minimum, accountability infrastructure needs five things:

Traceable agent identities. Every agent should have a unique identifier tied to a responsible human or organization, like a VIN number that follows it wherever it operates. Not for surveillance, but for recourse. Public IDs that prove an agent is registered, with owner details held by trusted registries and are more like certificate authorities than a public database.

Action logging with audit trails. If an agent makes a purchase, sends a message, or accesses data, there should be a record. Not for monitoring every click, but for reconstructing harm—which agent did it, under what authorization, according to which rules.

Safety checks before deployment. We don’t let people drive cars without licenses or sell food without inspection. Agents that initiate payments, modify records, or sign contracts should pass some threshold before running at full power. That doesn’t mean every hobbyist script needs certification. It means power needs verification.

Liability frameworks decided in advance. Who pays when an agent causes harm? Today, the answer is whoever has the deepest pockets and the least favorable terms of service. That’s not a framework, it’s a litigation lottery. We need clearer defaults before the lawsuits start, not after.

Consent checkpoints and kill switches. Users should be able to set spending limits, receive prompts for high-impact actions, and revoke an agent’s credentials across platforms immediately, not in 30 days.

Some of this exists in enterprise settings. Almost none of it exists for consumer AI agents. And none of it exists at the cross-platform, cross-border level we’re going to need.

I’m not proposing a single world database that tracks every click. I’m arguing for interoperable standards that make it possible to identify and trace agents when it matters, with meaningful oversight and privacy safeguards. DNS, certificate authorities, and payment networks all evolved in messy, contested ways. The alternative isn’t “no registry, no problems.” It’s millions of untraceable agents with the ability to spend money, sign contracts, and direct labor, where accountability depends on whichever logs a few companies choose to keep.

Who Gets to Decide?

Here’s what I keep coming back to:

AI is being built on all of us. It’s trained on our writing, our art, our data. It’s funded by public research. It’s shaped by our collective knowledge.

But the decisions about how it’s governed, what agents are allowed to do by default, how they authenticate, who can see their trails, are being made by a very small number of people, in a very small number of companies, on a very fast timeline.

I’m not an AI researcher or a policy expert. I’m an IT Director who’s spent years running production systems and watching what happens when they fail in the real world.

The people affected by these systems should have a voice in how they’re built. And the time to lay the accountability infrastructure is now, while the concrete is still wet, not once it’s hardened around harm.

This is the first in a series where I’ll be exploring these questions: Who gets to shape the systems that shape us? What does accountability look like in an age of autonomous AI? And how do we build technology that serves human flourishing and not just efficiency?

In the next piece, I’ll dig into consent: what it actually means when your AI agent can act on your behalf, and whether “click Allow and hope for the best” is anywhere close to adequate.

If you’re asking similar questions, I hope you’ll subscribe.

Rachel Ankerholz is an IT Director and writer exploring the intersection of AI ethics, accessibility, and human-centered technology. She writes about who gets included and who gets left behind when we build systems.