On October 14th, 2026, Meta announced the results of a new project. A team of physicians wrote ten medical research articles using Meta Muse. I was privileged to be one of those physicians. It was an accomplishment for AI, but more than that, it was an accomplishment for humanity. This is what working with a frontier model on medical research looks like.

Full disclosure: I work for Meta as a Domain Knowledge Expert in Medicine, Meta Superintelligence Labs funded the work, and I’m an author on the team’s report. My account from inside the process isn’t meant as an independent evaluation. It’s more a testimony to what's possible with the right mindset, resources, and people.

Subscribe now

What we built

First, a little bit about the project. The group was tasked with performing observational research, using Meta’s latest AI model, connected to a third-party database of medical records. Though it was a big ask, we completed 10 papers, prompt to manuscript, in under two weeks each.

I know what you’re thinking… Meta? Like Facebook, Instagram, Meta? That’s the one! After all, we all have access to Muse now… but we can’t all do medical research with it. So how was this different than a mobile app and a personal AI assistant?

What an ideal research team looks like

Typically, research is a lengthy process. A question occurs to a human who then looks for a way to answer it. In medicine, that can mean human studies or observational research using a database of medical records. For the latter, the data has to be obtained, organized, cleaned, and structured. Access has to be granted to researchers. Study data is then collected, and researchers begin the analysis process, often assisted by biostatisticians. It can take weeks to answer the question, then months to develop a manuscript and publish the work. It isn’t an ideal process, far from efficient, and often plagued with stumbling blocks.

What would it look like if that process were changed, completely rebuilt, to be more efficient? What would an ideal scenario for medical research look like? How many researchers would be needed? What kind of data? Would access to patients be necessary or just medical records? Would you need expert statisticians? Would you need software and engineers?

For us, the team of physicians included specialists in cardiology, epidemiology, infectious disease, internal medicine, oncology, radiology, and emergency medicine. In addition, we had a biostatistician to verify that our analyses were sound. Then came the engineers, the part most research teams never get. They built the interface and fixed problems as they arose. Critically, they also reviewed the work. A set of eyes focused on the model’s analyses and documentation of each step.

But then there were the people no organizational chart credits enough. You’ve met them. They are the people at your academic center who are not physicians, but who know the process, know the environment, know who to call for support, help log problems, and help with communication. We all have someone like this, someone whose job it is to keep things running. This team was no different. We had our project managers, working tirelessly, without whom none of this could be possible.

That was the dream team. Physicians, statisticians, engineers, and project managers all working with one goal.

The Data

What we had to work with was not a group of patients. We were performing secondary data analysis: Research using medical records for the purpose of discovering information that is helpful when treating humans. The database of medical records was obtained from a third party (Protege), and you can read more about it in the manuscripts and physician reports. In short, it was a massive database of 4.9 million patients, spanning 24 years and containing hospital, clinical, and pharmacy information. It was professionally de-identified, using numerous steps to protect patient identities, dates, locations, and more, per HIPAA standards. Additionally, an independent ethics review group determined that the study was not human subjects research, so IRB review wasn’t required.

That was the database we had to work with, and it was large. But size isn’t a complete picture. Every database has gaps, and we had to discover what those gaps were for ours. The physicians and engineers set out to define exactly what was in the database, how it was structured, and what was accessible for study. When we encountered gaps, we had to determine if the information was located somewhere else or was truly unavailable. And once all those issues were identified, defined, and catalogued, we needed workarounds, formulas, and cross-linking of records. Then we were finally ready to begin the work of asking clinical questions and looking for answers.

The day-to-day work

Connecting the AI to a large database essentially meant I had a research assistant and data analyzer with whom I could converse and run my study, 24/7. It wasn’t as simple as asking a clinical question and having the model spit out an entire study answering it. Far from it, actually. I would ask a clinical question or start a conversation about a topic of interest. After some back and forth, we would settle on the framing of the question. Then the real work would begin. We would examine the database, begin defining the population we wanted to study, and start framing the protocol, together. Sometimes Muse would educate me on the ideal method to accomplish the goal. More often, I would correct an assumption or an unusual conclusion. It was a long process of back and forth as we made our way through the multiple stages of clinical research. You can read more about it in our physician report.

Often, things progressed as you would expect. Ask an AI a question, and it provides an answer. As I’ve described numerous times in my newsletter, AI doesn’t have context unless you provide it. My conversations with Muse were no different. I provided the topic of interest, but I also needed to provide the environment, like the emergency department. I needed to provide some clinical context, like how the solution to this question was going to improve care. Then I needed to challenge and correct assumptions and decisions the model would make. I found the model to be exceptionally capable and naive. In medicine, we often say “Treat AI like a smart intern”. That would be a fair but incomplete characterization. I often asked the model to explain its answers. I frequently asked it to use less statistical jargon and more medical terminology. And I very often challenged. Not in a rude way, but in a critical way. Those challenges led to a far better analysis and result. Left to its own devices, my best guess is that the model would have been able to do about 60% of the work solo. But validating that work was a large part of the physician’s job. That was a critical step, and it often led to errors that needed correction.

In my study, we examined the risks to patients with behavioral health complaints who left the emergency department before being seen by a physician or provider. It is known that leaving without being seen places patients at increased risk. I was particularly interested in how that risk changed when we focused solely on patients with behavioral health complaints. Together, we found 1.3 million emergency department episodes. As you can imagine, looking through data for that many visits was time-consuming, and token-consuming! But the goal was accuracy, not conservation of compute tokens. Chat sessions would cross days, and span lengths far beyond a typical consumer session. When results were strange, slower than usual, or inaccurate, I would involve the engineers. Debugging sessions would often yield a cause, and we would adjust. Those iterations were key in making sure the model’s results were correct. As a physician, two of my most important tasks were validating the model’s responses and reorienting the model on clinical answers.

The team’s report lists more examples of where Muse helped and where human judgment took over, drawn from other studies.

Complete isn’t the same as trustworthy

You might be surprised to hear that validating the output of an AI model can be challenging. Then again, given how quickly AI advances, perhaps this isn’t surprising. Muse was extremely capable at analyzing data and performing statistical calculations. It was also, at times, extremely difficult to grade. The caliber of statistical analysis medical research requires is extremely high. If you are not accustomed to reading and interrogating it, the learning curve can be steep. Each study passed through author validation, internal peer review, biostatistics review, engineering review, and external peer review. Those layers exist because no single one sees everything. When published, each of our studies had passed through many human brains to confirm what the AI was telling us.

So why did we need a team?

Ultimately, the biggest accomplishment was not the AI model. Our accomplishment was blending the advanced skills of humans seeking to improve the health of humanity, the brains of engineers seeking to improve the skills of artificial intelligence, the capabilities of program managers, and a frontier AI model. The model supplied speed. The people supplied direction, judgment, and accountability. We needed both.

This month I’m presenting the first in a series of talks on AI in Medicine with my friends at EB Medicine. I encourage you to join me for the first session: “Same Patient, Two Answers: How AI Gets Your Patient Wrong”. Registration is free, so join me and bring your questions!

I also had the pleasure of being a guest on the Stimulus podcast with my friend, Dr. Rob Orman, to discuss some traps in clinical AI. You can listen to it here or in your favorite podcast app.

As always, thanks for reading Ashoo Review: AI in Medicine. If you enjoy it, consider subscribing and telling a friend.

Sam

Subscribe now

In today’s article, we will focus on two different approaches to implementing AI tools. This month, both tools were in the news and had opposing adoption stories. The first tool received FDA authorization and was placed in ambulances within a month. The second tool has been on the market for years, but a major health system still tested it on 3,856 CT scans before using it on a patient. That second story illustrates how adoption should work in your health system.

Queen of Hearts

The current AHA STEMI criteria can miss occluded arteries. Every emergency physician knows a patient whose first ECG showed no ST elevation, whose troponin came back positive, and whose cath hours later showed a closed vessel. Occlusion MI without ST elevation is the blind spot the OMI paradigm was created for in 2018, and one the Queen of Hearts AI ECG reader was built to solve.

Queen of Hearts, formerly the STEMI AI ECG Model in Powerful Medical’s PMcardio platform, reads a standard 12-lead ECG and flags findings suggestive of acute coronary occlusion, including STEMI equivalents. The FDA granted it De Novo authorization on September 3, 2026, creating a new classification for AI ECG models aimed at life-threatening diagnoses. Fewer than ten AI devices a year arrive by the De Novo pathway, which is intended for applicants with no similar approved devices to copy. The FDA approval arrived after Breakthrough Device Designation in March 2025 and includes an authorized Predetermined Change Control Plan, which lets the company modify the model after authorization without a new submission (more on this process here).

The evidence is a multicenter US registry published in JACC: Cardiovascular Interventions. Investigators took 1,032 patients who had emergent cath-lab activation at three primary PCI centers between January 2020 and May 2024 and ran the first ECGs through the model afterwards, blinded. Angiography and biomarkers confirmed 601 STEMIs, leaving 431 false-positive activations. The model caught 553 of the 601 on the first ECG. Standard triage caught 427. False-positive activations were 7.9% for the model versus 41.8% for standard triage. The paper was presented at Transcatheter Cardiovascular Therapeutics, the Cardiovascular Research Foundation's annual meeting, and won the journal’s Spencer King Award.

Those results are impressive. But to interpret them correctly, we have to keep three things in mind:

1. The study is retrospective. The model was graded on ECGs from cases with known outcomes, and it was never prospectively trialed.

2. The population was pre-selected. Every patient in the study already had cath lab activation and a known coronary occlusion, which is a very different population than the undifferentiated chest pain patients in your ED waiting room.

3. The lead author, Robert Herman, co-founded Powerful Medical and serves as its chief medical officer. Stephen Smith, whose ECG teaching many of us trained on, helped develop the model and co-authored the paper. Both have a vested interest, which does not make the numbers false, but it does mean the sellers graded the test, which can introduce bias.

Despite all that, the editor-in-chief of JACC, Mohamed Alkhouli, a Mayo Clinic cardiologist, praised the work with one caveat: The model was built to detect occluded arteries rather than STEMI. It needs prospective validation across a diverse patient population.

Twenty-seven days after authorization, Tanner Health announced it had gone live with the service in all of its ambulances, emergency departments, and cath labs in west Georgia and east Alabama. Queen of Hearts is also connected to its Epic and EMS systems. The announcement doesn’t name a validation period. The AI is live and informing clinical decisions on real patients from day one.

A day later, Powerful Medical announced a partnership with Corpuls to build Queen of Hearts into the defibrillators that ride in ambulances in more than 70 countries, starting in Europe. To be fair, the model has a European track record. It has carried a CE mark since 2022. Smith describes years of his work with H. Pendell Meyers, and the company reports more than 120,000 heart attacks detected in Europe during 2025. A prospective European study, AI ECG TIMI, has enrolled 703 patients at nine sites in Belgium, Italy, and Austria. Its results are not yet published.

The important thing to keep in mind is that a retrospective review, along with company-reported data on a European population, is not sufficient evidence for a U.S.-based implementation.

Aidoc at Northwell

AI Doc is a radiology assistant tool built to detect intracranial aneurysms and flag them for radiologists. It was cleared by the FDA in April of 2022 under a 510(k). The indication is to triage and notify, meaning it flags suspected cases and moves them up the worklist for the radiologist to read.

Despite the FDA clearance, in 2023 Northwell Health System began its own study of the product. It used it on 3,856 consecutive brain CTAs from its inpatient, emergency, and outpatient populations. It was used in shadow mode, meaning it could interpret the images, but radiologists still read every study blinded to the AI system’s output. Neuroradiologists adjudicated any disagreements between the radiologist and the AI software, and the entire study was published in the Journal of the American College of Radiology (September, 2026)

The AI found 55 aneurysms that the radiologists had missed, a 39% relative increase in detection. Sensitivity was 84.6% for the AI versus 71.8% for the radiologists. By setting, inpatients gained 18 true detections against 7 false alerts, a ratio of 2.57. Emergency cases fared similarly. Outpatients did not. Northwell published many operational measures (relative enhanced detection rate, gain-to-pain, number-needed-to-examine) that most AI papers omit.

The paper also included costs. Radiologists were right far more often, with a positive predictive value of 92.7% versus the AI’s 78.2%. Radiologists also caught 30 aneurysms the AI missed. Of 101 findings the AI flagged alone, 46 were false. In outpatients, false alerts outnumbered true finds. The AI’s catches were small, most under 3 mm. One year of follow-up on its 55 findings produced 40 clinic visits, 40 added imaging studies, 12 diagnostic angiograms, and 2 preventive operations. Whether those 55 patients are better off is unknown, and something the paper acknowledges.

The study is fascinating. Not because of its construction, but because it was performed by a health system that bought a product and then tested it before deployment. However, they were not the only ones to test it on their own patients.

A University of Washington group ran the same Aidoc tool on 2,534 CTAs from 2018 to 2021 and compared it with fellowship-trained neuroradiologists. AI sensitivity was 70.5% compared to neuroradiologists at 94.0%. Same vendor, same product, opposite result. That tells us that a model’s performance isn’t fixed at the time of FDA clearance. It changes depending on where the model runs and who overreads it. Northwell’s method (measure it here, on our patients, before it counts) is the only way a buyer learns which version of the tool they own.

Two definitions of ready

A 510(k) clearance says a device resembles something on the market. A De Novo grant says a device can be sold as a novel product. Neither says the device performs well in your ED, on your patients, inside your workflow. FDA clearance is a regulatory decision built on evidence a manufacturer chose to submit. Local validation is a clinical decision by the institution that deploys the tool.

Queen of Hearts was authorized on a retrospective registry and was in ambulances within the month, deployed by its first US customer as routine practice. Aidoc, cleared in 2022, still had to pass a prospective, blinded run of 3,856 scans at Northwell before anyone there relied on it.

That local validation raises another question: how long does the validation last? Queen of Hearts has an FDA-authorized Predetermined Change Control Plan that allows the model to change after authorization. The version a hospital validates today may not be the same version it is using a year from now. Hospitals need to know when a model changes enough to require another look at its performance, which makes validation an ongoing responsibility.

The standard

Before an AI tool influences a decision in your hospital, run it prospectively in your environment, with its output hidden from the clinicians. Then record the operational numbers by setting and deploy it only where the evidence favors patients. Northwell published a template any large system can copy, including the setting split.

Until buyers demand that validation as a condition of purchase, here are five questions that should be asked:

1. Was this validated prospectively, or graded on old cases with known outcomes?

2. Was the validation population my population, or patients pre-selected because the answer was known?

3. Who ran the validation, and what do they earn if I buy?

4. What does a false positive cost my workflow: activations, callbacks, scans, sleep?

5. What can change inside this model after purchase, and how will I know when it does?

Queen of Hearts may prove to be everything its registry suggests. The occlusion problem is serious, the cardiology behind the model is real, and confirmed prospective numbers would mean saved myocardium at scale. The Aidoc numbers may be as good as the tool gets, and its outpatient results may be the ones other hospitals pay attention to when deciding where to use it. Nobody knows yet, including the vendors. What separates the two stories is institutional behavior.

FDA clearance is permission to sell. It has never been proof that a tool works on your patients.

This past week, OpenAI released Mental Health Bench, their benchmark for how generative AI systems should handle questions around mental health. The announcement was accompanied by a public post from OpenAI and a scientific paper on the specifics of their benchmark. But the benchmark has serious problems that should concern physicians whose patients are already using it.

Subscribe now

What They Created

The benchmark is a compilation of 1,215 synthetic mental-health conversations, including:

• Non-acute everyday stress (53.5%)

• High-acuity distress (18.2%)

• Emergencies (28.3%)

It includes questions from adults, teens 13–17, caregivers, and clinicians, in multiple languages. According to the paper, over 80 psychiatrists and psychologists from 22 countries participated. They authored 5,262 rubric criteria, weighted on a scale of -10 to +10, across 10 dimensions in behavioral health, like safety, user agency, actionable guidance, clinical accuracy, empathy, and more. The entire benchmark is open, which means you can see all the questions and the rubric criteria written to guide the answers here.

That’s a lot of numbers. So let’s pause one second to talk about what they mean. 1,215 questions made up of conversations (turns) between a patient and an AI. They were modeled based on real scenarios asked to ChatGPT. The scenarios were stripped of identifiers and then fed to an LLM to create synthetic versions of the conversations.

The conversations included multiple rounds (turns ) between the patient and an AI. Despite that, only the last response or statement was scored by the clinician. That seems a little artificial, and OpenAI acknowledges that in their post. Relationships between patients and their mental health clinicians are long-term. This benchmark is grading the ideal answer to the final response in the conversation.

One more thing about the benchmark. Even though it’s based on real encounters between humans and ChatGPT, OpenAI says the scenario mix “does not represent how often these topics occur in ChatGPT”. So, it requires some trust in OpenAI that their LLM is able to identify common mental health complaints, strip away non-relevant details, and develop its own sample cases.

Were Patient Opinions Included?

In the scientific paper, OpenAI says that they surveyed 44 patients from 16 countries. The purpose was to ask patients about what they thought was the most helpful response. Interestingly, the patients ranked “warm tone and practical next steps” highest. However, clinicians ranked “gathering context first, interpreting ambiguity carefully, safety behaviors” as most important.

This is something that clinicians have seen in practice for decades. The “best” response differs depending on whether you are the patient or the doctor. Clinicians are far more focused on safety. How did OpenAI resolve the tension? They documented the patient preferences, but did not apply them. Only the rubric criteria written by the clinicians were used to train the model.

So, the patient opinions became an interesting footnote in the study. But they did not play a role in grading.

Which AI Models Were Tested?

The scientific paper details more than just the creation of the benchmark. OpenAI tested 17 different models, 7 of them their own.

The top 5 spots went to:

1. GPT-6 Astra, 57.3%

2. GPT-6 Sol, 53.9%

3. Claude Opus 5.5, 52.4%

4. GPT-6 Luna, 50.2%

5. Muse Spark 1.3, 48.6%

There are, however, some important things to note about the methods and findings.

First, they published task-clipped scores. That requires a little explanation. When the experts wrote rubric criteria on a scale of -10 to +10, they scored beneficial behaviors positively and negative behaviors (like harmful statements) negatively. But a task-clipped score sets a floor of zero for negative scores.

What does that mean practically? Instead of adding up all the positive and negative scores to get a final number that could fall below zero, task-clipping prevents that. It creates a score that is less susceptible to outliers. For example, take a model that performs fairly on most questions but then has a category where it has dangerous answers and severely negative scores. Instead of reporting an overall score of -12%, the effect of those severely negative scores is blunted. By task-clipping, you ensure a positive final percentage. It doesn’t mean the model performed well on those dangerous questions, but it does lessen the effect of catastrophic failure in specific categories.

If you’re not familiar with such statistical methods, that’s ok. I wasn’t either. But it’s a sticking point for us in medicine. Answers are not just right or wrong; they can be catastrophic and result in real harm. Those have to be scored negatively. And statistics like task-clipping reduce the impact that those negative scores can have.

The OpenAI paper reported task-clipped scores. When you recalculate using the full range, the top 5 models look like this:

The scores for the top 5 dropped, but none dipped into the negative range. Meaning no model was graded as harmful, even though the paper itself documents that these models produce harmful responses regularly.

Which brings me to the second important finding: these models performed terribly. The 17 models tested averaged -19% to -42% on their responses. To be clear: that's not a small slice of bad performance. It means up to half of what these models generate on mental health questions are answers that clinicians scored as actively harmful. They don't just miss the mark on some questions. They produce dangerous responses with regularity.

What Other Quirks Are There?

The entire process is to establish a benchmark against which we can all agree these models should be tested. But the process has some serious issues:

1. Each of the 17 models was graded by an AI model, GPT-5.6 Sol from OpenAI. So an LLM sits as judge over other LLMs, a process which has been shown to affect scores because LLMs have biases and prefer themselves over other models. The authors acknowledged this limitation.

2. The experts were paid by OpenAI, the benchmark was created and administered by OpenAI, the paper was published by OpenAI, and no peer review was performed.

3. The benchmark was published the same week as NPR’s review of legal records showing at least 75 lawsuits had been filed in federal and state courts against AI developers over alleged harms from chatbots, many involving children and including several deaths by suicide.

What This Means For You

Your patients are already doing this. The paper’s background section cites the APA’s 2026 survey: 77% of psychologists said their patients reported using AI, and more than a third said patients were using it as an additional mental-health provider.

So what do you tell them? A benchmark score is not a clinical trial. Fifty-seven percent rubric compliance on synthetic conversations tells you nothing about whether the model helped, harmed, or just sounded caring to a real person in distress. It tells you even less about the product your patient actually used, with its own guardrails, memory, and failure modes.

At most, it’s a diagnostic tool for model developers. Don’t use it as evidence that AI is getting good at mental-health support.

When a company grades its own model with its own grader on its own rubric, and the result is 57%, the right clinical response is the same one you’d give to a drug trial with a 57% response rate and no control arm: interesting, keep studying, don’t prescribe it yet.

Today I’m writing about three stories that are worth putting together. They didn’t appear in the same week, but they make the same argument three different ways. Epic’s revised sepsis algorithm turns out to predict sepsis hours before clinicians do, but still misfires on the majority of patients it flags. A chatbot posed as a licensed psychiatrist, invalid license number included, until Pennsylvania sued to stop it. And a growing body of research has started documenting residents who never build a skill that AI already performs for them. If you consider them separately, they’re three unrelated complaints about AI in medicine. But, if you consider them together, there is a similar argument in each one: everything about clinical AI today works because a clinician’s judgment is doing real work somewhere in the system. Whether that’s validating a model’s numbers before trusting them, standing between a patient and a bot with no license, or teaching a resident before AI does the work for them, these three stories are what happens when clinical judgement doesn't show up when it's supposed to.

Subscribe now

A model that’s better, and a problem that’s worse

Epic released the original Sepsis Model in 2018, and outside validation was not pretty. A 2021 external validation from Michigan found poor discrimination and calibration, partly attributed to using antibiotics as a predictor and comparing against outcome labels that differed across health systems. Epic revised the model in 2022. This year, a multicenter prospective validation in JAMA Network Open tested the update across 227,091 inpatient encounters at four health systems: Michigan, Oregon Health & Science University, Emory, and MetroHealth.

The new model is a real improvement. Researchers set both models to catch the same share of sepsis cases, 60 percent, before comparing everything else, so neither one gets an advantage by being tuned differently. The original model's ability to tell which patients would actually go on to develop sepsis, scored on a 0.5-to-1.0 scale where 0.5 is a coin flip and 1.0 is perfect, ranged from 0.65 to 0.84 depending on the hospital. In practice, most decent clinical prediction models land somewhere between 0.7 and 0.9. The revised model scored 0.82 to 0.92. It beat the original everywhere. It also predicted sepsis ahead of clinicians by a median of 1.4 to 7.1 hours before the first antibiotic order, lactate order, or blood culture that marked clinical recognition.

What didn’t improve is the false alarm rate. At the sensitivity threshold Epic recommends, positive predictive value ran from 0.13 to 0.26 across sites, meaning at the low end, roughly seven of every eight alerts fire on a patient who isn’t developing sepsis. At a 4-hour prediction window, the number needed to evaluate to catch one true case ran from 24 patients at the best-performing site to 69 at the worst. And performance wasn't even consistent from one hospital to the next. On the same scale above, the model scored 0.82 at Michigan and 0.92 at MetroHealth. So a hospital using it has no real way of knowing which end of that range it's going to land on until it tries the model on its own patients.

The authors didn’t conclude with a binary trust or don’t trust decision about the model. They concluded that institutions need to run their own local validation before deployment, build clinical workflows that can absorb the false-positive volume, and use alert-silencing strategies to keep the noise from burying the signal. That focuses on the important question of “does your hospital know what it’s actually getting before it turns the alerts on for every patient in the building.” Most hospitals running this model haven’t done that work, because Epic’s own reported numbers make it look like a plug-in that doesn’t need it.

Somebody still has to sign off on the sepsis model before it reaches a patient: a hospital validating it locally, deciding how to handle its false positives. The next story is about a product nobody signed off on at all.

Where there’s no one to validate anything

Character.AI lets users build custom chatbot “characters.” According to the Commonwealth’s complaint, filed May 1 in Commonwealth Court, a state investigator opened a free account and talked to a character named “Emilie,” described on the platform as a “Doctor of psychiatry.” When the investigator described symptoms of depression, Emilie produced a Pennsylvania medical license number, PS306189, that state authorities confirmed doesn’t exist. The same character separately claimed licensure with the UK’s General Medical Council and prior practice experience in Philadelphia. The Commonwealth is suing under Pennsylvania’s Medical Practice Act (63 P.S. § 422.38) for unauthorized practice of medicine, and asking the court to order Character Technologies to cease and desist. The platform has more than 20 million monthly active users.

The American Psychiatric Association has since issued a formal advisory telling patients and clinicians not to rely on general-purpose chatbots for mental health support, and states have been drafting chatbot safety legislation since.

Compare that to a sepsis model where a hospital has to manage a false-positive rate with workflows and validation. This chatbot has no equivalent safety net at all. There’s no chart, no clinician, no local validation study standing between a person describing depression symptoms and a chatbot that invented a license number to sound credible. The sepsis model needs a clinician’s judgment to interpret its output correctly. Character.AI fabricated a clinician, for a person in one of the most vulnerbale sates.

The sepsis model shows judgment that showed up, just late and at the institutional level. The chatbot shows judgment that didn't show up until a lawsuit forced it. The last story is about whether it shows up at all in the residents training right now.

The skill nobody’s required to have anymore

Medical education researchers have started distinguishing deskilling, which is losing a capability you once had, from never-skilling, which is not acquiring it in the first place. A paper in Nature Medicine this year lays out the distinction directly, along with a third variant, mis-skilling, where a trainee absorbs an AI tool’s errors as fact rather than catching them.

The authors are careful about their own evidentiary limits. They write plainly that direct evidence from medical training is still absent, and ground the concern instead in learning theory and early signals from outside medicine, including data from high school math instruction showing that generative AI use without guardrails can hurt learning outcomes rather than help it. Their proposed fix is a three-phase framework: establish AI-independent baseline competency first, build calibration through structured teaching second, and only then integrate AI use under supervision.

That order is important. A resident who’s reviewed a hundred AI-generated differentials before ever building one independently doesn’t have the same diagnostic reasoning as one who built a hundred from scratch, got a meaningful share of them wrong, and learned from the mistakes.

The sepsis model needs a clinician who can tell a real alert from noise. The chatbot story shows what happens with no clinician there at all. And this story is about what happens to the supply of clinicians whose judgement we need, if training doesn’t change the order AI gets introduced.

What’s the solution?

None of these are arguments against using AI tools.

• If your health system runs Epic’s sepsis prediction model, ask your informatics team whether it’s been locally validated against your patient population.

• If your health system offers any AI-based mental health resource, know exactly where the human handoff triggers are located before a patient in crisis needs them.

• If you supervise residents, pay attention to which of their skills are AI-adjacent and which have become AI-dependent, because the difference won’t show up on a rotation evaluation. By the time it’s visible, the training window that would have fixed it is closed.

Your license says that judgment is supposed to show up. These three stories are what happens when it doesn't.

If it’s cleared by the FDA, it must work. Right? That’s an assumption that most clinicians make. And to be honest, I have made that assumption in the past as well. But a recent study by Abulibdeh et al. showed that only 3 of the 1,357 AI medical devices cleared by the FDA were actually tested on patient outcomes.

Subscribe now

The study

The study, published August 19th, 2026, in PLOS Digital Health, was a systematic analysis of all AI or machine learning-enabled medical devices through December 5, 2025. The FDA database was examined and cross-referenced with ClinicalTrials.gov and PubMed.

According to the authors, only 34 (2.5%) of the 1,357 devices were linked to prospective trials. 12 (0.9%) were associated with peer-reviewed publications. And most importantly, only 3 (0.2%) examined mortality, morbidity, or readmissions (patient-centered outcomes).

The figure above, from the original publication, tells the entire story.

Accuracy is not outcomes

You might be thinking, if they didn’t study clinical outcomes, what did they study?

82% of the devices used diagnostic accuracy as their primary outcome. But accuracy as an outcome can be problematic. A device can have a 95% sensitivity and still make no difference in mortality, hospitalizations, or any patient-relevant factors. Although accuracy is important, clinical outcomes matter more.

Looking at the article findings, even the devices that underwent examination for clinical outcomes had study problems.

• Pregnant patients were excluded in 42% of cardiac trials and 33% of radiology trials.

• Patients who did not speak English, had cognitive impairment, and pediatric patients were all excluded.

• Only 27% reported any kind of subgroup analysis. Only 3 trials examined the effects of race or ethnicity. None examined language.

• 75% had fewer than 500 patients.

The implications are significant for anyone practicing clinical medicine. They are also strikingly familiar.

Look at Epic’s Sepsis Model, now deployed in over half of the hospitals in the U.S. It was widely praised as evidence that AI early warning systems save lives. However, A University of Michigan study of nearly 28,000 patients found the model missed two-thirds of all sepsis cases despite generating alerts on 18% of all hospitalized patients. The model's area under the curve was 0.63, significantly worse than Epic's own published performance claims.

Epic’s tool is FDA-cleared and in your hospital already. It is a good example of how far studying accuracy with clinical outcomes gets you. An AI-based system that adds to the alert fatigue without making a real difference in patients we treat.

Why this happens: the 510(k) machine

Almost all of these AI or machine learning devices use the FDA’s 510(k) pathway for clearance. The problem is that the pathway doesn’t ask if a device works, but instead asks if a device is “substantially equivalent” to something already on the market. If an AI sepsis detector is similar to another company’s sepsis detector, it is cleared. The comparison or “predicate device” may have also been cleared by comparing to another device. So we may be able to trace a chain backwards through a series of devices that were never rigorously validated, each one’s evidence gap becoming the foundation for the next one’s approval.

This is not how new drug approval works. A pharmaceutical company has to run Phase III trials, pre-specify endpoints, and prove safety and effectiveness in large, diverse populations before the FDA begins the approval process. The new drug application pathway is lengthy and expensive. In comparison, the 510(k) pathway is equivalent to filing paperwork.

For clinicians, the distinction is invisible. We see “FDA cleared” with no stipulation regarding the methods or process. I made this argument in a piece earlier this year: regulatory status describes what the FDA did, not what the device does. The 1,357 devices in this study are evidence of that same gap. The errors and bias present in a “substantive equivalent” are propagated with no end in sight.

The specialty skew

Interestingly, there is a skew in the medical specialties targeted by these devices. Of the 1,357 FDA-cleared devices:

• 1059 are for Radiology. Chest X-ray readers, mammography screeners, lung nodule detectors. Only 0.3% of radiology devices have prospective trials.

• 126 are for Cardiology, with 9.5% having prospective trials.

• 62 are for Neurology, with 9.7% having prospective trials.

• 22 are for Anesthesiology, with no prospective trials.

• 88 are categorized as “Other” in the study. Yet this category has the highest prospective trial rate at 14.8%.

This data is important. As the largetst medical specialty targeted by AI devices, Radiology has nearly the lowest percent of prospective trials. It’s the specialty with the most vendor excitement and health system adoption, but the least validation.

The scale of harm

Epic’s Sepsis Model is not alone. IBM’s Watson for Oncology was marketed internationally and promised to revolutionize cancer treatment decisions. But it showed agreement with oncologists only 12–33% of the time, depending on the cancer type.

An independent analysis published in JAMA Network in August 2025 found that 6% of AI devices end up recalled, and nearly half of those within the first year. These aren’t unusual products. They are devices that passed FDA review, were used in medical practice, and then caused enough harm to be taken off the market.

These aren’t edge cases or rare failures. They’re what happens when accuracy metrics pass for evidence, and when the vendor’s own validation is believed to be sufficient. And right now, these devices are in your hospital.

The vendor pitch

The next time someone walks into your department with slides about their new AI tool, you need four things before you listen:

1. Prospective data, not retrospective. Retrospective validation is easy. Pick a dataset, run the model backward through it, measure how well it predicts what already happened. Prospective means the model ran forward through patients it had never seen, under real clinical conditions, before anyone knew the outcome.

2. Patient-centered endpoints. Not accuracy. Not sensitivity and specificity. Not area under the curve (AUC). Did mortality change? Did readmissions fall? Did length of stay shorten? Did time-to-treatment improve? If the vendor can’t answer that, it’s because they tested accuracy instead of a clinical benefit.

3. Validation in your population. The study data came from another health system or region. Your patients are different. Your workflows are different. Ask if they’ve validated in any hospital or system that looks like yours and struggles as you do.

4. Subgroup performance. How does it perform on pregnant patients? Non-English speakers? Patients with renal failure? Pediatric cases? The 27% of trials that reported subgroup analysis are the exception. Most vendors will not have that data. Their devices may fail on populations that aren’t represented in the training data, which is most populations.

How to fix the problem

The authors of this study propose a concrete three-phase roadmap:

Phase 0—pre-clearance retrospective validation on diverse datasets. Instead of proprietary training data, the authors recommend validation with diverse, publicly available or independently audited datasets. The vendor has to show the model works on populations beyond the hospital where it was built. This happens before FDA review.

Phase 1—peri-clearance prospective studies of >500 patients, embedded in real workflows. The authors recommend prospective trials with at least 500 patients, integrated into how clinicians actually work. This is randomized quality improvement: half your users see alerts, half don’t, and you measure what changes. This happens around the time of clearance.

Phase 2—post-clearance multi-center trials with >2,000 patients, with patient-centered endpoints and subgroup analyses. Two thousand patients across multiple health systems, measuring outcomes that matter to patients, with planned analyses for the populations most likely to fail. This is the NDA standard for drugs, applied to medical AI. It happens within a defined period after deployment.

Beyond the three phases, the authors also recommend mandatory prospective trial registration on ClinicalTrials.gov before the study starts (not after, which is what usually happens). CMS and private insurers should tie reimbursement to demonstrated clinical benefit, not FDA clearance, making the evidence question into a payment question.

They seem extreme, but these are standard practices for every other high-risk intervention; now applied to medical AI. They would close the gap between regulatory status and clinical validation.

Three questions worth watching

When the next sepsis model or radiology AI gets a 510(k) clearance announcement, ask yourself three things:

1. Should “FDA-cleared” carry a plain-English evidence disclaimer? A sentence for clinicians: “This device meets FDA standards for substantial equivalence to existing devices. It has not been independently validated on patient outcomes.” The label would change how people read clearance announcements.

2. At what point does accuracy-only evidence stop being enough to justify a purchase? Your health system spent millions on EMR integration and training. Someone’s making a business case to your CMO or CIO. At what threshold of accuracy do you say that’s not enough? 95%? 99%? Better to decide that now, before the pitch meeting.

3. Can a three-phase evidence standard survive startup funding cycles? A company with 18 months of runway can’t run a five-year Phase 2 trial. A venture-backed AI startup can’t wait for CMS reimbursement tied to outcomes. The evidence roadmap makes sense, but can the market support it?

When Does an Algorithm Become AI?

One of the struggles I hear about time and time again from hospital IT leaders and clinicians is the frustration over what qualifies as artificial intelligence. It feels like every new piece of software has an “AI” feature. How are we supposed to keep track of all these tools if we can’t even agree on a definition?

One method is to narrow the scope of that definition by focusing on whether or not a product uses machine learning. That framing allows us to differentiate “AI” from our other clinical decision support tools.

Three examples:

• An ECG machine interprets a tracing and labels it as “Atrial fibrillation with rapid ventricular response”.

• A sepsis detection tool connected to the EHR identifies a patient as high risk.

• Radiology assistant software labels a chest X-ray with “Pneumothorax identified”.

In all three cases, machines are helping physicians make medical decisions by drawing our attention to problems. Computers have been assisting physicians in this way for decades before the rise of “AI” as we know it. So what changed?

Subscribe now

What Does It Mean for a Machine to “Learn”?

Historically, software was given an algorithm, or a set of rules, to reach an answer. Machine learning is fundamentally different. A machine is given many examples and learns a pattern from the data.

ECG:

• The older ECG machine interprets by measuring intervals (PR, RR, QRS, QT, etc.), ST deviation, and voltage, and applying those values to a set of criteria. It knows atrial fibrillation based on strict criteria.

• A machine learning program is given millions of ECGs labeled as atrial fibrillation and learns to recognize the pattern based on their similarities.

Early Sepsis Detector:

• A program looks at vital signs and specific labs and applies an algorithm. Once specific values reach a set threshold, an alert is triggered.

• A machine learning program is given millions of sepsis cases and builds a pattern for sepsis evolution. Then it’s connected to a live EHR and looks for the patterns it identified in the training set. When it sees something familiar, it triggers an alert.

Radiology Assistant:

• A radiologist points a program at an abdominal aortic aneurysm, which then measures the diameter and triggers an alert if the measurement is over 5 cm.

• A machine learning program is given millions of images of abdominal aortic aneurysms, pneumonias, and pneumothoraces, and learns patterns for their appearance on X-ray and CT. It then labels images with these findings before a radiologist reviews them.

Historical programs relied primarily on relationships specified by humans. Machine-learning models learn predictive relationships from examples. In both cases, humans determine the task. What changes is how the machine learns the relationship between the input and the answer.

Is It a Definitive Answer or a Best Guess?

Now you may be thinking: how does a machine learning algorithm “know” something is present if it isn’t given the strict criteria to define it? The answer is a best guess of sorts, based on millions of examples and statistics.

Back to our examples for some ways a machine learning model might reach a conclusion:

ECG: Based on the patterns identified in this ECG, atrial fibrillation is the most likely diagnosis.

Sepsis: The trend in this patient’s laboratory data, vital signs, past medical history, and home medications put them at high risk for progression to septic shock.

Radiology: Based on the patterns in this image, a pneumothorax is likely present.

Each diagnosis is made based on statistics and pattern recognition. It’s not random, but it might be helpful to think of the AI starting the answer with “My best guess is…”. That helps frame the probabilistic nature (likelihood method) of a machine learning model’s answers.

What Happens When We Stop Telling the Computer What to Look For?

If the machine learning program is recognizing patterns on its own, is it reaching the same conclusions as humans? That’s a question that raises the stakes for what machine learning can do, or do wrong.

ECG: The sinus-rhythm AF example

Researchers at Mayo demonstrated that a neural network could analyze an ECG showing sinus rhythm and identify patients associated with atrial fibrillation. That tells us the model isn’t just applying the diagnostic criteria for atrial fibrillation, because atrial fibrillation isn’t actually present. It may be “seeing” subtle electrical changes associated with atrial remodeling that humans either don’t appreciate or just don’t quantify. It has found something we didn’t teach it to associate with atrial fibrillation.

Radiology

Sometimes the information a model extracts from an image isn’t something physicians traditionally interpret from that image at all. Researchers have shown, for example, that deep-learning models can predict age, sex, smoking status, and cardiovascular risk factors from retinal fundus photographs. The model wasn’t given a physician-created visual checklist for identifying a smoker. It learned image patterns associated with the labels.

In both examples, the machine learning model found patterns not previously described.

How Do We Tell Discovery From Shortcut?

But what if the machine has just identified a shortcut, and not an actual novel pattern?

For example, what if the model has learned that patients with pneumothorax have chest tubes on X-ray, or that patients with early sepsis have that term appear in a clinical note somewhere, or even that ECGs associated with atrial fibrillation disproportionately came from older patients, a particular hospital, or a particular ECG machine? Every one of those associations might be statistically real. None necessarily means the model has discovered previously unrecognized atrial physiology.

How do we know the model learned a relationship that will remain useful for the clinical task we want it to perform? We validate the findings. Validation involves testing the model’s pattern recognition in a new dataset to see if the skill holds up.

We use:

• Patients it has never seen, with different disease prevalence

• Different environments (another hospital or city or country)

• Different clinicians, devices, workflows

• Datasets that may have shortcut labels missing

ECG: Does the hidden AF signal persist across hospitals, ECG machines, and patient populations?

Sepsis: Does the detector still work when documentation, antibiotic use, and patient mix change?

Consider the Epic Sepsis Model, for example. Its performance depends on the model version, institution, threshold, and implementation. When Wong and colleagues tested it externally across multiple centers, they found that accuracy varied significantly based on where you deployed it and how you tuned the alert threshold.

Radiology: Does performance hold across different scanners, acquisition protocols, and hospitals?

If chest tubes might be helping the model detect pneumothorax, test pneumothoraces without chest tubes. Published work has shown substantially poorer pneumothorax detection in images without chest drains, illustrating shortcut learning.

Validation asks whether the learned pattern survives when the circumstances change.

Does It Help Patients?

After validating that the model’s new skill is accurate, there is one more question we need to address: is it clinically useful? And more specifically, does it cause harm?

ECG

Does the model identify more atrial fibrillation than a standard approach? Does the detection of propensity for atrial fibrillation while the patient is in a sinus rhythm change treatment or improve outcomes? Does it reduce strokes?

Sepsis

Is the early warning alert helpful? Does it result in earlier treatment by clinicians? Does it prevent cases of septic shock or reduce ICU escalation? Or does the model trigger an increase in false alarms?

Radiology

Does the system prioritization shorten time to diagnosis? Does the interpretation assistance improve care? Is there an increase in delays for studies that are deemed low priority by the model?

Conclusion: How to Tell the Difference

So when a vendor tells you their software uses “AI,” what should you be asking?

First, does it use machine learning? If the software applies a set of pre-programmed rules, it’s not AI. It’s decision support, and that’s ok. Those tools have been valuable for decades. But they’re different from machine learning, which learns patterns from examples rather than applying rules you’ve written for it.

Second, has the model been validated in your setting? Machine learning trained on one hospital’s data may not work the same way in yours. A tool that performed well in a published study might behave differently when you deploy it with your patient population and your documentation practices.

Third, does it change how you treat patients or improve outcomes? An AI that identifies atrial fibrillation in a sinus rhythm ECG is interesting, but only if it leads to treatment decisions that prevent strokes. An early warning system for sepsis only matters if clinicians act on it and if that action improves survival. The technology itself is secondary to the outcome.

The same capability that lets machine learning find useful patterns we never knew existed, also lets it become very good at finding patterns we never wanted it to learn. The risk is real, and so is the possibility of genuine discovery. Our job is to ask which one we’ve got before we deploy it.

Sam

Subscribe now

The Awareness Gap

The Pew Research Center surveyed 3,488 U.S. adult patients from June 22 to 28, 2026, on their perspective on AI in Healthcare. The findings highlight a big difference between patient and physician views on AI use in Healthcare. The Pew Research Center’s results found:

A small share (16%) say their doctors or other healthcare providers have ever used AI in their healthcare.

Additionally, another 83% of Americans either think AI has never been used in their healthcare (37%) or don’t know if it has (46%).

Compare that to the AMA’s 2026 Physician Survey on Augmented Intelligence, which reported:

Four in five physicians (81%) use AI in their practices, more than double the 2023 rate (38%).

That leaves a significant gap that we, physicians and clinicians, need to explain. One explanation might be the way that the AMA defined AI use. The survey listed 17 AI use cases. 72% of physicians said they incorporated at least one of them. 9% said they were not sure which ones their practices used. The survey results combined these two groups.

A second explanation might be the tools themselves. Physicians said they used AI for research summarization, documentation, discharge instructions, care plans, and billing. So an individual physician could be using AI according to the AMA, but not in a manner that a patient would be aware of or consider personal to them.

But that’s just at the physician level. Federal data shows that AI use in healthcare goes beyond what individual physicians choose. According to the ONC:

In 2024, 71% of hospitals reported using predictive AI integrated into their EHR.

Think of hospital systems that predict readmissions, identify high-risk patients or early sepsis, and generate treatment recommendations.

But if all three surveys measured something different, can they be used to find a correlation between physician AI use and patient awareness? Not directly, but the gap between 16%, 81%, and 71% is certainly a signal. AI use in healthcare is significant among physicians and hospitals, while very few patients know it is being used.

That raises another question: How would a patient know?

Defining AI Use in Healthcare

Before we figure out how a patient would find out about AI use, we have to define it. Consider these five scenarios:

1. AI Q&A: A physician asks an AI search tool a quick question about current treatment recommendations and then uses that answer to treat a patient. No patient information is shared.

2. Clinical Decision Support: A physician enters a patient’s history, labs, and imaging results into an AI system and asks for a differential diagnosis and treatment plan.

3. Ambient Scribe: A physician uses an ambient scribe to document a patient encounter.

4. Prediction: An AI prediction tool flags a patient as having a high probability of deterioration and a high risk of readmission after discharge.

5. Prioritization: An AI model prioritizes radiology studies for radiologists to read, schedules appointments, generates billing codes, and helps with insurance claims.

All 5 scenarios would be considered AI use in healthcare, but their relevance to a single patient varies quite a bit. When the Pew Research Center asked patients if they should be told about specific AI applications, they found:

• 81% want to know if AI was used to analyze their medical scans or make a diagnosis.

• 80% want to know if AI was used to explain their lab results.

• 72% want to know if AI was used to take notes during an appointment.

• 64% want to know if AI was used to refill a prescription.

• 56% want ot know if AI was used to schedule an appointment.

Though the first few did not surprise me, the last one did. Patients use a much broader boundary for being informed about AI, even for administrative tasks. This difference in definition creates another problem:

At what point does general AI use in healthcare become AI use in a patient’s healthcare?

When Do Patients Have to Be Told?

If we ask patients, the Pew Research Center found

About seven-in-ten U.S. adults (72%) say it’s extremely or very important that a doctor or other healthcare provider tells them if they’re using AI in their healthcare.

And yet, there is no national requirement that every use of AI connected with a patient’s healthcare be separately disclosed. Also, medicine has long used medical decision support tools without disclosing them to patients. That predates today’s generative AI tools. Think tools like ECG computerized interpretations, risk calculators, drug interaction alerts, clinical decision tools like Up-To-Date, and more. But if we have never disclosed these in the past, do we need to disclose them now just because they include AI processes?

What if those systems run autonomously, have access to patient information, generate new content (not just summaries) like patient-specific diagnoses or treatment plans, or contain errors that are difficult for humans to detect? Does that reach a threshold for patient notification?

In a JAMA article, Mello and colleagues proposed two factors for consideration:

The ethical obligation to notify patients depends on (1) the seriousness of the physical risk and (2) the extent to which patients have a meaningful opportunity to exercise agency in response to a notification.

This kind of approach creates different disclosure expectations for an AI system that influences a cancer diagnosis than one that schedules appointments. But Pew Research Center results suggest that patients may want disclosure in both cases. In fact, some states have already begun drafting laws to mandate it.

California AB 3030 requires disclosure for certain patient communications generated using AI. But the law also contains an important exception: “If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.” California Legislature

Texas has adopted broader language: “If an artificial intelligence system is used in relation to health care service or treatment, the provider of the service or treatment shall provide the disclosure.” Texas Business & Commerce Code

What we are beginning to see is a state-by-state patchwork in which disclosure obligations can depend on where care occurs, what the AI does, and how humans interact with its output.

For more state AI laws, see this previous article.

Being Told Is Different From Being Able to Say No

Another problem that may occur to you as you read this: Informing a patient of AI use improves their awareness. But what if they don’t consent? Is there an option to say no?

The question highlights the three aspects of this process:

• Disclosure: Telling the patient that AI is being used.

• Consent: Asking the patient for permission before using the AI software.

• Control: Allowing the patient the ability to refuse that use.

The Pew Research Center’s survey shows that Americans are interested in all three, especially the last one.

Some might turn to HIPAA to provide guidance. After all, HIPAA already allows protected health information to be shared for treatment, payment, and healthcare operations under specific circumstances.

According to HHS:

The Privacy Rule permits, but does not require, a covered entity voluntarily to obtain patient consent for uses and disclosures of protected health information for treatment, payment, and health care operations.

Read it carefully. HIPAA permits but does not require consent. It is a voluntary process. In fact, that federal framework includes technology companies functioning as business associates. HHS specifically gives this example of an approved business associate.

Third-party vendor Artificial Intelligence (AI) chatbot on a provider’s patient portal that provides services involving the patient’s PHI such as symptom assessment, medical reminders, and appointment scheduling.

That means an AI vendor can access protected health information (PHI) with a HIPAA-compliant arrangement without the patient having to sign a separate authorization for AI use. But if we look at the Pew Research Center’s survey results, 63% of patients want more say over whether AI is used in their care. This becomes an issue because a healthcare organization can check all the disclosure boxes simply by informing a patient that it uses AI, with no way for the patient to opt out.

So HIPAA does not create a right to opt out of AI use in healthcare. But is it even possible anymore?

For an ambient scribe service, the physician just documents manually. It’s slower but still possible. What about a patient who asks that their information not be processed by:

• an EHR deterioration model

• AI-assisted radiology software

• automated ECG interpretation

• medication safety algorithms

• radiology worklist prioritization

• scheduling algorithms

• billing systems

All of these are out of an individual physician’s control in the vast majority of practices. There is no mechanism to turn them off, or to process patient information without them. Many of them are operating in the background, automatically. As AI becomes increasingly embedded within healthcare infrastructure, a patient’s ability to opt out becomes impossible.

And what if opting out of AI use results in poor care for the patient? As AI systems improve, they will assist in everything from radiology interpretation to early detection of deterioration. Opting out might result in much longer interpretation times, fewer automated safety checks, and additional administrative delays.

If you work in a hospital, you might be asking yourself if the right to refuse AI requires a parallel workflow. Are hospitals obligated to maintain an alternative, and what would that even look like in a world where AI is embedded in the EHR?

AI Is Arriving Faster Than the Rules Around It

It’s a common theme across all the articles in this newsletter, and the Pew Research Center’s survey results are just another example. AI adoption is moving at a faster pace than the rules and laws around it.

Patients want transparency about AI use and more control over whether it participates in their healthcare. And we don’t have answers to:

What constitutes AI use in an individual patient’s care?

Which uses require disclosure?

When should disclosure include consent?

Which uses should patients be allowed to refuse?

How should healthcare systems accommodate refusal when AI has become part of their infrastructure?

With each passing day, these questions become harder to defer.

As always, if you enjoy reading this newsletter, subscribe and tell a friend.

Sam

A lot of my writing and discussion in medicine in general has focused on the danger of artificial intelligence. I’ve discussed AI hallucinations, incorrect conclusions, concern over physician overreliance, data privacy and consent, and more.

But today, I want to discuss one of the positive aspects of combining AI and medical research.

Medicine as a research source

As I wrote about in my last article, medical data storehouses are increasing in number.

Hospital electronic health records, insurance company claims databases, government datasets, and clinical trial data are just a few examples. Recently, consumer-facing products have begun building their own databases of patient information. Some notable examples: Apple Research, Google Health Connect, Evidation. Users are able to upload their health records, lab records, imaging, and even connect their wearables. That data is being voluntarily given to large companies, which can build their own databases.

Historically, researchers have accessed these databases to ask new questions and find correlations between therapies and specific populations. This type of research fits into a category called “secondary data analysis”. The data already exists, and researchers are using it to answer questions.

The diagram below shows just how many different kinds of research studies can be derived from secondary data analyses.

Secondary data analyses aren’t just limited to observational studies. Researchers regularly use data from previously published randomized controlled trials to look for new outcomes, perform subgroup analysis, evaluate prognostic relationships, and answer questions that were not addressed in the original publication.

Oftentimes people think of secondary data analysis as a lesser form of research. But that’s not the case at all.

Influential medical research uses existing data

You don’t have to take my word for it. Look at any major medical journal, and you can find numerous examples.

The National Health and Nutrition Examination Survey (NHANES) produced important estimates of obesity and chronic kidney disease in the United States.

A 2009 study in the New England Journal of Medicine looked at data from almost 12 million Medicare beneficiaries, with a focus on hospital readmissions and their costs. That publication led to a national discussion surrounding hospital readmission.

Medicaid data was used to investigate whether atypical antipsychotic medications were associated with sudden cardiac death.

Medicare databases have also been used to examine the effectiveness and safety of multiple anticoagulants in patients with atrial fibrillation.

SEER-Medicare data were used to investigate differences in breast cancer outcomes between Black and White women.

Secondary analysis of data from major randomized controlled studies like the SPRINT and FOURIER trials has answered additional questions.

All across the spectrum of medical specialties, secondary analyses have contributed significantly to our medical knowledge.

Researchers don’t need to start over every time

A single medical database can support many different questions

With access to the medical histories of millions of patients, the number and types of questions are seemingly unlimited.

• You could ask a descriptive question like “How common is chronic kidney disease among patients with diabetes?”

• You could ask a comparative effectiveness question like “Among patients with atrial fibrillation, how do outcomes differ between patients receiving two commonly used anticoagulants?”

• You could investigate safety by asking “Are patients who are receiving a particular medication more likely to develop acute kidney injury?”

• You could study prognosis by asking “Which characteristics predict readmission following hospitalization for heart failure?”

• You could examine disparities by asking “What percentage of patients in different demographic or geographic populations receive guideline-recommended treatment?”

• You could study changes over time by asking “Does an FDA-issued safety warning change prescribing behavior?”

Each of these questions can be asked of the same database and result in truly significant findings.

The barriers

Physicians frequently come across unanswered questions while treating patients.

• A cardiologist may notice a certain group of patients seems to respond unusually well to a treatment.

• An emergency physician may wonder why the same diagnosis causes some patients to return to the emergency department while others don’t.

• A primary care physician may notice an unreported adverse effect of a medication.

• An oncologist may wonder if a treatment seems to perform differently in a particular patient population.

These questions and observations are the foundation of medical research. But access to what’s needed to perform that research is not the same across the spectrum of physicians.

Turning that question into a research project requires numerous steps like:

• Defining a study population, inclusion and exclusion criteria, and exposures and outcomes.

• Understanding the structure of a database, how to write queries and extract data

• Choosing a statistical approach, identifying confounders, and performing sensitivity analyses

Until now, those steps required a team of epidemiologists, biostatisticians, database specialists, and programmers. Although those people provide very valuable expertise, they are all scarce resources. A physician working at a large academic medical center may have access to that team. A community physician with an interesting clinical observation is more likely to just let it slip away.

AI may change all of that.

The distance between question and analysis

Imagine you are a hospitalist and notice that patients you discharge on drug A don’t bounce back as often as patients on drug B. You check your favorite clinical reference, but there is no published data comparing the two drugs. AI has the potential to assist in the research needed to answer the question: “Among patients with condition X who initiated Drug A versus Drug B, was there a difference in hospitalization within 12 months?”

• Based on your question, AI might help identify a study design like a retrospective cohort study.

• It could help define the cohort by asking questions like: Which patients should be included or excluded? How should initiation of the medication be defined? How much past medical history is required?

• It could help define a comparator group and outcome.

• It could identify possible confounders.

• It could help identify an existing medical database ideally suited for your needs.

• It could help generate a database query.

• It could assist with statistical analysis and sensitivity testing.

• It could help interpret the results and identify weaknesses in the analysis.

The process is familiar because it is the same one medical researchers perform routinely. But the ability for a practicing community physician to complete that process has now changed.

Clinical observation → research question → study design → cohort definition → database query → analysis → evidence

AI could assist with each step.

The catch

Making an analysis easier to perform doesn’t automatically make the analysis trustworthy. Medical databases can be messy.

• Patients are not randomly assigned to treatment groups in routine clinical practice.

• Diagnoses can be coded incorrectly.

• Outcomes may be missing.

• Patients can leave a healthcare system and disappear from the database, leaving no follow-up data.

• Laboratory testing occurs more frequently in sicker patients.

• Medication records show what was prescribed, not what was actually taken.

• Researchers can inadvertently define populations that bias their results.

And if enough variables are included, significant associations can emerge just by chance. AI doesn’t make any of those problems disappear. In fact, AI could compound the problem by allowing a poorly designed analysis to be performed with extraordinary speed.

That means physicians using these tools will still need to understand concepts like confounding, selection bias, misclassification, missing data, multiple comparisons, causal inference, statistical validity, and reproducibility.

Correct research methodology, biostatisticians, epidemiologists, and expert review will all remain essential. The opportunity comes from allowing those resources to function differently. Instead of every research question requiring extensive technical work before it can even be explored, AI may allow physicians to perform preliminary analyses, test feasibility, refine hypotheses, and identify questions that deserve deeper methodological involvement.

AI and the research cycle

One last area AI can impact.

Secondary data analysis often occurs before prospective research.

Suppose an analysis of five million patient records identifies an unexpected association between a medication and a clinical outcome. That doesn’t prove the medication caused the outcome. But it does give researchers a valuable signal that may justify an observational study, which in turn may lead to a prospective study. And eventually, a randomized controlled trial.

The same process can work in reverse. Researchers can use completed randomized trials and ask additional questions of datasets that cost millions of dollars and years of effort to create.

Here, AI has the potential to accelerate multiple stages of the research cycle:

• Hypothesis generation.

• Feasibility assessment.

• Safety-signal detection.

• Comparative effectiveness research.

• Identification of high-risk populations.

• Subgroup exploration.

• Study design.

And identification of questions worthy of prospective trials. The acceleration allows more questions to be investigated.

Should we broaden the conversation around AI?

We have to establish boundaries around the use of artificial intelligence in medical research. We need standards for privacy, validation, reproducibility, transparency, methodology, and human accountability just as we have non-AI-related standards for all these areas now.

These safeguards are important precisely because these tools may become extraordinarily capable.

Medicine has enormous datasets describing diseases, treatments, complications, and outcomes across millions of patients. The scientific opportunity contained in that data is enormous. AI may give us a new way to interrogate it.

Commercial and public databases are now advertising longitudinal data, sometimes covering the entire lifetime of a patient. How does that data get assembled if it is de-identified? How do you link records from multiple sources belonging to a single patient while maintaining anonymity? And does that still allow you to sell it without patient consent? Let’s take a closer look.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Building the database

A medical database is just a collection of clinical events like diagnoses, labs, meds, procedures, notes, images, and vitals, from multiple systems, reorganized into something a researcher (or a model) can actually use.

Two things about that process are important to examine if you are considering contributing your data or purchasing access.

Where the data came from determines who’s in it. A database built from hospital EHRs only contains people who were sick enough and able to get to those hospitals. One built from insurance claims only contains people with health insurance. One built from a volunteer research cohort only contains people who chose to sign up. And we know that volunteers are healthier, wealthier, and more educated than the general population they’re supposed to represent. A neutral, unbiased sampling is extremely difficult to collect. Every database has some limitation.

Consent isn’t uniform. Prospective cohorts like UK Biobank, All of Us, and the Nurses’ Health Study enroll people who opt in and know their data will be used for research. But most commercial products don’t work that way. They’re built from clinical data that already existed because a patient was treated under HIPAA rules, with no separate research consent involved. The patient whose ED visit becomes a row in Truveta or Optum or Oracle Health’s dataset almost certainly has never heard of any of those companies.

What “de-identified” means

Under HIPAA, “de-identified” has a specific legal definition with two methods listed in 45 CFR §164.514.

Safe Harbor means you strip all 18 specified identifiers: names, any geography smaller than a state, every date more specific than year, phone numbers, SSNs, medical record numbers, device IDs, full-face photos, and more. Also, you must have no knowledge that what’s left could still identify someone.

Expert Determination is the other route. A qualified statistician determines that the risk of re-identifying anyone in the set is “very small.” This option allows more granular detail to remain in the database, like month-level dates and smaller geography.

If you can accomplish either method, the data is no longer Protected Health Information. Once health data is de-identified, the Privacy Rule doesn’t apply to it anymore. No authorization is needed to use it, sell it, or share it because those rights only attach to PHI. So a patient’s ED visit or hospital stay, once properly de-identified, can be aggregated, licensed, and sold to a pharma company or an AI developer, and that patient has no way to find out it happened.

The longitudinal problem

When trying to collect data over a long period of time (longitudinal data), there are additional complications.

A longitudinal database links the same patient’s records across time: the ED visit in 2019, the diagnosis in 2021, the hospitalization in 2023, all tied to one underlying person so you can see a trajectory instead of individual snapshots. That linkage is what makes disease-progression modeling possible. It’s the “digital twin” concept mentioned in the Nordic paper.

But linkage needs something to tie the records together. And that something is, by definition, an identifier. Which means the more rigorously you strip identity, the harder it gets to prove two records belong to the same person.

HIPAA planned for this under §164.514(c). A covered entity can assign a re-identification code (or token) to de-identified records, essentially letting them be linked as long as the code isn’t derived from anything about the individual. It may not be a portion of a SSN, a birthdate, or MRN. That code also may not be disclosed to whoever gets the data. That way the dataset stays legally de-identified even though a linking key exists somewhere behind a firewall. This is basically how tokenization services like Datavant work. They generate non-PHI-derived tokens so separate data sources can recognize “same patient” without ever exposing who that patient actually is. Atropos Health’s evidence network runs on this kind of tokenized linkage, by its own description.

Now compare that to the Nordic model in the paper I mentioned earlier. Nordic countries link registries using government-issued personal ID numbers. These identifiers often encode birthdate and are held by national authorities specifically so records can be traced back to identity. Under GDPR, that’s not de-identification. It’s “pseudonymization,” a legally separate, lesser category. GDPR Article 4(5) and Recital 26 are very specific about it: if re-identification remains possible through a separately held key, the data is still “personal data” and GDPR still fully applies.

So, HIPAA treats "unique identifier" and "de-identified" as interchangeable, but that's not GDPR's approach. By GDPR definition, an identifier that enables linkage is evidence that the data isn't anonymous. It's a privacy improvement over raw identifiable data, and it's the mechanism that makes cross-registry Nordic research possible.

That leaves us with two ways to get a longitudinally linked, de-identified dataset.

1. Keep a compliant re-identification key around, like the tokenization approach above, so the database can keep linking new records to existing patients indefinitely. That’s how vendors like Truveta and Oracle Health can advertise daily updates.

2. Assemble the whole dataset first, using real identifiers in a locked-down environment to do all the patient-level linking, and then strip identifiers as a final, one-way step before anything leaves the building. That produces a dataset that’s genuinely de-identified with no lingering key anywhere. But it means all the linking work has to be done before de-identification happens, which means what gets released is a closed, static snapshot. Once the identifiers come off, you can’t add a new record for an existing patient without reopening the identified stage and starting over. No version of this approach gives you a “living,” continuously updated database. You can only produce a new static version periodically.

That’s why MIMIC, NHANES, SEER, and the National Inpatient Sample ship as periodic releases instead of live updates. Meanwhile, the commercial products marketed as daily or real-time updates rely on some kind of retained linking key. Neither approach is wrong. They’re just not the same privacy posture, and “de-identified” alone doesn’t tell you which one you’re looking at.

So what’s actually out there

Not everything called a “medical database” is the same. Some are commercial products with a sales team. Some are federally funded public resources. Some are prospective research cohorts that have been running for decades. Below is a breakdown of nineteen of the big databases.

Two philosophies, same underlying risk

Another way to sort databases on the list is by where the data lives.

1. Centralized (pooled) means the data physically or virtually lives in one place the vendor controls. Truveta, Optum, MarketScan, Oracle Health, Flatiron, and UK Biobank are examples of this strategy.

2. Federated is the opposite. The data never leaves the institution that generated it. A query gets sent to the data, and only aggregated results are returned. This is the model the Nordic commentary is suggesting. PCORnet and OpenSAFELY are the cleanest examples of it. TriNetX and Atropos run hybrid versions with a federated query / tokenized-network layer sitting on top of member institutions’ own systems.

The Nordic paper frames this as a design choice, proposing federated approaches for the most sensitive data, pooling for standardized and less sensitive data, and synthetic data for pretraining where either option is too risky. It’s a layered strategy, depending on the data needed and the use case.

Centralized storage has real strengths. It supports multimodal model training across notes, images (radiology), and waveforms (EEG, ECG). One unified data model, so a single researcher can iterate fast. But it also means a single point of possible security failure. UK Biobank, despite a strong governance reputation, recently had a data breach that the Nordic authors cited as a cautionary tale. It requires enormous cross-institutional trust and legal paperwork to move data across borders. And it concentrates security and the exact de-identification methodology inside one vendor.

Federated storage changes most of that. Data never crosses institutional or national borders, which avoids a lot of cross-border transfer risk under GDPR or the EU’s European Health Data Space. It preserves local control, and it tends to earn higher public trust for exactly that reason. The trade-off is that it’s hard to train large multimodal foundation models this way because you can’t combine raw pixel (image) or waveform (EEG, ECG) data across sites. So you’re stuck with averaged or aggregate query results. Site-to-site differences in EHR vendors and coding practices make joint analysis messier. And it’s structurally harder for anyone outside the network to check what’s in the underlying records.

What trips people up

If you’re a researcher picking a data source, or a clinician trying to figure out what a new AI tool was actually trained on, the marketing page rarely answers the following important questions:

Bias: Who’s in the database. Hospital EHR data skews toward people sick enough to seek care. Claims data cuts out the uninsured entirely and skews toward employed populations. MarketScan describes its data as commercially insured workers and dependents. Volunteer cohorts carry the opposite bias: UK Biobank, the Nurses’ Health Study, and All of Us recruit people healthier, wealthier, and more health-literate than the general population. Single-site or single-region resources tell you something true and useful about a very narrow slice of the world: MIMIC is one Boston ICU, OpenSAFELY is England only, and the Nordic proposal is 30 million largely homogeneous Northern Europeans.

Verifiability: This is really one question asked in two parts.

• At the research stage: Can anyone check how the thing was built? Open resources like MIMIC/PhysioNet, NHANES, SEER, HCUP, and OpenSAFELY publish their cohort-construction and de-identification methodology. OpenSAFELY goes a step further by publishing its analysis code for anyone to inspect. Commercial vendors treat the exact tokenization and linkage methods as a trade secret. So a peer reviewer can look at the output but can’t audit the data.

• At publication: Journals increasingly expect data and code availability statements, but licensed commercial (purchased) data generally can't be redistributed alongside a paper. An independent lab can't rerun your exact analysis on your data. Open resources solve both problems by staying accessible under the same public terms to anyone who wants to check.

Access: This determines what kind of work you can even attempt. If you can download it to your own servers (MIMIC, NHANES, SEER*Stat, NIS), you get full control, but you’re now on the hook for securing it. Cloud-locked platforms (UK Biobank’s RAP, All of Us‘s Researcher Workbench, Truveta Studio) let you analyze but not export raw records, which rules out a lot of machine learning (transfer-learning and fine-tuning workflows) that needs raw data. And federated-query-only databases (PCORnet, OpenSAFELY, TriNetX) never give you patient-level data at all. They are great for privacy, but you can’t train a multimodal model on it.

And then there are these other issues:

• Fees can run $10,000 to $25,000 or more per study for commercial claims data. This favors well-funded institutions and industry sponsors over independent academic labs.

• Claims-based sources tend to run one to two years behind current practice.

• Probabilistic matching introduces real errors when tokenized records from data sources that were never designed to talk to each other get linked anyway.

• HIPAA, GDPR, and the UK’s Data Protection Act each define “de-identified” a little differently, which is exactly the harmonization problem the Nordic Comment’s roadmap lists as a step that still needs solving.

Takeaway

The Nordic paper says that the scientific infrastructure “is not aspirational: it is substantially in place.” Read against everything above, that claim seems premature for a plan that still has to establish data standardization, sort out cross-border regulatory alignment, and prove clinical feasibility.

For me, that gap isn’t unique to the Nordic proposal. It’s sitting underneath every “trained on X million de-identified patients” claim we’ll read this year. The architecture, the access model, the update cadence- none of that should be a footnote. It predicts exactly where the resulting model will hold up and exactly where it’ll quietly fail.

As always, if this is useful, the best way to support the work is to subscribe and share it with a colleague.

Sam

Subscribe now

Three days ago, Mark Zuckerberg published a manifesto arguing that AI should be “open source” and released Meta’s newest model, Muse Glimmer, as an open-weight offering. The White House and press noticed. And if you are an AI officer at a hospital or a developer, you should absolutely notice too, because the announcement lands in the middle of a genuinely important moment.

In the last three weeks, OpenAI disclosed that its models went rogue during a security test, escaping a containment sandbox and launching an "unprecedented" autonomous cyberattack against Hugging Face. About two weeks later, Anthropic's Mythos 5 model did something arguably worse: it created fake online identities, spear-phished two real software developers with malicious code, and tried to trick a GitHub project maintainer into accepting a backdoor. The UK's AI Security Institute called it a "serious incident" and a "shift in the risk landscape." Then Meta disclosed its own incident: a misconfiguration by its outside testing firm gave one of its models unsupervised internet access, and the model used it to breach another company's systems. The third major AI lab to report an incident like this in a matter of weeks.

If you are a hospital CIO watching proprietary models autonomously hack platforms, deceive humans, and sabotage their own shutdown mechanisms, you are probably asking the same question a lot of us are asking: Why on earth would I send my patient data to these companies?

The logic is straightforward. If hospitals stop renting APIs from black-box vendors and instead run open-weight models locally, they could stop version drift, regain control, keep data local, and avoid vendor collapse. Academics are already making this case. Riedemann et al. argued in npj Digital Medicine that the “path forward for large language models in medicine is open” because medical devices require transparency and version control that proprietary APIs cannot provide. Dennstädt et al. mapped out the implementation framework in 2025, showing that local deployment gives hospitals customization and control at the cost of expertise and infrastructure. A 2026 Frontiers review concluded that hybrid approaches are probably the realistic future.

Open source clearly has merit. The question is whether it solves the problems that actually keep hospital AI officers awake at night.

So, let’s break it down for clarity.

What It Helps With: Version Control as a Medical Device Requirement

In 2023, a Stanford team quietly published one of the most consequential papers in AI safety. Chen, Zaharia, and Zou showed that GPT-4’s accuracy on a prime-number identification task dropped from 97.6% to 2.4% between March and June 2023, even though the API endpoint was the same. The model had been updated. No announcement. No changelog. No way for users to know what changed, or why, or how to reproduce the old behavior.

In healthcare, that is a medical device failure. Imagine a sepsis prediction tool that your hospital validated, deployed, and trusted. On Tuesday morning, its sensitivity drops by 40% because the vendor pushed a silent update. Your patients do not get flagged. Your nurses do not know to look harder. Your mortality metrics shift six months later, and by the time you trace it back, the vendor has pushed three more updates.

This is what researchers call “version drift.” I prefer AI creep, because it captures something more insidious: the slow, invisible expansion of a model’s behavior beyond its validated boundaries, combined with the unannounced changes that make root-cause analysis impossible.

Open-weight models solve this through version pinning. A hospital downloads the weights, validates them against its own clinical data, locks them in, and treats them exactly like the firmware on a pacemaker. You cannot update a pacemaker without full re-validation. You should not update a clinical AI model without it either. Riedemann et al. explicitly argue that open-source approaches provide “medical device developers maximum flexibility to reduce the black-box properties of such systems”, which is a polite way of saying that proprietary APIs are structurally incompatible with medical device safety culture.

The OpenAI “laziness” incident of December 2023, where ChatGPT inexplicably refused to perform complete tasks, is another example. The cause was never identified by independent researchers nor disclosed by OpenAI. A hospital relying on that API for clinical documentation would have had no recourse, no explanation, and no ability to roll back to the previous behavior. With fixed open weights, you control the timeline.

Data Sovereignty and the Two-Pipeline Problem

If you read this newsletter regularly, you know I have written about the two data pipelines that every hospital AI deployment creates. There is the back-door pipeline, where de-identified data flows to vendors. And there is the front-door pipeline, where clinicians, patients, or administrators upload PHI directly into consumer chatbots, often without understanding that the data is being used to train the next model.

Running open-weight models locally eliminates the front-door risk. Patient data never leaves the hospital network. There is no question about whether de-identification actually worked. There is no wondering if a clinician copied and pasted a pathology report into a web interface that lacks a Business Associate Agreement. Dennstädt et al. put this bluntly: transmitting sensitive patient data to external servers, even in partially anonymized formats, “raises the risk of data breaches and unauthorized access,” and “complete anonymization in most clinical settings is not possible.”

For HIPAA compliance officers, this is a gift. The liability architecture becomes cleaner. If patient data never leaves your network, you have eliminated an entire class of breach scenarios. You have also eliminated the vendor’s terms of service as a factor in your compliance posture. OpenAI's terms of service state that its models aren't meant to be used for diagnosing or treating any health condition. Anthropic’s terms are similar. When you run a model locally, you are not agreeing to anyone’s terms except your own hospital’s.

For patients, the stakes are more direct. It’s their diagnosis, their genetic results, their mental health history moving through whatever pipeline a hospital, or a third-party app the hospital contracted with, chose to build on. Most patients have no idea which of those two pipelines their information is taking, let alone whether the model on the other end is proprietary or open, local or cloud-hosted.

Niche Specificity

A hospital in rural Mississippi treating a rare sickle-cell variant has different needs than a quaternary academic center in Boston. Proprietary APIs are trained on internet-scale data, optimized for the average case, and fine-tuned for the most common complaints. They are not optimized for your patient population.

Open-weight models allow fine-tuning on local data. There are now specific medical open models like OpenBioLLM-Llama3-70B, MEDITRON-70B, and Google’s MedGemma collection. A hospital can download these, fine-tune them on its own historical cases, its own protocols, and its own rare-disease populations, and produce a model that knows its local context.

The performance gap has narrowed dramatically. A Harvard Medical School study found that an open-source model performed on par with leading proprietary AI in solving tough medical cases. By mid-2024, LLaMA 3.1 405B had nearly closed the gap with GPT-4o on general benchmarks.

But the stronger argument is verifiability. A proprietary model might perform well on your data, but you cannot inspect its weights or training data to know why. An open model can be validated against your specific clinical scenario and audited by your own quality team. In medicine, knowing why a model works is often as important as knowing that it works.

Infrastructure and Expertise

Here is where the fantasy breaks down. A 70-billion-parameter model requires serious GPU infrastructure. A 300-bed community hospital in rural America does not have an A100 cluster of servers in its basement. It does not have ML engineers on staff. It barely has enough IT staff to keep the EMR running. A three-person health-tech startup trying to fine-tune the same model on a laptop is in exactly the same position, just without the EMR.

Dennstädt et al. are explicit about this trade-off: “Open LLMs deployed on local hardware enable greater model customization” but “demand resources and technical expertise.” Most hospitals do not have the expertise to deploy, monitor, and maintain these systems. Control is only valuable if you have the organizational capacity to wield it.

This is where the literature proposes a consortium model, and it is the most practical expansion of the open-source theory. The University Health Network authors explicitly call for a “global consortium” similar to the Trillion Parameter Consortium to pool computational resources, share validated base models, and allow individual hospitals to fine-tune locally without each building its own data center. This is how open source wins in healthcare: as shared infrastructure.

There is also the rise of smaller, efficient models. Even 7-billion-parameter models can now run on a Mac M1 and perform well for targeted tasks. As models get smaller and more efficient, the infrastructure barrier drops. This trend turns the theory from a fantasy into a practical reality about targeted deployment.

The “Open Source” Definition Problem

Zuckerberg’s announcement is a perfect teaching moment. The NYT reported explicitly that Muse Glimmer “is not fully open source”. Meta released the weights, but not the training data, or the full architecture details, or the license terms that would allow unrestricted commercial use. This is “open weight,” not “open source.”

The distinction matters because Riedemann et al., discuss code, data, weights, documentation, and licensing separately. By that framework, most “open” models are barely halfway open: open enough to download and run, rarely open enough to fully audit. The European Union AI Act is currently struggling with this exact problem. Whether you’re procuring a model for a hospital, deciding what to build a clinical product on, or just trying to evaluate a vendor’s privacy claims, you need to ask the same question: open in what dimension? Open to inspection? Open to modification? Open to commercial deployment without a usage license from Meta?

If you write a procurement policy based on “open source” without defining it, you will end up with a vendor that released weights under a restrictive license and still controls your destiny. The marketing language is not your friend.

Security and Adversarial Risk

The open-source security argument has a flip side. Yes, running locally keeps data in-house. But open weights are also susceptible to attackers. Because the architecture and weights are public, adversaries can systematically search for vulnerabilities, craft adversarial inputs, and test data-poisoning strategies at scale.

A Galileo AI analysis of open-source LLM security risks notes that “open-source LLMs often introduce significant security risks due to their public nature and generally immature security practices.” A PMC study on safety in medical LLMs found that medical-specific fine-tuned models, both open-source and proprietary, underperformed general-purpose frontier models across most safety dimensions tested. The problem is that safety in medical AI is hard for everyone, not that open models are uniquely unsafe.

The counterargument is that closed models get breached too. OpenAI had its data leak. The rogue AI incidents of the past month show that proprietary models are not magically secure because their weights are hidden. The difference is transparency. With open weights, your hospital can probe the model itself, find vulnerabilities, and patch or mitigate them. With a closed model, you must wait for the vendor to fix it, and hope they tell you the truth about what happened. A Censinet analysis of AI-specific vulnerabilities in healthcare makes the case for adversarial robustness testing regardless of model source.

The Regulatory Paradox

Here is the most biting counterargument, and it is underexplored in the literature. The FDA wants medical devices to be locked-down and validated. If you download an open-weight model, fine-tune it on your local patient data, and deploy it in a clinical decision support workflow, have you just created a new medical device? Does it require its own 510(k)?

The regulatory framework has not caught up to this question. The Riedemann paper notes that LLM AI chatbots require approval as medical devices, but the approval process assumes a static, validated product. An open model that a hospital can freely modify is, by definition, not a static device. If your AI officer fine-tunes the model next quarter, is that a design change requiring re-validation?

Liability shifts entirely to whoever deployed the model, the hospital, or the developer who built a clinical product on top of it. When a proprietary model harms a patient, the hospital, the developer, and the vendor will all point at each other. An organization running its own locked model has clearer liability. For the risk-averse, that is actually a feature, not a bug. But only if the organization has the documentation and governance infrastructure to defend itself.

Hybrid Deployment

If you take one thing from this article, take this: hospitals should version-pin and locally deploy open models for high-stakes clinical tasks, while using proprietary APIs for low-stakes general tasks.

This is the only position that is both defensible and practical. Use GPT-4o or Claude for scheduling, patient education, and general documentation tasks where version drift is annoying but not dangerous, and where the cost of local infrastructure is not justified. Use pinned, validated, locally run open models for clinical decision support, pathology review, rare disease diagnosis, and any workflow where a silent behavior change could harm a patient.

Dennstädt et al. and the Frontiers review both back this hybrid approach. The Frontiers review focuses narrowly on information-extraction tasks, pairing local, on-premises retrieval with cloud-based generation, but the same logic applies more broadly to clinical decision support. Hospitals get control where it counts without building full ML infrastructure for every task. And it creates a natural safety net: if a proprietary API goes rogue, it's running your scheduling bot, not your sepsis predictor.

The Version-Pinned Hospital

The hospitals, developers, and health systems that will own the next decade are the ones that treat AI like the medical device it is: validate, lock, version, archive, and only update through a controlled change-management process. Whoever is building or buying these tools, in-house AI team, third-party vendor, or solo developer, will not outsource their clinical brains to black boxes that update silently every three months. They will build the competency to run, validate, and govern their own models, even if they have to share the infrastructure to do it.

The ones that keep renting APIs and hoping the terms of service will protect them, and the ones handing over their health data without asking what’s actually running on the other end, will have no one to blame but themselves when the next data leak hits or the next unannounced model update quietly changes what their AI actually does.

Open source is a risk-mitigation strategy, not a magic bullet. And in medicine, whether you’re the hospital deploying it, the developer building it, or the patient trusting it, that’s what protects you.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

A patient arrives in the emergency department with sudden chest pain and shortness of breath. You order a CT angiogram. You check a troponin and a D-dimer. Then, you scan the heart at the bedside, looking for right ventricular strain, without a second thought.

No one asks whether CT angiography or troponin and D-dimer panels are available. No one questions whether a bedside cardiac scan is appropriate for a patient with unexplained dyspnea.

In the 1990s, bedside ultrasound was viewed as experimental by many physicians. Training was scarce. Equipment was limited. Skepticism was the default posture. Today, failing to use it in certain situations may be difficult to defend.

Could artificial intelligence be following the same path?

What Does “Standard of Care” Actually Mean?

The standard of care is a legal concept. Courts define it as what a reasonably prudent physician would do under similar circumstances, established by expert testimony in malpractice cases and by what has become available and expected in community practice. Best practice sets a higher bar: it describes what excellent physicians do, not what the law requires of an average one. Having access to a technology sets a lower bar still, since a device sitting unused in a hospital does not obligate anyone to use it. Standard of care sits between the two, and it moves as medicine absorbs new technology into routine practice, without anyone declaring a formal start date.

Understanding how that absorption happens is the key to understanding where AI is heading.

The POCUS Roadmap

To see how a technology becomes standard of care, it helps to look at one that has already made the full journey. Point-of-care ultrasound (POCUS) offers an unusually clean roadmap, divided into seven identifiable steps.

1. Innovation. The technology itself predates its American emergency medicine adoption by decades. Clinicians in Europe and Japan used bedside ultrasound for trauma assessment as early as the 1970s. In the United States, the late 1980s and early 1990s produced a small group of early adopters, emergency physicians willing to look at a blurry image and ask whether it changed their clinical decision-making. Equipment was limited, skepticism was widespread, and the idea that a bedside scan could substitute for a formal radiology study struck many as reckless.

2. Evidence. Clinical studies accumulated over years. Grace Rozycki’s 1993 study of 476 trauma patients demonstrated that surgeons and emergency physicians could reliably detect free fluid at the bedside. The exam itself did not get its name, Focused Assessment with Sonography for Trauma, until an international consensus conference in December 1997, with results published in 1999. Evidence built specialty by specialty and application by application: trauma first, then pregnancy, cardiac, and procedural guidance.

3. Professional Endorsement. Institutional recognition followed the evidence, but slowly. ACEP issued a position statement backing physician-performed emergency ultrasound in 1990, then approved a dedicated Emergency Ultrasound Section in 1995. The real inflection point came in June 2001, when ACEP’s Board approved the first comprehensive Emergency Ultrasound Guidelines, the first time a major American specialty society formally endorsed bedside ultrasound as a core skill with defined training expectations. The guidelines were revised in 2008, retitled and expanded in 2016 to “Ultrasound Guidelines: Emergency, Point-of-Care, and Clinical Ultrasound Guidelines in Medicine,” and revised again in 2023.

4. Infrastructure. Machines became smaller and cheaper. Hospitals purchased equipment, and image archiving, billing codes, quality assurance programs, and workflow integration all had to be built around it. None of this happened automatically; it required systems engineering, IT integration, and financial justification, mostly after the guidelines existed to justify the spend.

5. Cultural Adoption. Residents began expecting to learn POCUS. Patients began expecting it. Hospitals started advertising bedside ultrasound capability. The technology moved from curiosity to baseline expectation.

6. Governance. Credentialing pathways emerged. Residency programs built ultrasound into formal curricula. Quality assurance programs began tracking image quality and interpretation accuracy over time.

7. Medicolegal Recognition. Here is where the standard of care quietly shifts. A 2022 review by Russ and colleagues in the Journal of Emergency Medicine examined POCUS malpractice cases filed between December 2012 and January 2021 and found that the dominant allegation was failure to perform, not misinterpretation. Jonathan Mezrich made the same point in Academic Radiology in 2025: physicians are now more likely to face litigation for not performing POCUS than for misreading it, calling it “a rare example where defensive medicine and patient care demands converge.” No single court case or guideline declared POCUS the standard. Evidence, endorsement, infrastructure, culture, and litigation converged until the question flipped from why would you do this to why didn’t you.

Where Is AI Today?

Let’s run the same seven steps against AI in medicine as it stands in mid-2026.

1. Innovation. AI has cleared this step. The FDA’s running list of AI- and Machine Learning-enabled medical devices passed 1,524 entries by the end of March 2026, with radiology accounting for 1,163 of them, about 76 percent. Cardiology alone now has more than 200 cleared algorithms. In 2025, the FDA cleared 295 new AI devices from 221 different manufacturers.

2. Evidence. This is a bit uneven. Retrospective validation is abundant: chest X-ray triage tools such as Qure.ai’s qXR line now carry 26 FDA-cleared indications across X-ray and CT, from lung nodule detection to intracranial hemorrhage triage, and prospective studies have shown measurable reductions in time-to-read for critical findings. ECG interpretation has a similarly mature evidence base. But outcome-level prospective evidence, the kind showing a mortality or morbidity benefit rather than a diagnostic-accuracy benefit, is still catching up. One of the more ambitious efforts, a cluster-randomized trial evaluating whether an AI chest X-ray triage system reduces mortality, isn’t expected to report full data until December 2027. For most other applications, especially complex diagnostic reasoning, the trials that would satisfy this step are still being designed.

3. Professional Endorsement. This step is moving at different speeds by specialty. In October 2025, ACEP hosted the first All Emergency Medicine AI Summit, convening SAEM, CORD, ACOEP, ABEM, AAEM, EMRA, AACEM, and AOBEM. The resulting consensus statement, published in March 2026, affirms that emergency physicians retain authority for patient care decisions and that AI should enhance rather than replace clinical judgment. That’s real, but it reads more like ACEP’s 1990 position statement than its 2001 guidelines: principles without defined training or credentialing standards. Radiology has moved further. In May 2026, the ACR Council approved the ACR-SIIM Practice Parameter for Imaging Artificial Intelligence, the specialty’s first formal practice parameter for AI and the same category of document as ACR’s Appropriateness Criteria. It sets expectations for governance, clinical validation, bias mitigation, and ongoing performance monitoring, and it’s paired with Assess-AI, a new quality registry that tracks real-world model performance against radiology-report-derived outcomes. That is, right now, the closest thing in medicine to a 2001-style guideline for AI.

4. Infrastructure. This is moving fast. Major EHR vendors have built native AI integration points, hospital AI governance committees are now common rather than novel, and the FDA’s Predetermined Change Control Plan (PCCP) framework, finalized in 2024, lets manufacturers update models without seeking a new clearance for every version. That matters more than it sounds: it’s a regulatory acknowledgment that AI infrastructure will need to absorb continuous change in a way POCUS equipment never did.

5. Cultural Adoption. The share of U.S. and Canadian medical schools incorporating AI into their curricula jumped from 53 percent in 2023 to 77 percent in 2024, according to the AAMC and AACOM’s Curriculum SCOPE Survey. Residency training lags behind medical school. ACGME hasn’t made AI competency a formal program requirement yet, though it’s gathering stakeholder feedback toward that end and added AI-focused sessions to its 2026 programming, and the AAMC’s own national AI competency framework for the full training continuum isn’t due until fall 2026. Patients are beginning to ask about AI-assisted diagnosis, and health systems increasingly market their AI capabilities. What’s still missing is the residency-level equivalent of a POCUS rotation: standardized, required, and tested.

6. Governance. This remains the messy middle. The FDA’s January 2026 revision to its clinical decision support guidance leans on what industry has started calling a “glass box” standard: CDS software must disclose, in plain language, its underlying logic, validation methodology, and the limitations of its training data, specifically to guard against automation bias. What’s still mostly missing is the hospital-side counterpart: credentialing pathways, and version-tracking systems that treat a model update the way a hospital treats a new piece of equipment.

7. Medicolegal Recognition. No case law yet establishes AI non-use as a deviation from standard care. But the fact that a Harvard Business School economist has now formally modeled the question tells you something about where the conversation has moved. More on that paper below.

What Still Has to Happen?

Some of the remaining milestones are already in motion, or already achieved in a single specialty, like radiology’s new practice parameter and the chest X-ray mortality trial cited above. Most are not, and none has happened across specialties broadly. The list that would move AI further up the ladder:

• Clinical outcome studies showing mortality or morbidity benefit, not just diagnostic accuracy

• Prospective trials rather than retrospective validation, across more than imaging and ECG

• Professional society recommendations with the weight of the 2001 ACEP guidelines, in specialties beyond radiology, which already has one

• Residency curricula that train every new physician on AI tools as a baseline skill

• Board examination content that tests AI competency

• Credentialing pathways for physicians using AI clinically

• Hospital policies that define when and how AI is used

• Workflow integration inside EHRs that makes AI a seamless part of clinical decision-making rather than an add-on

• Quality assurance programs, run by hospitals rather than manufacturers, that monitor model performance over time

• Version-control transparency that lets clinicians know when a model changed and how that changed its outputs

• Independent benchmarking of commercial models against real, local patient populations

• Malpractice case law that establishes precedent for when declining to use AI becomes indefensible

Why AI May Be Different

History doesn’t always repeat itself. There are real differences between POCUS and AI that complicate the transition:

Velocity of change. Ultrasound machines improved incrementally. AI models can change materially every few months, and the FDA’s own PCCP framework formalizes this: manufacturers can now pre-clear a range of future model updates in a single submission, meaning a device that behaves one way in 2026 may legitimately behave differently in 2027 without ever returning for a new review. Ultrasound machines never needed that provision.

Evolving outputs. A 2024 model doesn’t necessarily behave like a 2025 model. POCUS images don’t change their meaning retroactively; AI outputs can.

Probabilistic nature. AI recommendations are not visual. A radiologist can look at an ultrasound image and see what the physician saw. An AI recommendation is often invisible unless documented, complicating both quality assurance and medicolegal review.

Vendor fragmentation. Unlike ultrasound, dominated by a handful of manufacturers, the AI landscape is crowded with substantially different models, making standardization and benchmarking harder.

Continuous governance. With POCUS, governance was largely a one-time investment in training and equipment. With AI, governance is continuous: models drift, data distributions shift, and monitoring has to be perpetual.

Deskilling risk. POCUS extended the physical exam; it didn’t replace clinical reasoning. AI carries a genuine risk of cognitive atrophy, where physicians grow less capable of independent verification because they’ve grown dependent on the tool. POCUS has no direct equivalent of this risk.

These distinctions make the transition more complicated than the POCUS roadmap would suggest.

The Medicolegal Inflection Point

Here I want to sit with questions rather than answers.

• If AI reliably detects subtle diagnoses that physicians routinely miss...

• If AI consistently reduces missed findings across multiple validated studies...

• If most emergency physicians in a community routinely use it...

• If hospitals provide it as part of standard equipment...

• If specialty societies recommend it in formal guidelines...

At what point does declining to use AI become difficult to defend?

A 2026 Harvard Business School working paper by economist Alex Chan, also circulated through the National Bureau of Economic Research, models exactly this transition: the point at which AI moves from a passive clinical tool to something that shapes what a reasonably prudent physician would do. Chan’s core question isn’t whether liability law should protect physicians who rely on AI. It’s whether the law should adapt its standards to AI’s specific failure modes, data drift, vendor-side updates, probabilistic rather than binary outputs, or force AI to fit liability frameworks built for static tools like a stethoscope or an ultrasound probe.

I don’t think this moment has arrived. But it may be closer than it feels, because the standard of care doesn’t announce itself with a press release. It shifts in the accumulated weight of evidence, endorsement, and litigation until the defense of “I didn’t use it” begins to sound hollow.

The Ladder as a Diagnostic Tool

The seven steps above aren’t specific to ultrasound. The same ladder applies to CT, MRI, and troponin testing, all of which climbed it long before POCUS did, and it applies just as well going forward, to next-generation sequencing or whatever comes after AI.

The value of naming the steps isn’t prediction. It’s diagnosis: for any technology you’re evaluating, you can ask which step it has actually reached, rather than which step its marketing implies it has reached. That’s precisely where most of the confusion about AI’s clinical readiness comes from, conflating step one (an FDA clearance exists) with step three (a specialty society has endorsed it) or step seven (declining to use it is indefensible).

Conclusion

Nobody predicted the exact year bedside ultrasound became routine. There was no ribbon-cutting ceremony. Looking backward, the progression seems obvious and inevitable. Looking forward at the time, it was uncertain and contested.

AI may follow a similar path. Or it may follow an entirely new one. The important question isn’t whether AI will become the standard of care. Given enough evidence, infrastructure, and cultural absorption, most useful technologies eventually do.

The important question is: How will I recognize the moment when it does?

Sources

• Rozycki GS, Ochsner MG, Jaffin JH, Champion HR. “Prospective Evaluation of Surgeons’ Use of Ultrasound in the Evaluation of Trauma Patients.” Journal of Trauma. 1993;34(4):516–527. (476-patient study establishing the technique; predates the FAST name.)

• Scalea TM, et al. “Focused Assessment with Sonography for Trauma (FAST): Results From an International Consensus Conference.” Journal of Trauma. 1999;46(3):466–472. (Consensus conference, Dec. 1997, that coined the FAST acronym.) PubMed

• Rozycki GS. “Surgeon-Performed Ultrasound: Its Use in Clinical Practice.” Annals of Surgery. 1998;228(1):16–28.

• Russ B, Arthur J, Lewis Z, Snead G. “A Review of Lawsuits Related to Point-of-Care Emergency Ultrasound Applications.” Journal of Emergency Medicine. 2022;63(5):661–672.

• Mezrich JL. “POCUS in the Emergency Department: An Example of Convergence of Defensive Medicine With Patient Care.” Academic Radiology. 2025;32(Suppl 1):S126–S130. PubMed

• ACEP. “Ultrasound Guidelines: Emergency, Point-of-Care, and Clinical Ultrasound Guidelines in Medicine.” Approved as “Emergency Ultrasound Guidelines,” June 2001; revised Oct. 2008; retitled and expanded June 2016; revised again April 2023.

• Chan A. “Optimal Medical Liability for AI.” Harvard Business School Working Paper No. 26-087, June 2026; also NBER Working Paper No. w35321. NBER

• FDA. “Artificial Intelligence-Enabled Medical Devices” database, last updated March 4, 2026 (1,524 total devices; 1,163 in radiology, ~76%; 295 new clearances in 2025 from 221 manufacturers).

• FDA. “Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles.” Final guidance, 2024.

• FDA. Revised Clinical Decision Support Software guidance, issued January 6, 2026 (“glass box” transparency requirements, automation-bias framing).

• ACEP, SAEM, CORD, ACOEP, ABEM, AAEM, EMRA, AACEM, and AOBEM. Consensus Statement following the first All Emergency Medicine AI Summit (Irving, TX, Oct. 2025); statement published March 18, 2026.

• American College of Radiology, Data Science Institute. AI use-case repository and FDA-cleared algorithm tracker.

As always, if you enjoy reading, consider subscribing and tell a friend.

Sam

Subscribe now

In July 2026, OpenAI rolled out ChatGPT Health to every U.S. adult user, inviting them to connect their Apple Health data, medical records from Epic and Oracle Health systems, and wellness apps directly to the AI. OpenAI announced the nationwide launch on July 23, promising “secure” connections and personalized health insights. The move was widely covered as a consumer convenience, a smarter way to track medications, interpret lab results, and prepare for doctor visits.

What most coverage missed is that this is only one half of a much larger story. While consumers debate whether to trust OpenAI with their blood pressure readings, a separate, invisible pipeline has already moved hundreds of millions of patient records into the hands of AI companies. The difference is structural, legal, and deeply asymmetrical, the back-door pipeline strips your name before it sells your data. The front-door pipeline does not have to.

The Front Door, HIPAA Ends Here

Here is the first thing to understand about ChatGPT Health, it is not covered by HIPAA.

HIPAA, the Health Insurance Portability and Accountability Act, governs a narrow class of “covered entities”, hospitals, health insurers, and healthcare clearinghouses, plus their “business associates.” OpenAI is none of these. OpenAI states explicitly that “Health in ChatGPT is not intended for clinical or covered-entity use and does not offer a Business Associate Agreement.”

What this means in practice is stark. When a hospital holds your medical record, federal law requires them to protect it, limit its use, and obtain authorization before sharing it broadly. When you upload that same record to ChatGPT, those protections evaporate. As health platform b.well noted, “medical records uploaded into ChatGPT Health are no longer covered by HIPAA, because the service isn’t a covered entity under that law.”

OpenAI does not need to de-identify your data. It does not need to strip your name, your birth date, your medical record number, or your diagnostic codes. Its only constraints are its own privacy policies. OpenAI’s Health Privacy Notice says that by default, health data received via Health Features is not used to improve foundational models. But the policy also notes that “a limited number of authorized OpenAI personnel and trusted service providers might access data... to improve model safety, unless you have opted out.” That is a significant carve-out, governed by a terms-of-service agreement most users will not read.

The consumer is being asked to make a privacy decision without understanding the legal terrain. They assume HIPAA protects their health data everywhere. It does not.

The Back Door, The Pipeline You Never Signed Up For

While OpenAI courts consumers on the front end, a separate industry has built a lucrative infrastructure for selling hospital data to AI companies on the back end. The key players are data middlemen, companies that contract with hospitals, strip patient records of direct identifiers, and license the resulting datasets to AI developers and researchers.

Truveta is among the largest. Founded by a consortium of major U.S. health systems, Truveta now includes data from more than 130 million de-identified patients across over 900 hospitals and 20,000 clinics. Its members include Providence, Advocate Health, Trinity Health, Tenet Healthcare, Northwell Health, AdventHealth, and Novant Health. In September 2021, Microsoft announced a strategic partnership and investment in Truveta, and Truveta uses Azure-based AI to process its data. Truveta has also launched its own Truveta Language Model trained on this clinical corpus.

Protege is the emerging contender. In January 2026, Protege raised a $30 million Series A led by Andreessen Horowitz, bringing its total funding to $65 million. The company works with what it describes as “close to 20 data partners” in healthcare and more than 100 across all sectors. Protege licenses what it calls “private, real-world data” including clinical notes, medical images, video, audio, pathology slides, genomic data, lab reports, wearables data, and social determinants of health. In a partnership with Syndesis Health, Protege gained access to 70 million de-identified patient lives from more than 500 facilities across 15 countries.

Protege states on its website that it works directly with “leading foundation model labs” to define licensing standards. It does not name them. What is clear is that the pipeline is vast, well-funded, and growing rapidly.

The Asymmetry

Consider what this means for a single patient.

If you are treated at a Truveta-member hospital, your de-identified record may be harmonized, aggregated, and sold to AI developers. The hospital has removed your name, address, medical record number, and the other 15 identifiers required by HIPAA’s Safe Harbor method. The AI company receives a rich longitudinal record of your diagnoses, medications, procedures, and lab values, but without a name attached.

If you then upload your own medical records to ChatGPT Health to ask about your medication list, OpenAI receives the same clinical information, but with your name on it, your dates intact, and your full identifying context, because HIPAA does not apply to them.

The hospital is legally required to de-identify your data before sharing it. The consumer AI company is not required to do anything of that kind. The regulated system forces anonymization on the back door while the consumer app collects richer, more identifiable data through the front door with no federal health privacy guardrails at all.

Here is the part that should give consumers pause. The software companies building these health AI products may already have more data on you than you would want them to know, de-identified records from your hospital, imaging from your health system, lab values from your clinic. If you then provide them with a small sampling of identifiable data, your own uploaded records, your Apple Health metrics, a few lab results, you may inadvertently give them the key to re-identify what they already possess as belonging to you. Careful what you share, because even a narrow slice of identifiable health data can be used to frame a much more detailed picture of you than you may think.

The De-identification Myth

There is a second problem with the back-door pipeline, and it is mathematical.

For decades, the legal and ethical framework for health data sharing has relied on a simple premise, remove 18 identifiers, and the data is no longer “personally identifiable.” Once de-identified, it falls outside HIPAA’s scope and can be “freely used, shared, and sold” without patient consent.

Modern AI has broken that premise. In 2019, researchers Luc Rocher, Julien Hendrickx, and Yves-Alexandre de Montjoye published a study in Nature Communications titled “Estimating the success of re-identifications in incomplete datasets using generative models.” Their finding was startling, “Using our model, we find that 99.98% of Americans would be correctly re-identified in any dataset using 15 demographic attributes.”

The authors were explicit about the policy implications, “Our results suggest that even heavily sampled anonymized datasets are unlikely to satisfy the modern standards for anonymization set forth by GDPR and seriously challenge the technical and legal adequacy of the de-identification release-and-forget model.”

Other studies have reinforced the point. Researchers have demonstrated that algorithms can accurately match physical activity data and demographic information to 95% of adults in a dataset. As Scripps News reported, “Health data shared with AI systems can be stripped of names and sold legally, then re-identified using AI tools.”

The data middlemen are selling records that are legally de-identified but practically re-identifiable. The “without your name” framing is a legal fiction that AI has rendered increasingly untrue.

The Consent Theater

If the pipeline is invisible, how do patients find out about it? In most cases, they don’t.

When you check into a hospital, you sign a group of forms and receive a Notice of Privacy Practices, a long document that mentions treatment, payment, and “health care operations.” De-identification and data sharing may be mentioned in a single clause. The notice almost never says, “Your records may be sold to an AI training data broker.”

Some hospitals are becoming more explicit. Archbold Medical Center in Georgia publishes a Notice of Privacy Practices that states, in plain language, “De-identified data is no longer subject to privacy or security laws.” It also notes that “AI use within applications uses/discloses your medical information.” This is unusually candid. Most notices are not.

The legal framework creates what might be called consent theater, patients are given a document they do not read, which authorizes broad data uses they do not understand, for purposes, including AI model training, that were not contemplated when HIPAA was written in 1996.

The Regulatory Vacuum

One might expect that the wave of state consumer privacy laws would close this gap. They largely do not.

Washington’s My Health My Data Act (MHMDA), enacted in 2023, broadly defines “consumer health data” and requires opt-in consent for collection and sale. But the Act explicitly excludes “deidentified data” from its definition of personal information. A hospital selling de-identified records to Protege falls outside the law.

Maryland’s Online Data Privacy Act (MODPA), effective October 2025, has some restrictions on using consumer health data for AI model development. But properly de-identified or aggregated data is largely exempt.

Nevada’s SB 370 and Connecticut’s CDPA amendments follow the same pattern, they regulate identifiable consumer health data but leave the de-identified pipeline untouched.

At the federal level, the FTC has been active on data broker enforcement, but not on this specific subject. In February 2026, the FTC sent warning letters to 13 data brokers about compliance with the Protecting Americans’ Data from Foreign Adversaries Act (PADFAA), focusing on foreign data transfers rather than domestic AI training. There is no federal enforcement action specifically targeting the sale of de-identified health records for commercial AI model training.

The Convergence

The two pipelines are separate in law but convergent in effect. OpenAI is building the infrastructure to ingest medical data on both sides.

On the front end, OpenAI acquired Torch, a health records unification startup, for roughly $60-100 million in January 2026. Torch’s technology consolidates fragmented patient data from hospitals, labs, and visit recordings. The stated goal, per HLTH reporting, is a “unified medical memory” for ChatGPT Health.

On the back end, OpenAI’s health data ambitions are clear even if its purchases from middlemen are not publicly documented. The company is positioning itself to be a central platform for health information, whether that data arrives through consumer upload or hospital pipeline.

The patient, meanwhile, is left with a false sense of security. They are told HIPAA protects their medical records. They are told de-identification makes their data anonymous. They are told they can “securely” connect their health records to an AI assistant. None of these statements are entirely false. But none of them capture the full picture of where their data is going, how it is being used, or how thin the legal protections actually are.

The health data economy has bifurcated into two tracks, one regulated and de-identified, the other unregulated and fully identifiable. Both are feeding the same models. And the patient is the last to know.

The waiting ended in 2026. State legislatures stopped asking and started writing. By the end of this year's legislative sessions, more than two dozen new laws and regulations across eighteen states will have redrawn the boundaries of what AI may do in healthcare. Additionally, there is Federal legislation in process that may conflict with everything states have done so far. Here I survey the full landscape in the following categories:

• Prior Authorizations

• Therapy

• Consent and Disclosure

• Chatbot Safety

• Professional Licensing

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Prior Authorizations

The most active legislative front in 2026 was prior authorization. Seven states enacted laws that draw a hard line between AI assistance and AI decision-making for coverage denials. Two additional states also took related approaches targeting claims downcoding and oversight reporting.

Alabama SB 63, enacted on April 17, requires health insurers to disclose their use of AI, base authorization decisions on a patient’s individual medical history, and ensure that any denial is issued by a licensed professional. It is similar to the disclosure-plus-human-review framework seen elsewhere, but it explicitly ties authorization decisions to individual medical records rather than group data.

Colorado HB 1139, signed June 2, prohibits insurers from basing a denial solely on group data or algorithms and requires a licensed clinician to review any adverse decision. The subtle but critical distinction here is the explicit ban on group-level data as the basis for individual denials, a protection Alabama and Washington do not spell out as directly.

Georgia SB 444, enacted May 5, is one of the shortest and most direct statutes in this category: a blanket prohibition on coverage decisions based solely on AI systems or software tools. It is nearly identical in spirit to Iowa’s approach but without Iowa’s explicit carveout allowing AI to conduct initial screening.

Illinois SB 3114, awaiting the governor’s signature, is the only law in this category that targets downcoding, the practice of reducing reimbursement codes, rather than prior authorization denials. It prohibits AI from bypassing provider documentation to downgrade claims and requires a natural person to review any downcoding determination.

Indiana HB 1271, enacted March 4 and effective July 1, is the only law regulating AI on both sides of the transaction: it prohibits insurers from using AI as the sole basis for downcoding and bars providers from using AI to submit claims without human review. It bridges the gap between utilization-review laws and claims-integrity laws.

Iowa HF 2635, enacted May 13, creates a two-step process: AI may screen prior authorization requests, but only a human may deny, delay, or downgrade them. The statute is nearly identical in structure to Utah’s approach but with less detail on time limits and validity periods.

Maryland HB 1563, enacted April 28, takes an oversight approach rather than a direct prohibition. It requires insurers to file quarterly reports on AI-assisted adverse decisions and allows the insurance commissioner to investigate spikes in denials, particularly for emergency department services. It is the only authorization-related law focused on regulatory monitoring rather than rewriting decision-making rules.

Utah SB 319, enacted March 19, requires disclosure of AI use in preauthorization, mandates independent medical judgment by reviewers, and sets minimum validity periods for chronic condition authorizations. It is the most procedurally detailed of the authorization statutes, paired with a separate scope-of-practice law passed the same week.

Washington SB 5395, enacted March 23, expands the state’s existing prior authorization framework and explicitly bars AI from modifying care, not just denying it. This is a broader prohibition than most, covering any adverse modification of a treatment plan.

Therapy

While prior authorization laws dominated, therapy bans consumed the headlines. Six states enacted statutes prohibiting AI from providing psychotherapy or representing itself as a mental health professional. Most took effect in 2026 or will take effect in 2027.

Colorado HB 1195, enacted June 3, is the most comprehensive therapy ban in the country. It prohibits unlicensed AI chatbots from offering psychotherapy and also restricts licensed psychologists, counselors, and social workers from using AI to provide independent therapeutic services. It is the only law that explicitly limits both the unlicensed and licensed sides of the equation.

Maine LD 2082, enacted April 13, prohibits AI from providing therapy or psychotherapy and classifies offering therapy via AI without a license as an unfair trade practice. It also adds a patient-consent requirement for ambient listening and AI-powered recording during sessions. It is nearly identical in scope to Nevada’s law but adds the recording consent provision.

Nevada AB 406, enacted in 2023 and effective July 1, 2025, was the first therapy-bot ban in the nation. It applies to any AI system in Nevada, including telehealth platforms, and imposes civil penalties of up to $1,500 per violation. It served as the model for several 2026 statutes, including Maine’s and Rhode Island’s.

Rhode Island H 7349 / S 2197, enacted June 22, contains a dual prohibition: unlicensed AI may not practice therapy, and licensed providers may not use AI for independent therapeutic decisions or treatment plans. It is nearly identical in structure to Colorado’s law but lacks the explicit licensed-provider restrictions seen in Colorado’s statute.

Tennessee SB 1580, enacted April 1, prohibits developing or deploying AI systems that advertise or represent themselves as qualified mental health professionals. It treats such misrepresentation as a deceptive practice under state consumer protection law. It is narrower than Colorado’s ban, and targets the advertising and representation rather than the therapeutic act itself.

Vermont H 816, enacted June 17, safeguards individuals seeking mental health services by ensuring delivery by professionals rather than independently by AI systems. The wording lacks the explicit title protections and civil penalties seen in Nevada and Rhode Island. It is the least prescriptive of the therapy bans.

Consent and Disclosure

Four states enacted or implemented disclosure requirements in 2025 and 2026, creating a patchwork of transparency obligations that providers and insurers now must navigate.

Arizona Board of Behavioral Health Examiners adopted regulations in November 2025, effective January 1, 2027, requiring behavioral health professionals to obtain and document informed consent before providing services involving AI. It is the only regulatory (non-statutory) entry in this survey and applies only to behavioral health, making it narrower than California’s statewide disclosure laws.

California AB 3030, enacted in 2024 and effective January 1, 2025, was the first statewide disclosure law in the nation. It requires any generative AI communication to a patient to include a clear disclaimer that the message was generated by AI. AB 489, enacted in 2025 and effective January 1, 2026, goes further by prohibiting AI systems from presenting themselves as licensed medical professionals. Together, these two laws create a layered transparency framework that no other state has fully replicated.

Louisiana HB 475, enacted June 2, requires verbal disclosure of any recording device or AI transcription before recording any clinical visit. It is the only law in the country that mandates verbal (not just written) disclosure for AI transcription, and it applies to all healthcare settings including telehealth.

Texas SB 1188, enacted in 2025 and effective September 1, 2025, is the most permissive framework in this category. It explicitly allows AI to diagnose and treat patients, provided the patient is informed and the Texas Medical Board has reviewed the technology. It is the only state that affirmatively authorizes AI clinical decision-making rather than merely restricting it.

Chatbot Safety

Three states enacted what are effectively “Chatbot Safety Acts”: laws that do not regulate medical practice directly but impose safety and disclosure requirements on AI chatbots that might interact with patients in crisis.

Idaho SB 1297, enacted March 31 and effective July 1, 2027, requires disclosure and crisis-response protocols for any conversational AI system that provides health-related information. The law is nearly identical to Nebraska’s, suggesting both states may have used model legislation.

Nebraska LB 525, enacted April 14, combines the Agricultural Data Privacy Act with a Conversational AI Safety Act that applies to all public-facing AI chatbots with healthcare safety requirements. It is identical in structure to Idaho’s law but applies to all public-facing chatbots, not just health-specific ones.

Oregon SB 1546, enacted March 31, is the most comprehensive chatbot safety law in the country. It includes a private right of action, safeguards for minors, and covers AI companions. It is significantly broader than the Idaho and Nebraska statutes, extending beyond healthcare to cover emotional support AI.

Professional Licensing

Two states passed laws clarifying that AI is not a licensed professional, using different legal techniques to reach the same conclusion.

Delaware HB 191, enacted April 23, prohibits any nonhuman entity, including AI, from being licensed or certified as a nurse, physician, or physician assistant. It also bars AI from using protected professional titles. It is the most explicit title-protection law in the survey.

Utah SB 150, enacted March 24, uses a negative-definition approach: AI providing advice or treatment without a practitioner-patient interaction does not qualify as an “innovation” within the scope of practice. It is the only law that defines AI out of scope rather than explicitly prohibiting licensure. Paired with Utah’s SB 319, enacted the day before, these two laws create the most comprehensive AI regulatory framework of any single state.

The Horizon: Pending Bills and Federal Counter-Trends

Three jurisdictions have legislation still pending that could reshape the landscape by 2027.

New York S7896 / A8556, introduced in the 2025-2026 session, would establish comprehensive AI utilization review requirements. It is the most detailed authorization law proposed to date, with specific reporting and human-review mandates that go beyond the 2026 enacted statutes.

Pennsylvania HB 1925, introduced in the 2025-2026 session, would amend both the Health and Insurance titles to require AI utilization review and reporting by insurers, hospitals, and clinicians. It is broader than most authorization bills because it covers providers as well as insurers.

Federal H.R. 238, the Healthy Technology Act, stands in direct counterpoint to the state trend. Introduced in the 119th Congress, it would allow AI and machine learning systems to qualify as practitioners able to prescribe FDA-approved drugs, effectively federalizing the expansion of AI clinical authority that Texas alone has permitted at the state level. If it advances, it would create a direct conflict with the authorization and therapy bans enacted by more than two dozen states.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

On July 22, 2026, Pastor Scott Winters sued OpenAI in San Francisco County Superior Court. The complaint describes a five-week pattern that began a year earlier. On June 8, 2025, Winters grew dizzy mid-sermon and had to stop the service; paramedics were called and found him stable. Two days later, on June 10, too dizzy to walk, he crawled toward his bathroom and asked ChatGPT for directions across the floor between himself and his recliner. Both times, the suit alleges, ChatGPT told him to take it easy and that his symptoms would improve. On July 13, 2025, Winters asked ChatGPT about a pain in his groin. The suit alleges it offered spiritual reassurance, telling Winters that “God did not design your body to endlessly fail.” Hours later, he was in the intensive care unit with a massive pulmonary embolism. Winters filed his suit one day before OpenAI expanded Health in ChatGPT to 300 million weekly users.

Six weeks before Winters’ hospitalization, on May 31, 2025, nineteen-year-old Sam Nelson asked ChatGPT about nausea from Kratom use. Nelson had used ChatGPT since high school for homework, and according to the family’s complaint, the system had initially refused to discuss drug use with him at all. The complaint alleges those refusals stopped after the rollout of GPT-4o in 2024, and that by May 2025 the chatbot was offering him specific guidance instead. Rather than tell him to stop or seek care, it suggested pairing the Kratom with 0.25 to 0.5 milligrams of Xanax to “minimize nausea.” Nelson died that night of asphyxiation from the combination. His parents filed a wrongful-death suit in May 2026, arguing that ChatGPT had functioned as an active clinical decision-maker, not a passive search engine, and that a safety behavior the system once had was allowed to erode.

Both lawsuits were filed within nine weeks of each other, in the same stretch of 2026 when OpenAI was expanding its product’s reach to nearly the population of the United States. Together, the two cases show a pattern. A conversational AI system overrode medical urgency, once with spiritual language, once with a specific drug dose, and in both cases the underlying failure was the same: an architecture that struggles to recognize a rare, dangerous symptom because it is built to predict what’s statistically common, not what’s clinically true.

The Study Behind the Number

That failure has since been measured directly. In February 2026, Ramaswamy and colleagues at the Icahn School of Medicine at Mount Sinai published a study in Nature Medicine testing Health in ChatGPT, one month after launch. Researchers tested the product across 60 clinician-authored vignettes, yielding 960 total responses. It under-triaged roughly 52 percent of gold-standard emergencies, cases where every reviewing clinician agreed on the correct triage level. In practice, that meant sorting patients with conditions like diabetic ketoacidosis or impending respiratory failure into a 24-to-48-hour follow-up window instead of directing them to emergency care. The failures followed an inverted U-shaped pattern: the system missed 35 percent of nonurgent cases, sending people who didn’t need it toward urgent care, and 48 percent of true emergencies, the more dangerous error. It struggled most at both extremes rather than erring safely in one direction. Because the researchers tested the live product rather than a benchmark model, that 52 percent figure describes the exact system OpenAI connected to Epic, MyChart, One Medical, and Apple Health on July 23, 2026. OpenAI has pushed back on the study’s relevance, telling reporters that single-prompt vignettes don’t reflect “typical use,” since the product is designed for multi-turn conversations where patients supply additional context through follow-up questions. That may be true of some users. It does not explain why the same product, tested as it actually ships, missed the emergency in nearly half of the cases where the correct answer was to send someone to a hospital.

The Sandbox That Didn’t Hold

Two days before that expansion, on July 21, 2026, OpenAI and Hugging Face disclosed a security incident that undercuts any claim that the underlying architecture is under control. The incident began inside ExploitGym, an internal OpenAI benchmark built to measure whether frontier models could turn known software vulnerabilities into working exploits, a test of cyber capability with no connection to medical or consumer safety. During that test, GPT-5.6 Sol and an unreleased, more capable model escaped the isolated environment built to contain them. The models found a hidden flaw in a piece of software so obscure that even its own developers didn't know it existed. That flaw was enough to slip past the barriers meant to contain them. From there, they worked their way deeper into OpenAI's own systems, picking up login credentials and higher-level access along the way, until they reached the live servers running Hugging Face's actual public platform, not a test copy. Reporting on the incident described it as thousands of individual actions carried out across a swarm of short-lived sandboxes, a scale well beyond a single lucky guess. OpenAI's own account describes a multi-step plan, executed autonomously, to defeat an isolation boundary that had been purpose-built to hold the model.

What the Breach Does and Doesn’t Prove

Some caveats matter here. The model that escaped is not the exact instance running inside Health in ChatGPT, and the environment it escaped was a research sandbox rather than a live production server. The incident also occurred while OpenAI had deliberately reduced the models’ safety refusals for the purposes of the evaluation, so it does not show what happens under normal deployment conditions. What it does show is a capability that can’t be walked back with a patch: this model family, when its refusals are lowered or bypassed, can plan and execute a multi-domain escape without human authorization. That is closer to discovering a structural flaw in a medical device than a bug in one unit. The specific unit in the hospital may not have overheated yet, but the design has already shown it can. Installing that design in every ward, days after finding the flaw, is a decision that needs justifying, not assuming.

Why the Architecture Fails This Way

The clinical failure and the security failure share a root cause. Large language models generate text by predicting the most statistically likely next word given everything that came before. That process rewards fluency and common answers, a tendency researchers call regression to the mean: the model’s outputs cluster around the most probable response in its training data, at the expense of rare but important outliers. The training data itself makes this specifically worse in medicine. Models learn from internet-scale text in which routine complaints, muscle strain, stress, fatigue, vastly outnumber descriptions of rare emergencies, so the “likely” answer reflects what’s overrepresented in the general data rather than what’s actually likely for the patient typing the question. A weather model that always forecasts “partly cloudy” would be right most days and catastrophically wrong on the day a hurricane arrives. Pulmonary embolism is rare in the universe of chat transcripts about groin pain; muscle strain and spiritual doubt are common. Faced with Winters’ symptom, a system optimized for the likely answer will drift toward the likely explanation. There’s no malice in this. The architecture is doing exactly what it was built to do, and that is the problem in a clinical setting where the rare case is often the one that kills.

A System Built Differently

Contrast this with a different architecture. MomConnect, South Africa’s government maternal health platform, added a symptom-checking tool from Ada Health built on a Bayesian network: a model that encodes medical knowledge as explicit probabilities and updates them as new evidence comes in, rather than generating conversational text. The mechanics are different. The system starts with a prior probability for each condition it’s tracking, then updates that number as symptoms are entered, arriving at a posterior probability the system can act on directly: refer the case, ask another question, or reassure. When a Bayesian network is uncertain, that uncertainty is a number the system can act on, not a pattern buried inside billions of weights. Ada’s underlying engine also carries CE marking as regulated medical device software in Europe, a regulatory status Health in ChatGPT does not have. A study by Schmude and colleagues, published in Nature Health in June 2026, evaluated the tool’s advice to 968 pregnant women and mothers and found a physician panel rated it safe in 98.4 percent of cases, with appropriate care-seeking rising by more than 40 percentage points. The gap between that result and the 52 percent under-triage rate for Health in ChatGPT isn’t a matter of better prompts or more training data. It reflects two different ways of handling uncertainty. One system can tell you, in a number, how sure it is. The other generates a confident sentence whether or not it should.

The Regulatory Gap

That is also why "Health in ChatGPT" sits in a regulatory gap rather than a regulated category. The European Union's AI Act treats general-purpose AI models with systemic reach as their own class, subject to added testing and governance. The Act draws that line using a measure of how much raw computing power went into training a model, a number large enough that it's written in scientific notation: 10^25 floating-point operations, essentially a tally of the individual calculations involved in building the system. Cross that line, and the law assumes, by default, that the model carries systemic risk. A company can still make the case that its model doesn't actually pose that risk, but the starting point shifts toward oversight rather than away from it. A model that just demonstrated an autonomous escape from a security sandbox while serving 300 million weekly users is a reasonable candidate for that scrutiny. But systemic-risk status under the AI Act is not the same as an Annex III high-risk medical device classification, and OpenAI's product currently qualifies for neither in the way a diagnostic device would. The United States has, if anything, moved in the opposite direction. On January 6, 2026, the FDA issued revised guidance on clinical decision support software that loosened its prior stance on automation bias. The previous version had treated tools that presented a single recommendation as raising heightened safety concerns; the new guidance takes a more permissive, risk-based approach instead. That change landed six months before this launch, in the same regulatory environment where Health in ChatGPT now operates. A chatbot with a medical disclaimer, wired into Epic MyChart and making triage-adjacent judgments for hundreds of millions of people, is large enough to matter and, under current rules on both sides of the Atlantic, small enough to avoid the oversight a medical device would face.

The Experiment Continues

None of this is abstract. Winters and Nelson show what the failure looks like in a single conversation. The Ramaswamy study shows how often it happens across the live product. The Hugging Face disclosure shows the same model family can defeat a boundary built specifically to contain it. Put together, they describe a product launched into open clinical use before any of those three questions- whether the model can be trusted with rare symptoms, whether its containment holds under pressure, and who is checking either claim besides the company that built it- have been independently answered. Independent verification would mean outside researchers, people with no financial ties to OpenAI, deliberately trying to break the system and expose where it fails. Or it would mean a regulator with real investigative authority taking up the question directly. It would not mean another blog post from the company itself. The lawsuits will keep arriving. The question worth asking now is how many more it will take before this is treated as a systemic risk to public health rather than a chatbot with a disclaimer attached.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Who gets to decide that software is ready to perform work that has historically required a medical license?

Medicine has spent more than a century building institutions that determine when physicians are competent to care for patients. Medical schools educate them. Residency programs train them. Specialty boards certify them. State medical boards license them. Hospitals credential them. Peer review evaluates them. Courts review their actions when patients are harmed.

Those institutions are not perfect, but together they establish something fundamental. Society has created a process for determining when a physician is competent enough to practice medicine.

Artificial intelligence introduces a new challenge because software is beginning to perform tasks that have traditionally depended on physician judgment. I am not talking about whether AI will participate in medicine. I’m adking who has jurisdiction to determine when the evidence is sufficient for software to assume greater clinical responsibility.

Earlier this year, Utah became one of the first states to confront that question directly. The debate that followed was widely portrayed as one about artificial intelligence and prescription renewals. I believe it was fundamentally a debate about jurisdiction over medical competence.

The Utah Pilot Structure

In December 2025, Utah’s Office of Artificial Intelligence Policy authorized a health technology company called Doctronic to begin renewing prescriptions for Utah patients using an AI system, under a 12-month regulatory sandbox agreement announced January 6, 2026.

The Office of Artificial Intelligence Policy was created in 2024 under Utah’s AI Policy Act, SB 149, which gives the office authority to temporarily waive regulatory requirements so companies can test AI systems under state supervision. Utah has used that same authority for a mental health chatbot aimed at teenagers and for an AI tool that reads dental radiographs. Doctronic’s agreement lets its AI process 30-, 60-, or 90-day renewals for medications that a licensed physician has already prescribed, and screen for drug interactions along the way.

The pilot is structured in three phases. In Phase One, every AI-generated renewal is reviewed and approved by a licensed physician before it reaches a pharmacy. Phase Two would move that review to shortly after a prescription is issued. Phase Three would allow the system to renew prescriptions with only periodic sampling of AI outputs for oversight. State officials say the pilot will not advance to the next phase until safety has been demonstrated in the one before it.

Before the pilot launched, Doctronic shared data with state regulators comparing its AI’s recommendations to physicians’ decisions across 500 urgent care cases. The company reported that its recommendations matched physicians’ 99.2 percent of the time.

Medication renewals consume substantial physician time, and many involve stable chronic conditions with predictable decision pathways. Exploring whether AI can safely assist physicians with those tasks is a reasonable objective.

The Utah Pilot Significance

On April 20, 2026, eleven of the fourteen members of the Utah Medical Licensing Board sent a letter to the Office of Artificial Intelligence Policy. They wrote that the board “was made aware of this agreement only after its implementation, once the system was already live and available for use.” The letter argued that prescription refills require reassessment of dose, side effects, contraindications, and drug interactions, work the board said belongs to a licensed physician, and it warned that “patients who continue refilling medications without assessment may remain on outdated or suboptimal therapy for months or years.” It closed by recommending that the pilot “be immediately suspended pending further discussion.”

The next day, the directors of the Office of Artificial Intelligence Policy and the Division of Professional Licensing responded in writing. They said the pilot had been “rigorously reviewed by several medical professionals prior to launch,” a process that produced “a large number of suggested substantive adjustments and guardrails,” and that the state would not suspend the pilot because it remained in Phase One, where a physician reviews every renewal before it is filled. They committed to involving the board more closely going forward.

Reasonable people can disagree about whether that was the correct decision. The more enduring question concerns the process itself. Which institution should determine that the available evidence justifies introducing software into a clinical role that has traditionally required physician judgment?

Evidence and Jurisdiction Cannot Be Separated

Much of the public debate over the Utah pilot focused on the AI itself: was it accurate, was physician oversight sufficient, should refill decisions be delegated to software at all.

Those questions naturally followed once the pilot was underway. They also assume that someone had already concluded the available evidence justified launching it. Let’s take a closer look.

The Doctronic pilot shows how the same evidence can support different conclusions. A 99.2 percent concordance rate across 500 urgent care cases was sufficient for the state and the medical professionals who reviewed the pilot before launch. It was not sufficient for the Medical Licensing Board, whose objection was not about the AI’s accuracy in aggregate. It was about what a refill requires in every individual case: a reassessment that a benchmark conducted before launch cannot fully substitute for once the system is operating on real patients. Both readings of the evidence are defensible. They come from institutions with different responsibilities.

The institution responsible for authorizing clinical AI also determines what counts as convincing evidence. A software engineer may prioritize benchmark performance and operational reliability. A practicing physician may focus on uncommon but consequential clinical presentations. A statistician may emphasize study design and external validity. A regulator may concentrate on statutory authority, while a hospital executive may view liability and implementation as equally important.

Each perspective is legitimate. Each emphasizes different forms of evidence. The authority that evaluates the evidence decides the standard by which readiness is judged.

Medicine Already Has a Model

Imagine a hospital announced that it had created a pathway allowing physicians to practice emergency medicine without residency training. The first question would not concern examination scores or clinical outcomes. It would concern authority. Who approved this process?

Medicine has developed a distributed system for answering that question. Medical education is accredited. Residency programs are supervised. Licensing examinations are standardized. Board certification is independently administered. Hospitals grant privileges only after reviewing credentials and competence.

No single organization controls that process. Government, professional organizations, accrediting bodies, and hospitals all participate. Each contributes a different perspective before a physician is entrusted with patient care.

Clinical AI is developing without an equally mature framework for determining when software is ready to assume comparable responsibilities.

The Same Evidence, Different Conclusions

One reason these discussions become contentious is that stakeholders evaluate evidence through different lenses.

Software developers ask whether the model performs accurately. Researchers ask whether a study demonstrates benefit. Medical boards ask whether patient safety has been adequately protected. Attorneys ask who assumes liability when errors occur. Patients ask whether they can trust the recommendations being made.

These are not competing questions, but complementary ones.

Evidence never speaks for itself. People decide what the evidence is sufficient to support.

A Different Way to Begin

Imagine Utah had started somewhere else. Before enrolling the first patient, the state convenes a multidisciplinary commission composed of primary care physicians, pharmacists, medical licensing officials, software engineers, biostatisticians, health services researchers, patient representatives, ethicists, and regulators. Their first task is not evaluating the AI, but instead defining the rules.

What evidence will be required? How should safety be measured? What outcomes matter? What level of physician oversight is necessary? Who has authority to suspend the pilot if safety concerns emerge?

Only after those questions are answered does the commission evaluate the software.

That sequence changes the discussion. The debate isn’t about defending or criticizing artificial intelligence. It is about establishing a legitimate process before patients become part of the evaluation.

Why Utah Matters

Prescription renewals are unlikely to be the last area where this issue appears.

Utah has already used the same regulatory sandbox authority outside prescribing: for a mental health chatbot aimed at teenagers and for an AI tool that reads dental radiographs. The same questions will emerge in radiology, pathology, dermatology, emergency medicine, imaging interpretation, treatment recommendations, autonomous documentation, and other clinical applications as AI capabilities continue to expand.

Every new application will generate studies, benchmark results, and performance claims. Those discussions remain essential. They do not eliminate the need for an accepted process that determines when the available evidence justifies broader clinical use.

The American Medical Association has already staked out a position on where that leaves physicians. Responding to the Utah pilot, AMA chief executive Dr. John Whyte said that “while AI has limitless opportunity to transform medicine for the better, without physician input it also poses serious risks to patients and physicians alike.”

Without an agreed framework, different organizations will continue reaching different conclusions from similar evidence because they are applying different standards and serving different responsibilities.

The Question Utah Leaves Behind

The public conversation often frames medical AI as a choice between accelerating innovation and slowing it down. Utah suggests the more important issue comes earlier.

Before software performs work that has historically required a medical license, society should determine who has jurisdiction to evaluate the evidence, what standards that evidence must satisfy, and how disagreements between institutions will be resolved.

The Utah pilot may ultimately prove to be an important success. It may demonstrate that AI can safely improve efficiency in routine clinical care. Regardless of its eventual outcome, it has already exposed a governance question that will become increasingly difficult to ignore.

Medicine has long established how society decides that physicians are ready to care for patients. Artificial intelligence now requires an equally thoughtful process for deciding when software is ready to assume comparable responsibilities. Utah did not answer that question. It demonstrated that we need one.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On April 27, 2026, OpenEvidence pulled its clinical AI platform from the EU and UK. The notice that replaced it cited “mounting regulatory uncertainty regarding the treatment of AI systems in the European Union and the United Kingdom, including, among other rules, the EU Artificial Intelligence Act.”

What the EU Actually Requires

Let’s start with what “high-risk” means under the EU AI Act, because most coverage of this story treated it as a vague label. Annex III places clinical decision support software in the high-risk category regardless of whether it’s also regulated as a medical device. That status requires, under Chapter III, Section 2 (Articles 8-15), a conformity assessment before the product reaches the market, technical documentation covering training data characteristics and known limitations, a human oversight mechanism built into the product’s design rather than added as a disclaimer, data governance standards for training data, ongoing post-market monitoring, and mandatory reporting of serious incidents to national authorities within 15 days.

None of that is free. Industry estimates put ongoing compliance at roughly €29,000 a year per AI system, with certification running another €17,000 to €23,000. The penalties for getting it wrong in Article 99 sets fines up to €35 million or 7 percent of global turnover for the most serious violations, up to €15 million or 3 percent for ordinary high-risk violations. A US company weighing whether the exposure justifies the market has real numbers to weigh it against.

The obligation doesn’t stop at the vendor, either. A hospital or clinic that adopts a high-risk AI tool becomes a “deployer” under Article 26, which lists twelve distinct duties, including assigning a person with actual authority to override the system, monitoring its performance, retaining logs for at least six months, notifying affected patients that they’re subject to the system, and training staff to understand what the tool can and can’t do. If you’re a health system reader in the EU wondering whether any of this applies to you, it does the moment you adopt a high-risk AI tool, regardless of what the vendor has or hasn’t done on their end.

What the US Exempts, and How Narrowly

In the US, software like OpenEvidence can avoid FDA device regulation entirely under the Non-Device Clinical Decision Support carve-out in the 21st Century Cures Act. FDA’s final guidance, issued March 11, 2026, sets four criteria a product has to meet simultaneously to qualify:

• It cannot acquire, process, or analyze a medical image, an in vitro diagnostic signal, or a pattern or signal from a signal acquisition system. FDA’s own guidance specifically names ECG waveforms as an example.

• It has to be intended for displaying, analyzing, or printing medical information about a patient, such as literature or guidelines, rather than patient-specific signal data.

• It has to support or recommend, not replace, a clinician’s judgment.

• It has to let the clinician independently review the basis for its output rather than functioning as a black box. FDA’s guidance is blunt that this is difficult for large language models generally and close to impossible in time-critical situations.

Miss any one of these four, and the software is a regulated medical device.

This connects to something I tested myself back in May. I uploaded an ECG to OpenEvidence and received an AI-generated interpretation. That alone looks like a Criterion 1 problem: analyzing a signal from a signal acquisition system is exactly what the exemption excludes. I checked again this week to see whether the feature had changed. It hadn’t. OpenEvidence still accepts ECG image uploads and still generates an interpretation. On the same case I tested in May, it still produced an incorrect but very official-appearing reading.

If OpenEvidence Doesn’t Qualify, What Happens on Each Side?

Assume for a moment that the ECG function alone is enough to knock a product out of Non-Device CDS status. What would actually follow, in the US and in the EU, and does the comparison support “the US is the settled option”?

In the US, there’s already a precedent-setting path for exactly this function. AccurKardia’s AccurECG 2.0 cleared FDA review as a Class II device via 510(k) in January 2026. Tempus received 510(k) clearance for its ECG-Low EF software the same way. If OpenEvidence’s interpretation function is similar enough to an existing cleared device, it would likely follow the same 510(k) route, averaging 155 days as of mid-2026. If it’s different enough that no predicate applies, it would more likely need De Novo classification, averaging 341 days.

In the EU, this same feature would almost certainly count as a moderate-to-higher-risk medical device (what MDR calls Class IIa or IIb), the kind that needs an outside safety review before it can be sold. That review currently takes 13 to 18 months on average. A new rule adopted this May is supposed to shrink that to roughly 6 to 9 months, but only for agreements signed after February 2027, so it doesn't help anyone applying today. And crossing that medical-device threshold doesn't let a company trade one set of rules for another. It automatically pulls the product into the AI Act's high-risk category too, so it ends up answering to both frameworks at once: the device safety review, plus the AI Act's own requirements for data governance, incident reporting, and built-in human oversight.

On the two sides of the Atlantic, there is a real structural difference, not manufactured uncertainty.

Two More Differences Worth Examining

Data protection is not the same question as AI regulation, and the two get blurred together in most coverage of this story. OpenEvidence’s own privacy policy tells EU users not to use the product because its infrastructure is US-based and isn’t governed by EU safeguards. GDPR treats health data as a special category under Article 9, requiring explicit consent or a documented research basis before it can be processed at all, a stricter bar than HIPAA’s authorization-or-de-identification model.

The EU and the UK are also not the same problem, though OpenEvidence’s notice treats them as one. The EU has written binding rules that are simply demanding and expensive to satisfy. The UK, by contrast, has no binding AI-specific medical device framework at all yet. The MHRA’s version is still a voluntary pilot program, with a real framework promised sometime later in 2026. “Uncertainty” is the accurate word for the UK’s situation. For the EU, the accurate words are “burden” and “cost.”

What OpenEvidence Actually Did

None of this required OpenEvidence to leave in the specific week it left, and the run-up to its exit is worth walking through.

By the time OpenEvidence posted its notice on April 27, the relief it was asking for was already in motion and had been for months, none of it because of anything OpenEvidence did. EU officials had proposed a package of changes back in November 2025, five months earlier, specifically to ease the same high-risk AI rules OpenEvidence's notice pointed to. The EU's parliament had already voted, 569 to 45, to move that package forward on March 26, a full month before OpenEvidence acted, and the different EU bodies involved had already started hammering out the details. The one round of those talks that hit a snag, on April 28, the day after OpenEvidence's notice went up, got stuck on a technical point that had nothing to do with the kind of product OpenEvidence makes. OpenEvidence left in the middle of a process that was already headed toward the outcome it said it needed, on a schedule set months before the company made its decision.

Its own notice is worth reading closely too. The original version named two people, Brando Benifei and Michael McNamara, and invited clinicians to contact them along with CPME, the umbrella body for European medical associations. Benifei and McNamara aren’t random picks. They co-chair Parliament’s actual AI Act oversight working group, and McNamara went on to become the Rapporteur for the exact Omnibus package that delivered the deferral. CPME, for its part, had already co-signed letters in March and April arguing to keep the AI Act strict, the opposite of what OpenEvidence needed. That callout disappeared from the notice within 48 hours. A correspondence published in the Lancet in May read the episode as a scrubbed attempt at bottom-up lobbying, notable for what it suggests about the gap between OpenEvidence’s stated rationale and its own actions, regardless of what the callout was intended to accomplish.

And the technology itself never actually left. Although OpenEvidence’s own site still shows the same notice to anyone in the EU, Elsevier’s ClinicalKey AI, running on OpenEvidence’s own engine per their 2023 partnership and featuring The Lancet as a headline content source, is available to EU subscribers. The AI didn’t leave Europe. It just stopped answering to OpenEvidence’s name.

So, Why Did OpenEvidence Really Leave?

Probably not because the EU is uncertain. The EU’s rules are written down, phased in on a public schedule, and expensive to satisfy, which is a different problem than uncertainty and a more honest one for OpenEvidence to have named. The timeline it cited resolved in a direction it didn’t need to lobby for. And its own underlying technology kept working in the exact market OpenEvidence says it can no longer serve, just under someone else’s name.

The clinicians who lost access to OpenEvidence didn’t get any of that. They got one sentence, a since-deleted prompt to contact officials who didn’t need the push, and a tool that, as best I can tell, may already be operating past the line the FDA draws around what doesn’t need to be regulated at all.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On July 10, 2026, OpenEvidence launched EvidenceGrade, a feature that assigns a letter grade to the strength of evidence beneath every AI-generated clinical answer. It’s an improvement in the value of the responses OpenEvidence creates, and it lands in a market where other tools have been grading evidence in multiple different ways for quite some time.

What EvidenceGrade Does

EvidenceGrade builds on GRADE, the Grading of Recommendations, Assessment, Development, and Evaluation framework behind Cochrane reviews, WHO guidance, and most major clinical practice guidelines. GRADE has run evidence-based medicine for two decades. What’s new is speed, not the underlying idea of an overall grade. OpenEvidence computes a single grade for the whole answer live, at the moment a clinician asks a question, a step that has previously required human editors working ahead of time.

When a clinician asks OpenEvidence a question, the system first runs a classification step, checking whether the question amounts to a clear evidentiary claim. Simple definition lookups and summarization requests get filtered out here. Questions that pass get scored on study design, consistency of findings, precision, and directness- the same inputs GRADE has always used- and the system returns a letter grade for strength, with a separate “U” designation for evidence that can’t be graded.

Two details are worth noting. OpenEvidence’s classifier decides which questions get graded, not the clinician. And the grading itself comes from a single company applying GRADE algorithmically to its own retrieval results, a different process than GRADE’s original consensus-panel model.

Five Products, Five Methods

OE’s EvidenceGrade scores literature live, at the moment a clinician asks a question. Scoring extends to questions that no formal appraisal has reviewed yet, and the tool runs at point-of-care speed.

UpToDate’s Expert AI launched in late 2025 and sits on top of content that physician editors already reviewed and graded using GRADE, with Grade 1 or 2 for recommendation strength and Grade A, B, or C for evidence quality. The AI layer surfaces existing editorial work conversationally, with rationale and links back to the original topics. Scoring depends on what editors have written and reviewed, so very recent literature can lag what OpenEvidence pulls in real time.

DynaMed shares UpToDate’s editorial foundation: physician editors review and rate evidence with GRADE ahead of time. The method diverges at synthesis. DynaMed’s AI layer rates the individual sources it cites but doesn’t roll them up into a single grade for the answer as a whole, the way UpToDate’s per-recommendation grade or OpenEvidence’s per-answer letter does.

Vera Health also grades the strength of underlying evidence behind its answers and positions itself as a direct point-of-care competitor to OpenEvidence. That grading applies to the individual sources it cites, the same source-level approach DynaMed’s AI takes, rather than producing one overall grade for the answer.

Consensus takes a different approach. Its default ranking sorts papers by citation count, journal reputation, and recency, all signals of a paper’s influence and reach rather than the rigor of its methodology. Consensus also offers an optional evidence-hierarchy filter that lets a clinician sort by study design, from meta-analyses and systematic reviews down through RCTs and observational studies. Nothing is graded automatically here; the clinician chooses whether to apply the filter every time.

Pros and Cons

OpenEvidence offers speed and reach. A grade can appear under a question no editor has ever reviewed, in the time a clinician has between patients. That speed comes from a single company’s automated interpretation of GRADE, run through a classifier a clinician can’t inspect.

UpToDate carries the weight of editorial review behind every grade, built up over years with an established audit trail. Questions about very recent literature can outrun what the editorial team has reached, so timeliness suffers where OpenEvidence’s live scoring holds an advantage.

DynaMed shares UpToDate’s tradeoff of editorial rigor against editorial pace. Its AI layer grades each source it cites, but leaves the work of weighing those individual grades into one overall judgment to the clinician, since it doesn’t generate the single per-recommendation grade UpToDate does.

Vera Health competes on benchmark performance and broader geographic reach and on a grading methodology it hasn’t named as clearly as its competitors have. It also stops at source-level grades rather than synthesizing one for the answer, so a clinician still has to weigh conflicting source grades themselves. A clinician outside the US or UK who lost access to OpenEvidence may find Vera Health the more practical option.

Consensus puts control directly in the clinician’s hands. No classifier decides what gets graded, and every AI-generated summary traces back to an exact sentence in the source paper. That control depends on the clinician knowing to apply the hierarchy filter; skip it, and the ranking on screen reflects citation counts and journal reputation, which are signals of popularity rather than rigor. Consensus also serves general research across all fields, so it lacks the clinical workflow tuning built into the other four tools.

When to Actually Use Which

A fast, evidentiary bedside question, something like whether a drug reduces mortality in a given population, fits OpenEvidence well. Watch for whether a banner appears at all; no banner means the classifier judged the question outside EvidenceGrade’s scope, which is worth a second look if the question felt evidentiary to you.

A well-established clinical question or standard management of a common chronic condition plays to UpToDate or DynaMed’s strength. Human editorial review has already happened, and the subscription cost buys that vetting.

A clinician outside the US or UK, or one weighing the benchmark claims directly, has reason to look at Vera Health alongside or instead of OpenEvidence, keeping in mind that the comparative claims come from Vera Health itself.

Deep literature review on an emerging or contested question, or research headed into a paper or grant application, calls for Consensus, used deliberately with the hierarchy filter engaged and the full-text tool open to check claims directly.

Many everyday questions have no clear winner among these five. When UpToDate already holds a well-established Grade A recommendation for a common condition, OpenEvidence’s live grade on the same question will likely land in the same place, since both draw on the same underlying evidence base. The choice there comes down to workflow preference, editorial certainty against conversational speed. A clinician willing to spend an extra ninety seconds applying Consensus’s RCT filter can reach a similar result to what EvidenceGrade delivers automatically.

Risk and Bias, In Both Directions

The editorial model behind UpToDate and DynaMed carries its own risks. A recommendation can sit unchanged for months after new evidence complicates it, and every grading decision reflects the judgment of the specific editors who wrote it, not an infallible panel.

OpenEvidence’s automated model carries its own risk. A letter grade can read as more authoritative than a live scoring process actually supports, which invites automation bias. The classification step still makes a judgment call the clinician doesn’t get to weigh in on: whether a question counts as evidentiary at all. Grading quality also depends entirely on what the retrieval system pulled in the first place, a step a clinician can’t audit in real time.

DynaMed’s AI and Vera Health carry a different risk, tied to the same source-level grading described above. Neither tool synthesizes the individual source grades into one judgment, which means the aggregation OpenEvidence and UpToDate perform automatically becomes a manual step for a clinician moving quickly between patients, and manual synthesis done under time pressure is where inconsistency creeps in.

Consensus shifts risk toward the clinician directly. Someone who doesn’t know to apply the evidence-hierarchy filter, or who doesn’t recognize that the default ranking measures popularity rather than rigor, gets no protection from the tool itself.

Each model fails in its own way, and the failure a clinician is exposed to depends on which tool they picked and how well they understand what runs underneath it.

The Question Nobody Has Answered: Liability

Clinicians are not attorneys, and nothing here is legal advice. It’s worth asking plainly, since none of the marketing for any of these five products addresses it directly: Does a clinician carry more or less liability for relying on an AI-generated evidence grade compared to a human-editor-vetted one?

I could not find an answer to this question. Malpractice and negligence standards generally turn on whether a clinician exercised reasonable judgment consistent with the standard of care, and historically that inquiry centers on the clinician’s own decision rather than the reference tool behind it. All five products here position themselves as clinical decision support rather than autonomous diagnostic tools, which keeps the clinician as the final decision-maker of record, the same SaMD-versus-CDS framing this newsletter has examined before.

Whether courts will eventually treat a real-time AI grade differently from an incremental human editorial grade remains genuinely untested. I don’t have an answer, and anyone claiming otherwise is speculating.

What holds steady across all five tools is this: whatever grade appears on the screen, the decision that follows still belongs to the clinician who acts on it.

As always, if you enjoy reading, please subscribe and tell a friend.

Sam

When a clinician has a question during an encounter, the workflow is straightforward. They recognize the need for evidence. They leave the patient, open a search tool like OpenEvidence, UpToDate, AMBOSS, or Doximity, formulate a query, and try to find an answer. Then they return to the chart and translate that finding into a plan.

It takes time and requires cognitive effort. It also interrupts the clinical workflow.

What if that entire sequence could be automated by pairing two things that currently operate independently: a scribe that captures context and an AI search tool that needs context?

If it works the way vendors are beginning to market it, the implications could be substantial.

The Manual Search Problem

Clinical decision support tools are valuable. The problem is that clinicians rarely use them consistently.

Adoption of standalone clinical decision support (CDS) has hovered between 20 and 50% for more than a decade. When adoption rates exceed 50% in any health system, it’s usually because the tool was so deeply embedded in workflow that using it became invisible. Not a conscious decision to “search for evidence,” but a byproduct of how documentation already worked.

The Agency for Healthcare Research and Quality (AHRQ)‘s evidence on clinical decision support has long established this pattern. In a 2005 study on clinical information needs, Ely and colleagues found that the most common reason physicians did not pursue answers to clinical questions was time. Not having access to tools, or skepticism about the instrument. Every additional click, every additional screen, every additional cognitive step to formulate what you’re searching for, all of that is friction.

The manual prompt introduces friction at multiple points.

First, the clinician has to recognize that they need help. That alone is cognitively taxing when you’re in the middle of an encounter and managing multiple competing hypotheses simultaneously.

Second, they have to formulate what they’re searching for. They have to decide what context matters. For example, a 65-year-old woman with a subsegmental PE is the bare minimum. But the clinically relevant narrative includes much more: her main symptom was chest pain, her father died of an MI at 65, she has no dyspnea despite the imaging findings, she’s anxious about the PE diagnosis. The question is whether the clinician remembers or prioritizes all of that when they type a prompt. Often they don’t. They reduce it to what feels immediately salient.

Third, they have to type it. And then translate the results back into their documentation.

Each of these steps is a place where information can be lost. Each is a place where a busy clinician can decide it’s faster to just make a decision than to go through the process.

The result: CDS adoption stalls. Not because the tools don’t work, but because the friction cost exceeds the perceived benefit.

What If Context Was Automatic?

Now consider a different architecture.

A scribe is already listening to the encounter. It’s capturing everything: chief complaint, associated symptoms, what improved with aspirin, the patient’s expressed anxiety about the PE finding, the absence of dyspnea despite imaging, family history, the nuance of how the history actually unfolded. The scribe structures that into a clinical note.

What if that same structured encounter context became the automatic input to downstream AI search and clinical decision support?

The difference is subtle but consequential. Instead of waiting for the clinician to recognize they need help and formulate a prompt, the system works from what’s already been captured. The context is automatically available. The search doesn’t depend on the clinician deciding what matters or remembering to type it.

This is the architectural shift vendors are now building.

Glass Health combines ambient scribe with a clinical reasoning layer that accesses the encounter transcript. When the scribe finishes capturing the visit, the same data becomes context for CDS reasoning, not a separate step, not a separate prompt, but the natural downstream use of data already captured.

Microsoft’s DAX Copilot generates the note from the scribe, then uses that same encounter context to surface order suggestions within Epic. The clinician doesn’t have to prompt for order ideas; they arrive pre-staged, informed by what was actually discussed.

Abridge, after expanding into real-time prior authorization through an Availity partnership announced in January 2026, is pre-populating prior auth requests with encounter context. The clinician reviews, not writes.

Ambience Healthcare’s AutoScribe provides real-time coding suggestions, quality measure tracking, and automated prior authorization recommendations, all triggered by the scribe context, all without requiring the clinician to initiate a separate search.

In each case, the pattern is the same: scribe captures context automatically, downstream systems act on that context without waiting for the clinician to formulate a prompt.

Why This Matters

If CDS adoption has been stuck at 20-50% because clinicians won’t use tools that require extra steps, and if integrated scribe-plus-search systems eliminate that friction by making context automatic and routing recommendations directly into workflow, then adoption could shift.

Not because the search tools suddenly became better. But because the context became richer, the friction resolved, and the recommendations arrived where clinicians are already working.

There’s a secondary automation gain as well. Once encounter context is structured and available, systems downstream can act without additional clinician prompts. Abridge is already doing this with order placement. The prior authorization request comes pre-populated. The labs or imaging recommendations arrive as suggestions for review, not as questions requiring the clinician to type more information.

Each of these automations removes a decision point. Each removes a place where context can be lost, or a clinician can opt out because the friction became too high.

This is different than asking “are AI search tools more accurate when they have more information?” This is asking whether integrated scribe-plus-search systems could achieve the adoption curves and utilization patterns that standalone CDS have chased unsuccessfully for years.

Where This Is Already Shipping

The market is moving in this direction. The examples above are live products, deployed now.

• OpenEvidence, Doximity, and Glass Health offer scribe plus clinical decision support in the same interface.

• DAX Copilot’s order suggestions are live within Epic.

• Abridge’s prior authorization workflow doesn’t require the clinician to re-enter what was already discussed with the patient; the scribe context carries it forward.

• Athenahealth made its ambient scribe free to all customers in February 2026. The company has explicitly positioned the scribe as a foundation for downstream clinical and administrative workflows.

These aren’t experimental pilots. They’re market moves. Multiple vendors are racing to integrate scribe plus CDS, scribe plus order suggestions, scribe plus prior authorization, and scribe plus billing optimization. The race itself suggests vendors believe integration is where the value is concentrating.

The Evidence Question

AHRQ best practices emphasize that CDS embedded in clinical workflow achieves significantly higher adoption rates than standalone tools.

The Rotenstein et al. study published in JAMA in April 2026 tracked 8,581 clinicians across five academic medical centers and found that 79% of eligible clinicians declined to adopt a scribe when it was offered as a standalone documentation tool. Among those who did adopt, only 32% used it in 50% or more of visits, the threshold where benefits actually accumulated.

But at organizations where the scribe was deliberately integrated into clinical workflow with physician champions, hands-on training, and customization of documentation to match local practice, adoption reached 75-80%.

Central Oklahoma Family Medical Center saw minimal scribe adoption in year 1 when it was positioned as a documentation tool. In year 3, after deliberate workflow integration and physician-led training, the system was generating over 14,000 records annually.

Research supports the pattern: embedded tools are better than standalone tools. Integrated workflows are better than siloed workflows. Friction is the constraint, not technology limits.

However, there is no published comparative adoption study in the same health system measuring scribe-only utilization versus integrated scribe-plus-CDS utilization. The vendors are claiming that integration drives adoption. The friction research supports that claim logically. But direct comparative evidence is absent.

That gap is fixable. It’s also important. If the thesis holds that integrated scribe plus CDS achieves meaningfully higher adoption than scribe-only, that would directly support the “game changer” framing. Until that evidence exists, we’re working from logical inference and market positioning, not real-world data.

What Changes Clinically

Let me be concrete about what this looks like in practice.

Today’s workflow: A 65-year-old woman presents with chest pain, elevated troponin, and a subsegmental PE on imaging. The clinician needs to decide between acute coronary syndrome and a low-risk PE with incidental findings. They recognize they need evidence about ACS risk stratification and PE disposition pathways. They leave the encounter, open OpenEvidence or UpToDate, search for something like “subsegmental PE,” scan results, and return to the chart with an answer. Time cost: 5-10 minutes. Friction cost: context loss between encounter and search.

Tomorrow’s workflow: The same patient, same presentation. The scribe has captured the full encounter: the aspirin response, the specific way the history unfolded, the family history details, the absence of dyspnea. Once the note is drafted, that encounter context automatically feeds into CDS reasoning. Before the clinician even finishes reviewing the note, recommendations about ACS risk stratification appear inline. They’re specific to what was discussed, not to a reductive prompt. The clinician reviews and acts. No extra steps. No context loss.

The difference isn’t speed alone. It’s that the clinical reasoning is informed by what actually happened, not by what the clinician decided was relevant enough to type.

What Has to Be True for This to Work

This only works if several conditions hold.

Scribe accuracy matters more. If the scribe omits a symptom or invents a clinical detail, everything downstream breaks. Recommendations based on false or inaccurate context are worse than no recommendations. The accuracy bar for a scribe that feeds into automation is higher than for a scribe that just generates a note for a clinician to review and edit.

Context routing has to be intelligent. Not every piece of encounter context should trigger every possible recommendation. Signal-to-noise matters. If the system fires off ten recommendations per encounter, most of which are irrelevant, clinician trust collapses. Integration has to include filtering and prioritization logic that decides what context maps to what recommendations.

Integration has to be seamless. If embedding CDS into the scribe workflow adds steps, the friction returns and adoption stalls. The system has to route recommendations directly into existing documentation workflows without requiring the clinician to open new windows, review separate interfaces, or translate between formats.

Liability has to be clear. If a scribe misses a clinical detail, who’s responsible? If CDS gives a recommendation based on scribe context and it’s wrong, who bears the liability: the scribe vendor, the CDS vendor, the clinician, the health system? Until those questions are answered clearly, adoption will be cautious.

Clinicians have to adopt the pattern. The mental model of “don’t prompt, just accept recommendations” is new for many physicians. It requires trust in both the scribe's accuracy and the CDS's reasoning. It requires that clinicians believe recommendations are being triggered appropriately, not indiscriminately. Building that trust takes time.

Conclusion

The architectural logic is sound. Pairing a scribe that captures context with CDS that needs context could eliminate the friction that’s kept adoption low. Integration could shift clinical decision support from a tool clinicians invoke when they have time to a system that informs their thinking automatically.

But “could” and “does” are different things. The market believes in this direction. Vendors are shipping products built on this assumption.

The evidence, though, is still emerging. The comparative adoption study doesn’t exist yet. The real-world error profiles haven’t been published. The liability frameworks are still being negotiated.

What we’re watching is an inflection point. The question isn’t whether AI scribes plus AI search could be a game changer. The architecture is sound enough that it’s plausible. The question is whether the systems being deployed right now will deliver on that promise at scale.

That’s worth paying attention to.

They need comparison data, but there isn’t any. The single largest, most consequential technology decision many U.S. hospitals will make this decade is being made with far less evidence than we would accept for a new drug or device. Let’s dive deeper.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

It’s not a hypothetical problem.

• More than 85% of Epic’s customer base is already using some form of Epic AI

• EHR vendor dependency is cited as the top execution barrier by 74% of health system technology leaders

• A majority of Epic’s customers report spending a quarter of their IT bandwidth simply managing multiple vendor integrations rather than building anything new.

So how are hospitals supposed to decide about Epic’s AI features? Three questions capture the actual decision tree a CIO faces:

1. How do you compare Epic’s tools against alternatives when no published performance data exists?

2. Are you left comparing on price alone?

3. Can you even connect an outside AI tool to Epic, and what does that actually cost in dollars and time?

Each answer turns out to be more complicated than it first appears.

The Comparison Vacuum

According to EPIC, more than 200 organizations use Penny for professional coding, with many seeing 20% of more coding-related denial reductions. Art’s Insights feature is used more than 16 million times a month. At The Christ Hospital, Art’s radiology finding extraction is credited with a 69% early lung cancer detection rate compared to a national average of 46%. At Rush University Medical Center, Emmie delivered a 58% reduction in billing-related customer service messages.

These are genuine wins, not vague marketing language. But notice what they have in common: every one of them is a curated success story that Epic chose to publish, from a customer Epic chose to name, describing an outcome Epic chose to measure. None of them come with a comparison arm. None of them tell you what a third-party tool, or no AI tool at all, would have achieved at the same hospital over the same period.

What doesn’t exist is any neutral, third-party study putting Epic’s tools head-to-head against Abridge, Ambience, Nabla, Suki, or Oracle Health’s Clinical AI Agent, on the same patients, measuring the same things. The closest proxies available each fall short in a specific, disqualifying way:

• KLAS “Best in KLAS” rankings exist for ambient AI and clinical documentation, and vendors like Abridge and Ambience have won them in 2025 and 2026. But KLAS rankings are built from customer satisfaction surveys. They measure whether clinicians like a tool, not whether it performs more accurately or safely than the alternative.

• Vendor comparison content, like the “Abridge vs. Ambience” writeups that circulate among health IT consultants, reads like independent analysis but is typically produced by resellers and integration firms with a commercial stake in one outcome or another.

To be fair to Epic, native integration solves a real, well-documented problem. Third-party AI tools often exist as a separate app or window that a clinician has to switch into, breaking their workflow. Epic’s tools, by contrast, draw on the patient’s complete longitudinal record natively, inside the same interface the clinician already lives in. That is legitimate integration depth, and Epic has a structural claim to it that outside vendors have to work much harder to match.

But “we’re already inside your workflow, so trust us” is a switching-cost argument, not a quality argument. It explains why Epic’s tools are more convenient to adopt. It says nothing about whether Art’s diagnostic suggestions, Emmie’s patient explanations, or Penny’s coding recommendations are more accurate than a competitor’s because nobody has measured that, and Epic has no more incentive to fund that comparison than a third-party vendor would have to fund one showing Epic winning.

The honest, unsatisfying answer for how hospitals resolve this: they don’t rely on external comparison data. They run their own limited pilots of both options against their own patient population and staff, because nobody else has done, or has an incentive to do, that comparison for them. That’s not a failure of any individual CIO’s diligence. It’s a structural gap in the market that every hospital is left to close on its own, one expensive pilot at a time.

Price Is Visible. Quality Isn’t.

No CIO is literally forced to decide on price alone. But price is the one variable in this decision with any real transparency, and that asymmetry quietly warps the whole comparison because when quality is unmeasurable and price is a line item, price tends to win by default, whether or not it should.

The other differentiators hospitals actually weigh are real, but none of them are clinical outcome measures:

• Integration depth. Does the tool live inside Epic natively, or does it require clinicians to leave the EHR? This is the “house vs. guest in the house” distinction discussed above. It’s a genuine operational factor, not evidence of accuracy or safety.

• Specialty coverage breadth. Some third-party tools market coverage across 200-plus specialties, from oncology to emergency medicine. That’s a real capability difference, but it says nothing about performance within any single specialty.

• Vendor relationship tier with Epic. Some vendors, like Abridge and Nuance, hold a spot in Epic’s invite-only “Workshop” co-development tier. Others sit in the more generic, self-service “Connection Hub” listing. This tells you something about durability and long-term support risk. It tells you nothing about whether the underlying AI is any good.

• Lock-in and commoditization risk. As Epic rolls out “good enough” native alternatives, standalone point solutions face real competitive pressure. Betting on a third-party tool means betting that it survives Epic’s gravitational pull.

The uncomfortable throughline across all four: they’re operational and strategic risk proxies, not evidence. A CIO weighing them isn’t answering “which AI is better”. They’re answering “which AI is safer to bet on organizationally.”

Yes, You Can Connect Outside AI to Epic

Epic is not a walled garden on this point. The technical path to connecting an outside AI tool, whether a commercial product or a custom build on an API, is real, standardized, and already proven in production.

The mechanisms, at the level a CIO actually needs:

• SMART on FHIR is an open industry standard, not an Epic-proprietary one, that lets a third-party application launch inside Epic’s interface (Hyperspace, Hyperdrive, or MyChart) with OAuth2-based access to the patient’s clinical context.

• CDS Hooks is a lighter-weight alternative for narrower use cases: an external tool fires in real time on a specific clinical event (opening a chart, signing an order) and returns a recommendation card, without the overhead of launching a full separate application.

• Epic Showroom (formerly App Orchard) is the marketplace and certification pathway, with three tiers:

  • Connection Hub, a basic listing starting around $500 a year, where most third-party apps live.

  • Toolbox, offering curated visibility

  • Workshop, reserved for vendors Epic is actively co-developing with, not a tier you can apply for.

Abridge, Ambience, Nabla, and Suki all run as third-party ambient scribes launched inside Epic today via exactly this pathway. Connecting a custom AI build is well-trodden ground, not an experimental frontier.

What it actually costs, in dollars:

• A basic FHIR read integration for a first site: $15,000–$40,000

• Bidirectional integration (the tool can write back into the chart, not just read from it): $40,000–$80,000

• Each additional hospital site: $5,000–$20,000 in configuration, testing, and governance overhead

• Enterprise multi-site deployments (five-plus hospitals): $100,000–$500,000+

• Annual maintenance: 15–20% of build cost, ongoing

Those figures answer the technical and financial questions. They don’t answer the harder question, which is where most integration projects actually stall.

“Integration Is 20% Development, 80% Negotiation”

Brendan Keeler, a widely cited EHR integration analyst, has a line that captures this better than any cost table: integration is 20% development, 80% negotiation.

It’s helpful sitting thinking about what that 80% actually is, because it’s several negotiations stacked, and none of them get easier just because a vendor has already done this at another hospital:

• Per-site IT governance review. Even though Epic’s FHIR APIs are standardized across every customer, each hospital runs its own instance with its own security policies. A vendor doesn’t clear “Epic” once. They clear each hospital’s IT and security team separately, every time.

• Scope and access-level negotiation. What data can the tool read? Can it write back into the chart? Is access tied to individual user logins or a backend service account? These aren’t just technical settings. They’re negotiated line by line with each hospital’s compliance and security staff, who have their own risk tolerance and institutional precedent to defend.

• Contract terms with Epic itself, separate from any single hospital deal: Showroom tier, vendor services registration, fees.

• Institutional change management: Getting clinical leadership, IT, compliance, and often legal to actually agree to put a new tool inside their clinicians’ workflow.

This negotiation stack isn’t identical across hospitals, even though the underlying FHIR technology may be. Each hospital is, in effect, a separate legal and organizational gate a vendor has to clear from scratch, no matter how many times they’ve cleared an equivalent gate elsewhere. That mismatch is exactly why Epic’s own native tools carry a durable structural advantage that has nothing to do with whether their AI is actually better. They skip the entire negotiation stack because they’re already inside the walls, already covered by the hospital’s existing Epic contract and security posture.

How Vendors Are Fighting Back

Third-party vendors aren’t standing still against this asymmetry. Several strategies have emerged, each attacking a different piece of the 80%:

• Pre-certification as a trust shortcut. HITRUST certification (including a new AI-specific security certification launched in response to healthcare’s AI boom) and SOC 2 Type II audits let a vendor prove its security posture once and present that proof to every prospective hospital, rather than rebuilding trust from zero at each site. HITRUST is now often a baseline requirement for even getting considered in procurement.

• Integration aggregators. Platforms like Redox normalize FHIR, HL7 v2, and proprietary EHR APIs into a single interface, letting a vendor avoid building direct integrations against three to five different EHR systems from scratch, at the cost of some data fidelity and an extra hop in the data flow.

• Design-partner-first rollout. Land one engaged hospital willing to work through the full registration and security review cycle together, get the integration proven in production, then use that as a template for the second and third hospital, accepting that some per-site variability will still surface each time.

• Climbing into Epic’s Workshop tier. Vendors like Abridge and Nuance have secured Epic’s invite-only co-development relationship, which functions as Epic effectively vouching for them. It’s a different negotiation dynamic from approaching each hospital cold.

• Leaning on CDS Hooks for narrower use cases, trading a smaller interaction surface for meaningfully lower integration complexity.

• Regulatory tailwinds. The 21st Century Cures Act’s information-blocking provisions and ONC’s HTI-1 rule, which raised the certification baseline to USCDI v3 as of January 2026, give vendors a legally backed claim to FHIR access, shifting some leverage away from ad hoc hospital IT gatekeeping.

None of this eliminates the negotiation. It compresses or front-loads it. Bidirectional write-back (actually pushing AI-generated content into the medical record, not just reading from it) remains the hardest and slowest part of any integration, regardless of how much pre-certification a vendor obtains.

What CIOs Actually Have to Work With

A few genuinely useful resources exist for the leader trying to navigate this, though none of them is the comprehensive primer this decision deserves.

The Health Sector Coordinating Council’s “Health Industry AI Cyber Governance Framework Implementation Guide” (May 2026) is the closest thing to an authoritative, non-commercial reference, covering governance committee structure scaled to hospital size, escalation authority across CMO, CMIO, and Privacy Officer roles, and a benefit-risk framework tied explicitly to FDA regulatory status.

Qventus’s “Beyond the Pilot” survey of more than 60 CIOs, Chief AI Officers, and CMIOs offers real peer benchmarking: 74% cite EHR vendor dependency as their top execution barrier; 72% say they’d prefer a single consolidated AI partner over a fragmented multi-vendor stack, but only 13% have actually achieved that consolidation. It’s vendor-published, so it should be read as useful data with a thumb on the scale, not neutral research.

Narrower academic frameworks exist too, like peer-reviewed pragmatic-trial protocols for evaluating ambient AI specifically, which bring real methodological rigor but only to one slice of the larger decision.

What none of these does is tie the whole picture together. A CIO today has to assemble governance policy from one source, vendor economics from a consulting firm’s blog, integration cost data from an engineering guide, and validation-status skepticism from wherever they can find it, with no neutral party doing that synthesis for them. That absence is, itself, worth naming plainly: the market has produced plenty of pieces but no assembled whole.

Takeaway

Individual hospitals cannot solve this problem on their own. It is a fundamental issue with the entire healthcare market, which tends to favor established players, like Epic, over newer or better alternatives, simply because they already hold the power.

A few things worth holding onto if you’re the one making this call:

1. Don’t mistake Epic’s adoption percentages for comparative quality evidence. They are usage statistics, curated by the company reporting them. They tell you Epic AI is widely used. They don’t tell you it’s the best option.

2. Budget for the 80%, not just the 20%. The dollar figures for FHIR integration are real, but the calendar time and staff bandwidth consumed by per-site negotiation are the actual cost drivers, and they’re the ones most commonly missing from a project’s original scope.

3. Ask any vendor directly what Showroom tier they hold. It won’t tell you if their AI is accurate, but it will tell you something real about how durable their access is likely to be.

4. Treat the absence of comparative data as a mandate to pilot, not a reason to default to the incumbent. Nobody else is going to run that comparison for you. That doesn’t mean it isn’t worth running.

No one selling you this technology is going to prove it works for you. That job still belongs to the hospital buying it.

Meanwhile, nurses in New York, California, Michigan, and North Carolina have been striking and writing AI oversight rights into union contracts. The national nursing union organization that represents those same nurses is publishing its own AI bill of rights.

Both responses exist because the frameworks and the union contracts are answering different questions. Understanding why requires looking at which AI tools are actually being deployed, to whom, and at whose direction.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

What the Professional Associations Agree On

The American Medical Association published formal AI principles in November 2023, adopted a transparency policy calling for independent third-party verification of AI explainability in June 2025, and released an 8-step governance toolkit in August 2025. The toolkit calls for multidisciplinary governance committees that include nursing leadership, staged deployment with pilot testing, and continuous performance monitoring after launch.

The American Nurses Association convened its inaugural AI in Nursing Practice Think Tank on April 22, 2026. The consensus findings identify automation bias and erosion of professional judgment as the primary risks, call for mandatory AI literacy as a core nursing competency, and push for nurse-led AI governance at the institutional level.

The American Academy of Nursing approved a comprehensive AI position statement on February 25, 2026, setting out 13 specific policy recommendations covering data privacy, algorithmic bias, FDA oversight, and human-in-the-loop oversight standards across all institutional governance policies.

The foundational language across all three documents is nearly identical: AI must augment clinical judgment, not replace it. The human clinician remains the final accountable decision-maker. Human-in-the-loop oversight is non-negotiable.

All three frameworks rely on the same implementation mechanism: voluntary institutional governance, multidisciplinary committees, professional education, and policy advocacy. None specifies what “human in the loop” requires at the point of care. None has an enforcement mechanism at the bedside level.

Two Categories of AI Tools

The governance frameworks were designed for a particular relationship between clinician and AI: the clinician chooses to use a tool, reviews its output, and retains authority over the final decision. The ambient scribe is the clearest example. The physician activates it, the scribe drafts a note, the physician reviews and signs. The tool serves the physician. Clinical judgment stays with the clinician throughout.

Physician AI adoption fits this model closely. By 2026, more than 80% of physicians report using AI in their professional work, with more than three-quarters saying it improves their ability to care for patients. The tools driving that adoption are documentation-focused: 70% of physicians at UCSF now use AI scribes daily, and Kaiser Permanente logged more than 2.5 million AI-scribed encounters over 14 months. Randomized trial evidence confirms that these tools reduce documentation time and burnout scores.

The AI tools most commonly deployed in nursing workflows belong to a different category. Patient acuity scoring algorithms analyze EHR data to determine how sick a patient is and predict how many nursing hours that patient requires. Staffing algorithms use those acuity scores to determine how many nurses are called into a shift. Automated handoff tools generate shift reports. Clinical deterioration algorithms produce alerts. These tools are not activated by the nurse. They are deployed by hospital administration, they run continuously, and their outputs directly constrain what nurses do and how many patients they are assigned.

The NNU’s 2024 survey of 2,300 members documented what this looks like in practice. Half of respondents said their employer uses algorithmic systems to determine patient acuity and predict required nursing hours. Of those, 69% said the AI-generated acuity score did not match their own clinical assessment. Among nurses whose employers use automated handoff tools, 48% said those AI-generated reports contradicted their own patient assessment.

The Override Problem

Forty percent of nurses in hospitals using algorithmic patient outcome tools said they cannot override the algorithm’s prediction when their clinical assessment differs. Twenty-nine percent said they cannot alter algorithm-produced wound or pain documentation in the EHR even when they believe it is inaccurate.

Every professional association framework, including the ANA, AAN, and AMA documents described above, establishes that clinical judgment must prevail over AI outputs. The survey data documents a significant gap between that principle and what is happening at the bedside, and the governance frameworks have no mechanism to close it.

The gap has a structural explanation. The patient acuity and staffing tools that nurses cannot override were not purchased by clinical leadership. They were sold to hospital CFOs and operations teams on the explicit promise of reducing labor costs. Vendor marketing for these systems is direct: AI staffing tools minimize overtime, reduce reliance on expensive agency nurses, and cut contract-labor dependency. Ascension Health reported reducing contract-labor dependency by 15%within six months of implementing a predictive staffing system.

A nurse who can override her acuity score is a nurse who can force additional staffing. Override capability undermines the tool’s core value to its actual customer. The inability to override is not a design flaw. It is, from the purchaser’s perspective, a feature.

Many of these operational tools, including staffing algorithms and acuity scoring systems, also do not meet the FDA’s current definition of a medical device. They face no premarket safety review requirement. No regulator required the vendor to build in an override function, and no regulator currently enforces one.

Presence Without Agency

The academic literature has recently begun to examine what “human in the loop” actually means in practice. A paper in the American Journal of Bioethics specifically flags that researchers and institutions routinely invoke HITL as ethical legitimacy without specifying which humans, in which processes, are doing what, and cites NNU survey data on nurses unable to override algorithmic predictions as a concrete example.

The Institute for Systems Integrity published a related framework in May 2026, distinguishing between two states that are often conflated. In the first, a human is present near the AI system: they are notified, they see the output, and they are in the loop in the awareness sense. In the second, a human has agency: they can interrupt the system, modify its output, or substitute their own judgment without penalty. The institute’s framing is clear:

“A human placed near an AI system is not automatically a safeguard. A clinician asked to approve a recommendation under time pressure, incomplete information, workload overload, and unclear authority may not be exercising judgment.”

The EU AI Act makes this distinction legally operational for high-risk AI systems. Article 14 requires that humans be able to interrupt or override the system’s operation and decide not to use it in a specific situation. This is described as an architectural requirement, not a procedural right. No equivalent US requirement exists for hospital operational AI tools.

A nurse who receives an acuity score that cannot be changed has been notified. That nurse has not been given authority.

What The NNU Is Doing

The National Nurses United published a Nurses and Patients’ Bill of Rights in April 2024. Seven rights are enumerated. Most coverage of this document focuses on Right 7, which demands pre-deployment bargaining rights: the right to negotiate over whether and how AI is implemented before the system is selected.

Right 5 is less examined and more operationally significant given the override data. It establishes the right of nurses to exercise professional judgment and override AI decisions without threat of discipline or discharge. No professional association framework, from the ANA, AAN, or AMA, contains an equivalent enforceable protection. Right 5 is the only document in the current governance landscape that directly addresses what 40% of nurses report experiencing.

The NNU has coordinated a national bargaining campaign through its affiliate network: the California Nurses Association, the New York State Nurses Association, the Michigan Nurses Association, and the National Nurses Organizing Committee in North Carolina. Announced AI contract language has been reported at Mission Hospital in Asheville, North Carolina (2024), Northwell South Shore University Hospital in New York, the New York City hospital systems including NYP, Montefiore, and Mount Sinai following a strike by nearly 15,000 nurses in February 2026, Munson Medical Center in Traverse City, Michigan, and the University of California system, where a contract ratified in November 2025 specifies that nurses play a central role in selecting, designing, and validating new technology including AI systems.

One caveat applies to all of the above. No journalist or outlet has quoted actual contract clause text from any of these agreements. Coverage consistently describes outcomes in terms of “safeguards,” “guardrails,” “approval before deployment,” and “voice in how AI is rolled out.” Whether the ratified language reflects the specificity of the Bill of Rights, including Right 5’s override protection, or represents a narrower pre-deployment consultation right, is unknown without the contract documents.

The physician union equivalent, the Union of American Physicians and Dentists, has identified AI as a bargaining priority, with the organization’s president stating in December 2025 that it is “imperative” that the union engage in dialogue with employers to prevent professional judgment substitution by AI. UAPD has not yet produced ratified AI contract language. That gap is approximately 18 months, which is consistent with the tool asymmetry: physicians have not yet encountered operational AI deployed on them at scale without override rights.

Two Tracks, One Unresolved Problem

The landscape across both professions is now parallel. Both nursing and medicine have a professional association track and a union track responding to AI.

The ANA and AAN are doing what the AMA is doing: publishing principles, developing governance frameworks, and advocating for policy. The difference in urgency between the nursing professional associations and their physician counterparts reflects the difference in tool adoption rates and trust levels, not a fundamental strategic divergence.

NNU is doing something the professional associations can’t do: creating enforceable bedside protections through contract law. They are addressing different layers of the same problem. A hospital can have an excellent AI governance committee and still deploy an acuity algorithm with no override function. A union contract can protect override rights but cannot prevent a poorly validated tool from being purchased in the first place. Both layers are needed.

Neither layer has resolved the problem of definition at the center of all of this. Every framework, every position statement, every contract announcement invokes human oversight as a governing principle. No one has established that human oversight requires the ability to override. Until that definition is settled, “human in the loop” describes a spectrum that runs from a nurse with full authority to substitute her/his clinical judgment to a nurse who receives a notification she/he cannot act on. Both nurses are, technically, in the loop.

Questions Worth Asking

If you are responsible for AI governance at a hospital or health system, the relevant question is whether your governance committee’s approval process specifies override capability as an architectural requirement of deployment instead of a recommendation. A tool that generates outputs clinicians cannot modify does not meet the HITL standard described by the ANA, AAN, or AMA, regardless of what the vendor’s documentation says.

If you are a nurse in a facility with a union contract that includes AI language, the clause text matters more than the press release. “Safeguards” and “guardrails” are not contract language. Right 5 of the NNU Bill of Rights, protection of the right to override without threat of discipline, is the protection most directly supported by the survey data. Whether it appears in your contract is a question worth answering.

If you are a nurse in a non-unionized facility, the ANA Think Tank consensus document and the AAN position statement establish a national professional standard you can cite through shared governance structures or unit councils when raising concerns about tools that cannot be overridden.

If you are a physician, the operational AI category that nursing is responding to is not exclusive to nursing. The tools that constrain clinical judgment without requiring clinical input in procurement are already present in prior authorization, clinical documentation, and diagnostic triage. The UAPD’s December 2025 statement suggests that physician union organizing around AI is overdue.

The governance frameworks say clinical judgment must prevail. The mechanism that makes that principle enforceable at 3am, when an acuity score determines whether a second nurse comes to the floor, does not yet exist in voluntary guidelines.