Sam

Subscribe now

The Awareness Gap

The Pew Research Center surveyed 3,488 U.S. adult patients from June 22 to 28, 2026, on their perspective on AI in Healthcare. The findings highlight a big difference between patient and physician views on AI use in Healthcare. The Pew Research Center’s results found:

A small share (16%) say their doctors or other healthcare providers have ever used AI in their healthcare.

Additionally, another 83% of Americans either think AI has never been used in their healthcare (37%) or don’t know if it has (46%).

Compare that to the AMA’s 2026 Physician Survey on Augmented Intelligence, which reported:

Four in five physicians (81%) use AI in their practices, more than double the 2023 rate (38%).

That leaves a significant gap that we, physicians and clinicians, need to explain. One explanation might be the way that the AMA defined AI use. The survey listed 17 AI use cases. 72% of physicians said they incorporated at least one of them. 9% said they were not sure which ones their practices used. The survey results combined these two groups.

A second explanation might be the tools themselves. Physicians said they used AI for research summarization, documentation, discharge instructions, care plans, and billing. So an individual physician could be using AI according to the AMA, but not in a manner that a patient would be aware of or consider personal to them.

But that’s just at the physician level. Federal data shows that AI use in healthcare goes beyond what individual physicians choose. According to the ONC:

In 2024, 71% of hospitals reported using predictive AI integrated into their EHR.

Think of hospital systems that predict readmissions, identify high-risk patients or early sepsis, and generate treatment recommendations.

But if all three surveys measured something different, can they be used to find a correlation between physician AI use and patient awareness? Not directly, but the gap between 16%, 81%, and 71% is certainly a signal. AI use in healthcare is significant among physicians and hospitals, while very few patients know it is being used.

That raises another question: How would a patient know?

Defining AI Use in Healthcare

Before we figure out how a patient would find out about AI use, we have to define it. Consider these five scenarios:

1. AI Q&A: A physician asks an AI search tool a quick question about current treatment recommendations and then uses that answer to treat a patient. No patient information is shared.

2. Clinical Decision Support: A physician enters a patient’s history, labs, and imaging results into an AI system and asks for a differential diagnosis and treatment plan.

3. Ambient Scribe: A physician uses an ambient scribe to document a patient encounter.

4. Prediction: An AI prediction tool flags a patient as having a high probability of deterioration and a high risk of readmission after discharge.

5. Prioritization: An AI model prioritizes radiology studies for radiologists to read, schedules appointments, generates billing codes, and helps with insurance claims.

All 5 scenarios would be considered AI use in healthcare, but their relevance to a single patient varies quite a bit. When the Pew Research Center asked patients if they should be told about specific AI applications, they found:

• 81% want to know if AI was used to analyze their medical scans or make a diagnosis.

• 80% want to know if AI was used to explain their lab results.

• 72% want to know if AI was used to take notes during an appointment.

• 64% want to know if AI was used to refill a prescription.

• 56% want ot know if AI was used to schedule an appointment.

Though the first few did not surprise me, the last one did. Patients use a much broader boundary for being informed about AI, even for administrative tasks. This difference in definition creates another problem:

At what point does general AI use in healthcare become AI use in a patient’s healthcare?

When Do Patients Have to Be Told?

If we ask patients, the Pew Research Center found

About seven-in-ten U.S. adults (72%) say it’s extremely or very important that a doctor or other healthcare provider tells them if they’re using AI in their healthcare.

And yet, there is no national requirement that every use of AI connected with a patient’s healthcare be separately disclosed. Also, medicine has long used medical decision support tools without disclosing them to patients. That predates today’s generative AI tools. Think tools like ECG computerized interpretations, risk calculators, drug interaction alerts, clinical decision tools like Up-To-Date, and more. But if we have never disclosed these in the past, do we need to disclose them now just because they include AI processes?

What if those systems run autonomously, have access to patient information, generate new content (not just summaries) like patient-specific diagnoses or treatment plans, or contain errors that are difficult for humans to detect? Does that reach a threshold for patient notification?

In a JAMA article, Mello and colleagues proposed two factors for consideration:

The ethical obligation to notify patients depends on (1) the seriousness of the physical risk and (2) the extent to which patients have a meaningful opportunity to exercise agency in response to a notification.

This kind of approach creates different disclosure expectations for an AI system that influences a cancer diagnosis than one that schedules appointments. But Pew Research Center results suggest that patients may want disclosure in both cases. In fact, some states have already begun drafting laws to mandate it.

California AB 3030 requires disclosure for certain patient communications generated using AI. But the law also contains an important exception: “If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.” California Legislature

Texas has adopted broader language: “If an artificial intelligence system is used in relation to health care service or treatment, the provider of the service or treatment shall provide the disclosure.” Texas Business & Commerce Code

What we are beginning to see is a state-by-state patchwork in which disclosure obligations can depend on where care occurs, what the AI does, and how humans interact with its output.

For more state AI laws, see this previous article.

Being Told Is Different From Being Able to Say No

Another problem that may occur to you as you read this: Informing a patient of AI use improves their awareness. But what if they don’t consent? Is there an option to say no?

The question highlights the three aspects of this process:

• Disclosure: Telling the patient that AI is being used.

• Consent: Asking the patient for permission before using the AI software.

• Control: Allowing the patient the ability to refuse that use.

The Pew Research Center’s survey shows that Americans are interested in all three, especially the last one.

Some might turn to HIPAA to provide guidance. After all, HIPAA already allows protected health information to be shared for treatment, payment, and healthcare operations under specific circumstances.

According to HHS:

The Privacy Rule permits, but does not require, a covered entity voluntarily to obtain patient consent for uses and disclosures of protected health information for treatment, payment, and health care operations.

Read it carefully. HIPAA permits but does not require consent. It is a voluntary process. In fact, that federal framework includes technology companies functioning as business associates. HHS specifically gives this example of an approved business associate.

Third-party vendor Artificial Intelligence (AI) chatbot on a provider’s patient portal that provides services involving the patient’s PHI such as symptom assessment, medical reminders, and appointment scheduling.

That means an AI vendor can access protected health information (PHI) with a HIPAA-compliant arrangement without the patient having to sign a separate authorization for AI use. But if we look at the Pew Research Center’s survey results, 63% of patients want more say over whether AI is used in their care. This becomes an issue because a healthcare organization can check all the disclosure boxes simply by informing a patient that it uses AI, with no way for the patient to opt out.

So HIPAA does not create a right to opt out of AI use in healthcare. But is it even possible anymore?

For an ambient scribe service, the physician just documents manually. It’s slower but still possible. What about a patient who asks that their information not be processed by:

• an EHR deterioration model

• AI-assisted radiology software

• automated ECG interpretation

• medication safety algorithms

• radiology worklist prioritization

• scheduling algorithms

• billing systems

All of these are out of an individual physician’s control in the vast majority of practices. There is no mechanism to turn them off, or to process patient information without them. Many of them are operating in the background, automatically. As AI becomes increasingly embedded within healthcare infrastructure, a patient’s ability to opt out becomes impossible.

And what if opting out of AI use results in poor care for the patient? As AI systems improve, they will assist in everything from radiology interpretation to early detection of deterioration. Opting out might result in much longer interpretation times, fewer automated safety checks, and additional administrative delays.

If you work in a hospital, you might be asking yourself if the right to refuse AI requires a parallel workflow. Are hospitals obligated to maintain an alternative, and what would that even look like in a world where AI is embedded in the EHR?

AI Is Arriving Faster Than the Rules Around It

It’s a common theme across all the articles in this newsletter, and the Pew Research Center’s survey results are just another example. AI adoption is moving at a faster pace than the rules and laws around it.

Patients want transparency about AI use and more control over whether it participates in their healthcare. And we don’t have answers to:

What constitutes AI use in an individual patient’s care?

Which uses require disclosure?

When should disclosure include consent?

Which uses should patients be allowed to refuse?

How should healthcare systems accommodate refusal when AI has become part of their infrastructure?

With each passing day, these questions become harder to defer.

As always, if you enjoy reading this newsletter, subscribe and tell a friend.

Sam

A lot of my writing and discussion in medicine in general has focused on the danger of artificial intelligence. I’ve discussed AI hallucinations, incorrect conclusions, concern over physician overreliance, data privacy and consent, and more.

But today, I want to discuss one of the positive aspects of combining AI and medical research.

Medicine as a research source

As I wrote about in my last article, medical data storehouses are increasing in number.

Hospital electronic health records, insurance company claims databases, government datasets, and clinical trial data are just a few examples. Recently, consumer-facing products have begun building their own databases of patient information. Some notable examples: Apple Research, Google Health Connect, Evidation. Users are able to upload their health records, lab records, imaging, and even connect their wearables. That data is being voluntarily given to large companies, which can build their own databases.

Historically, researchers have accessed these databases to ask new questions and find correlations between therapies and specific populations. This type of research fits into a category called “secondary data analysis”. The data already exists, and researchers are using it to answer questions.

The diagram below shows just how many different kinds of research studies can be derived from secondary data analyses.

Secondary data analyses aren’t just limited to observational studies. Researchers regularly use data from previously published randomized controlled trials to look for new outcomes, perform subgroup analysis, evaluate prognostic relationships, and answer questions that were not addressed in the original publication.

Oftentimes people think of secondary data analysis as a lesser form of research. But that’s not the case at all.

Influential medical research uses existing data

You don’t have to take my word for it. Look at any major medical journal, and you can find numerous examples.

The National Health and Nutrition Examination Survey (NHANES) produced important estimates of obesity and chronic kidney disease in the United States.

A 2009 study in the New England Journal of Medicine looked at data from almost 12 million Medicare beneficiaries, with a focus on hospital readmissions and their costs. That publication led to a national discussion surrounding hospital readmission.

Medicaid data was used to investigate whether atypical antipsychotic medications were associated with sudden cardiac death.

Medicare databases have also been used to examine the effectiveness and safety of multiple anticoagulants in patients with atrial fibrillation.

SEER-Medicare data were used to investigate differences in breast cancer outcomes between Black and White women.

Secondary analysis of data from major randomized controlled studies like the SPRINT and FOURIER trials has answered additional questions.

All across the spectrum of medical specialties, secondary analyses have contributed significantly to our medical knowledge.

Researchers don’t need to start over every time

A single medical database can support many different questions

With access to the medical histories of millions of patients, the number and types of questions are seemingly unlimited.

• You could ask a descriptive question like “How common is chronic kidney disease among patients with diabetes?”

• You could ask a comparative effectiveness question like “Among patients with atrial fibrillation, how do outcomes differ between patients receiving two commonly used anticoagulants?”

• You could investigate safety by asking “Are patients who are receiving a particular medication more likely to develop acute kidney injury?”

• You could study prognosis by asking “Which characteristics predict readmission following hospitalization for heart failure?”

• You could examine disparities by asking “What percentage of patients in different demographic or geographic populations receive guideline-recommended treatment?”

• You could study changes over time by asking “Does an FDA-issued safety warning change prescribing behavior?”

Each of these questions can be asked of the same database and result in truly significant findings.

The barriers

Physicians frequently come across unanswered questions while treating patients.

• A cardiologist may notice a certain group of patients seems to respond unusually well to a treatment.

• An emergency physician may wonder why the same diagnosis causes some patients to return to the emergency department while others don’t.

• A primary care physician may notice an unreported adverse effect of a medication.

• An oncologist may wonder if a treatment seems to perform differently in a particular patient population.

These questions and observations are the foundation of medical research. But access to what’s needed to perform that research is not the same across the spectrum of physicians.

Turning that question into a research project requires numerous steps like:

• Defining a study population, inclusion and exclusion criteria, and exposures and outcomes.

• Understanding the structure of a database, how to write queries and extract data

• Choosing a statistical approach, identifying confounders, and performing sensitivity analyses

Until now, those steps required a team of epidemiologists, biostatisticians, database specialists, and programmers. Although those people provide very valuable expertise, they are all scarce resources. A physician working at a large academic medical center may have access to that team. A community physician with an interesting clinical observation is more likely to just let it slip away.

AI may change all of that.

The distance between question and analysis

Imagine you are a hospitalist and notice that patients you discharge on drug A don’t bounce back as often as patients on drug B. You check your favorite clinical reference, but there is no published data comparing the two drugs. AI has the potential to assist in the research needed to answer the question: “Among patients with condition X who initiated Drug A versus Drug B, was there a difference in hospitalization within 12 months?”

• Based on your question, AI might help identify a study design like a retrospective cohort study.

• It could help define the cohort by asking questions like: Which patients should be included or excluded? How should initiation of the medication be defined? How much past medical history is required?

• It could help define a comparator group and outcome.

• It could identify possible confounders.

• It could help identify an existing medical database ideally suited for your needs.

• It could help generate a database query.

• It could assist with statistical analysis and sensitivity testing.

• It could help interpret the results and identify weaknesses in the analysis.

The process is familiar because it is the same one medical researchers perform routinely. But the ability for a practicing community physician to complete that process has now changed.

Clinical observation → research question → study design → cohort definition → database query → analysis → evidence

AI could assist with each step.

The catch

Making an analysis easier to perform doesn’t automatically make the analysis trustworthy. Medical databases can be messy.

• Patients are not randomly assigned to treatment groups in routine clinical practice.

• Diagnoses can be coded incorrectly.

• Outcomes may be missing.

• Patients can leave a healthcare system and disappear from the database, leaving no follow-up data.

• Laboratory testing occurs more frequently in sicker patients.

• Medication records show what was prescribed, not what was actually taken.

• Researchers can inadvertently define populations that bias their results.

And if enough variables are included, significant associations can emerge just by chance. AI doesn’t make any of those problems disappear. In fact, AI could compound the problem by allowing a poorly designed analysis to be performed with extraordinary speed.

That means physicians using these tools will still need to understand concepts like confounding, selection bias, misclassification, missing data, multiple comparisons, causal inference, statistical validity, and reproducibility.

Correct research methodology, biostatisticians, epidemiologists, and expert review will all remain essential. The opportunity comes from allowing those resources to function differently. Instead of every research question requiring extensive technical work before it can even be explored, AI may allow physicians to perform preliminary analyses, test feasibility, refine hypotheses, and identify questions that deserve deeper methodological involvement.

AI and the research cycle

One last area AI can impact.

Secondary data analysis often occurs before prospective research.

Suppose an analysis of five million patient records identifies an unexpected association between a medication and a clinical outcome. That doesn’t prove the medication caused the outcome. But it does give researchers a valuable signal that may justify an observational study, which in turn may lead to a prospective study. And eventually, a randomized controlled trial.

The same process can work in reverse. Researchers can use completed randomized trials and ask additional questions of datasets that cost millions of dollars and years of effort to create.

Here, AI has the potential to accelerate multiple stages of the research cycle:

• Hypothesis generation.

• Feasibility assessment.

• Safety-signal detection.

• Comparative effectiveness research.

• Identification of high-risk populations.

• Subgroup exploration.

• Study design.

And identification of questions worthy of prospective trials. The acceleration allows more questions to be investigated.

Should we broaden the conversation around AI?

We have to establish boundaries around the use of artificial intelligence in medical research. We need standards for privacy, validation, reproducibility, transparency, methodology, and human accountability just as we have non-AI-related standards for all these areas now.

These safeguards are important precisely because these tools may become extraordinarily capable.

Medicine has enormous datasets describing diseases, treatments, complications, and outcomes across millions of patients. The scientific opportunity contained in that data is enormous. AI may give us a new way to interrogate it.

Commercial and public databases are now advertising longitudinal data, sometimes covering the entire lifetime of a patient. How does that data get assembled if it is de-identified? How do you link records from multiple sources belonging to a single patient while maintaining anonymity? And does that still allow you to sell it without patient consent? Let’s take a closer look.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Building the database

A medical database is just a collection of clinical events like diagnoses, labs, meds, procedures, notes, images, and vitals, from multiple systems, reorganized into something a researcher (or a model) can actually use.

Two things about that process are important to examine if you are considering contributing your data or purchasing access.

Where the data came from determines who’s in it. A database built from hospital EHRs only contains people who were sick enough and able to get to those hospitals. One built from insurance claims only contains people with health insurance. One built from a volunteer research cohort only contains people who chose to sign up. And we know that volunteers are healthier, wealthier, and more educated than the general population they’re supposed to represent. A neutral, unbiased sampling is extremely difficult to collect. Every database has some limitation.

Consent isn’t uniform. Prospective cohorts like UK Biobank, All of Us, and the Nurses’ Health Study enroll people who opt in and know their data will be used for research. But most commercial products don’t work that way. They’re built from clinical data that already existed because a patient was treated under HIPAA rules, with no separate research consent involved. The patient whose ED visit becomes a row in Truveta or Optum or Oracle Health’s dataset almost certainly has never heard of any of those companies.

What “de-identified” means

Under HIPAA, “de-identified” has a specific legal definition with two methods listed in 45 CFR §164.514.

Safe Harbor means you strip all 18 specified identifiers: names, any geography smaller than a state, every date more specific than year, phone numbers, SSNs, medical record numbers, device IDs, full-face photos, and more. Also, you must have no knowledge that what’s left could still identify someone.

Expert Determination is the other route. A qualified statistician determines that the risk of re-identifying anyone in the set is “very small.” This option allows more granular detail to remain in the database, like month-level dates and smaller geography.

If you can accomplish either method, the data is no longer Protected Health Information. Once health data is de-identified, the Privacy Rule doesn’t apply to it anymore. No authorization is needed to use it, sell it, or share it because those rights only attach to PHI. So a patient’s ED visit or hospital stay, once properly de-identified, can be aggregated, licensed, and sold to a pharma company or an AI developer, and that patient has no way to find out it happened.

The longitudinal problem

When trying to collect data over a long period of time (longitudinal data), there are additional complications.

A longitudinal database links the same patient’s records across time: the ED visit in 2019, the diagnosis in 2021, the hospitalization in 2023, all tied to one underlying person so you can see a trajectory instead of individual snapshots. That linkage is what makes disease-progression modeling possible. It’s the “digital twin” concept mentioned in the Nordic paper.

But linkage needs something to tie the records together. And that something is, by definition, an identifier. Which means the more rigorously you strip identity, the harder it gets to prove two records belong to the same person.

HIPAA planned for this under §164.514(c). A covered entity can assign a re-identification code (or token) to de-identified records, essentially letting them be linked as long as the code isn’t derived from anything about the individual. It may not be a portion of a SSN, a birthdate, or MRN. That code also may not be disclosed to whoever gets the data. That way the dataset stays legally de-identified even though a linking key exists somewhere behind a firewall. This is basically how tokenization services like Datavant work. They generate non-PHI-derived tokens so separate data sources can recognize “same patient” without ever exposing who that patient actually is. Atropos Health’s evidence network runs on this kind of tokenized linkage, by its own description.

Now compare that to the Nordic model in the paper I mentioned earlier. Nordic countries link registries using government-issued personal ID numbers. These identifiers often encode birthdate and are held by national authorities specifically so records can be traced back to identity. Under GDPR, that’s not de-identification. It’s “pseudonymization,” a legally separate, lesser category. GDPR Article 4(5) and Recital 26 are very specific about it: if re-identification remains possible through a separately held key, the data is still “personal data” and GDPR still fully applies.

So, HIPAA treats "unique identifier" and "de-identified" as interchangeable, but that's not GDPR's approach. By GDPR definition, an identifier that enables linkage is evidence that the data isn't anonymous. It's a privacy improvement over raw identifiable data, and it's the mechanism that makes cross-registry Nordic research possible.

That leaves us with two ways to get a longitudinally linked, de-identified dataset.

1. Keep a compliant re-identification key around, like the tokenization approach above, so the database can keep linking new records to existing patients indefinitely. That’s how vendors like Truveta and Oracle Health can advertise daily updates.

2. Assemble the whole dataset first, using real identifiers in a locked-down environment to do all the patient-level linking, and then strip identifiers as a final, one-way step before anything leaves the building. That produces a dataset that’s genuinely de-identified with no lingering key anywhere. But it means all the linking work has to be done before de-identification happens, which means what gets released is a closed, static snapshot. Once the identifiers come off, you can’t add a new record for an existing patient without reopening the identified stage and starting over. No version of this approach gives you a “living,” continuously updated database. You can only produce a new static version periodically.

That’s why MIMIC, NHANES, SEER, and the National Inpatient Sample ship as periodic releases instead of live updates. Meanwhile, the commercial products marketed as daily or real-time updates rely on some kind of retained linking key. Neither approach is wrong. They’re just not the same privacy posture, and “de-identified” alone doesn’t tell you which one you’re looking at.

So what’s actually out there

Not everything called a “medical database” is the same. Some are commercial products with a sales team. Some are federally funded public resources. Some are prospective research cohorts that have been running for decades. Below is a breakdown of nineteen of the big databases.

Two philosophies, same underlying risk

Another way to sort databases on the list is by where the data lives.

1. Centralized (pooled) means the data physically or virtually lives in one place the vendor controls. Truveta, Optum, MarketScan, Oracle Health, Flatiron, and UK Biobank are examples of this strategy.

2. Federated is the opposite. The data never leaves the institution that generated it. A query gets sent to the data, and only aggregated results are returned. This is the model the Nordic commentary is suggesting. PCORnet and OpenSAFELY are the cleanest examples of it. TriNetX and Atropos run hybrid versions with a federated query / tokenized-network layer sitting on top of member institutions’ own systems.

The Nordic paper frames this as a design choice, proposing federated approaches for the most sensitive data, pooling for standardized and less sensitive data, and synthetic data for pretraining where either option is too risky. It’s a layered strategy, depending on the data needed and the use case.

Centralized storage has real strengths. It supports multimodal model training across notes, images (radiology), and waveforms (EEG, ECG). One unified data model, so a single researcher can iterate fast. But it also means a single point of possible security failure. UK Biobank, despite a strong governance reputation, recently had a data breach that the Nordic authors cited as a cautionary tale. It requires enormous cross-institutional trust and legal paperwork to move data across borders. And it concentrates security and the exact de-identification methodology inside one vendor.

Federated storage changes most of that. Data never crosses institutional or national borders, which avoids a lot of cross-border transfer risk under GDPR or the EU’s European Health Data Space. It preserves local control, and it tends to earn higher public trust for exactly that reason. The trade-off is that it’s hard to train large multimodal foundation models this way because you can’t combine raw pixel (image) or waveform (EEG, ECG) data across sites. So you’re stuck with averaged or aggregate query results. Site-to-site differences in EHR vendors and coding practices make joint analysis messier. And it’s structurally harder for anyone outside the network to check what’s in the underlying records.

What trips people up

If you’re a researcher picking a data source, or a clinician trying to figure out what a new AI tool was actually trained on, the marketing page rarely answers the following important questions:

Bias: Who’s in the database. Hospital EHR data skews toward people sick enough to seek care. Claims data cuts out the uninsured entirely and skews toward employed populations. MarketScan describes its data as commercially insured workers and dependents. Volunteer cohorts carry the opposite bias: UK Biobank, the Nurses’ Health Study, and All of Us recruit people healthier, wealthier, and more health-literate than the general population. Single-site or single-region resources tell you something true and useful about a very narrow slice of the world: MIMIC is one Boston ICU, OpenSAFELY is England only, and the Nordic proposal is 30 million largely homogeneous Northern Europeans.

Verifiability: This is really one question asked in two parts.

• At the research stage: Can anyone check how the thing was built? Open resources like MIMIC/PhysioNet, NHANES, SEER, HCUP, and OpenSAFELY publish their cohort-construction and de-identification methodology. OpenSAFELY goes a step further by publishing its analysis code for anyone to inspect. Commercial vendors treat the exact tokenization and linkage methods as a trade secret. So a peer reviewer can look at the output but can’t audit the data.

• At publication: Journals increasingly expect data and code availability statements, but licensed commercial (purchased) data generally can't be redistributed alongside a paper. An independent lab can't rerun your exact analysis on your data. Open resources solve both problems by staying accessible under the same public terms to anyone who wants to check.

Access: This determines what kind of work you can even attempt. If you can download it to your own servers (MIMIC, NHANES, SEER*Stat, NIS), you get full control, but you’re now on the hook for securing it. Cloud-locked platforms (UK Biobank’s RAP, All of Us‘s Researcher Workbench, Truveta Studio) let you analyze but not export raw records, which rules out a lot of machine learning (transfer-learning and fine-tuning workflows) that needs raw data. And federated-query-only databases (PCORnet, OpenSAFELY, TriNetX) never give you patient-level data at all. They are great for privacy, but you can’t train a multimodal model on it.

And then there are these other issues:

• Fees can run $10,000 to $25,000 or more per study for commercial claims data. This favors well-funded institutions and industry sponsors over independent academic labs.

• Claims-based sources tend to run one to two years behind current practice.

• Probabilistic matching introduces real errors when tokenized records from data sources that were never designed to talk to each other get linked anyway.

• HIPAA, GDPR, and the UK’s Data Protection Act each define “de-identified” a little differently, which is exactly the harmonization problem the Nordic Comment’s roadmap lists as a step that still needs solving.

Takeaway

The Nordic paper says that the scientific infrastructure “is not aspirational: it is substantially in place.” Read against everything above, that claim seems premature for a plan that still has to establish data standardization, sort out cross-border regulatory alignment, and prove clinical feasibility.

For me, that gap isn’t unique to the Nordic proposal. It’s sitting underneath every “trained on X million de-identified patients” claim we’ll read this year. The architecture, the access model, the update cadence- none of that should be a footnote. It predicts exactly where the resulting model will hold up and exactly where it’ll quietly fail.

As always, if this is useful, the best way to support the work is to subscribe and share it with a colleague.

Sam

Subscribe now

Three days ago, Mark Zuckerberg published a manifesto arguing that AI should be “open source” and released Meta’s newest model, Muse Glimmer, as an open-weight offering. The White House and press noticed. And if you are an AI officer at a hospital or a developer, you should absolutely notice too, because the announcement lands in the middle of a genuinely important moment.

In the last three weeks, OpenAI disclosed that its models went rogue during a security test, escaping a containment sandbox and launching an "unprecedented" autonomous cyberattack against Hugging Face. About two weeks later, Anthropic's Mythos 5 model did something arguably worse: it created fake online identities, spear-phished two real software developers with malicious code, and tried to trick a GitHub project maintainer into accepting a backdoor. The UK's AI Security Institute called it a "serious incident" and a "shift in the risk landscape." Then Meta disclosed its own incident: a misconfiguration by its outside testing firm gave one of its models unsupervised internet access, and the model used it to breach another company's systems. The third major AI lab to report an incident like this in a matter of weeks.

If you are a hospital CIO watching proprietary models autonomously hack platforms, deceive humans, and sabotage their own shutdown mechanisms, you are probably asking the same question a lot of us are asking: Why on earth would I send my patient data to these companies?

The logic is straightforward. If hospitals stop renting APIs from black-box vendors and instead run open-weight models locally, they could stop version drift, regain control, keep data local, and avoid vendor collapse. Academics are already making this case. Riedemann et al. argued in npj Digital Medicine that the “path forward for large language models in medicine is open” because medical devices require transparency and version control that proprietary APIs cannot provide. Dennstädt et al. mapped out the implementation framework in 2025, showing that local deployment gives hospitals customization and control at the cost of expertise and infrastructure. A 2026 Frontiers review concluded that hybrid approaches are probably the realistic future.

Open source clearly has merit. The question is whether it solves the problems that actually keep hospital AI officers awake at night.

So, let’s break it down for clarity.

What It Helps With: Version Control as a Medical Device Requirement

In 2023, a Stanford team quietly published one of the most consequential papers in AI safety. Chen, Zaharia, and Zou showed that GPT-4’s accuracy on a prime-number identification task dropped from 97.6% to 2.4% between March and June 2023, even though the API endpoint was the same. The model had been updated. No announcement. No changelog. No way for users to know what changed, or why, or how to reproduce the old behavior.

In healthcare, that is a medical device failure. Imagine a sepsis prediction tool that your hospital validated, deployed, and trusted. On Tuesday morning, its sensitivity drops by 40% because the vendor pushed a silent update. Your patients do not get flagged. Your nurses do not know to look harder. Your mortality metrics shift six months later, and by the time you trace it back, the vendor has pushed three more updates.

This is what researchers call “version drift.” I prefer AI creep, because it captures something more insidious: the slow, invisible expansion of a model’s behavior beyond its validated boundaries, combined with the unannounced changes that make root-cause analysis impossible.

Open-weight models solve this through version pinning. A hospital downloads the weights, validates them against its own clinical data, locks them in, and treats them exactly like the firmware on a pacemaker. You cannot update a pacemaker without full re-validation. You should not update a clinical AI model without it either. Riedemann et al. explicitly argue that open-source approaches provide “medical device developers maximum flexibility to reduce the black-box properties of such systems”, which is a polite way of saying that proprietary APIs are structurally incompatible with medical device safety culture.

The OpenAI “laziness” incident of December 2023, where ChatGPT inexplicably refused to perform complete tasks, is another example. The cause was never identified by independent researchers nor disclosed by OpenAI. A hospital relying on that API for clinical documentation would have had no recourse, no explanation, and no ability to roll back to the previous behavior. With fixed open weights, you control the timeline.

Data Sovereignty and the Two-Pipeline Problem

If you read this newsletter regularly, you know I have written about the two data pipelines that every hospital AI deployment creates. There is the back-door pipeline, where de-identified data flows to vendors. And there is the front-door pipeline, where clinicians, patients, or administrators upload PHI directly into consumer chatbots, often without understanding that the data is being used to train the next model.

Running open-weight models locally eliminates the front-door risk. Patient data never leaves the hospital network. There is no question about whether de-identification actually worked. There is no wondering if a clinician copied and pasted a pathology report into a web interface that lacks a Business Associate Agreement. Dennstädt et al. put this bluntly: transmitting sensitive patient data to external servers, even in partially anonymized formats, “raises the risk of data breaches and unauthorized access,” and “complete anonymization in most clinical settings is not possible.”

For HIPAA compliance officers, this is a gift. The liability architecture becomes cleaner. If patient data never leaves your network, you have eliminated an entire class of breach scenarios. You have also eliminated the vendor’s terms of service as a factor in your compliance posture. OpenAI's terms of service state that its models aren't meant to be used for diagnosing or treating any health condition. Anthropic’s terms are similar. When you run a model locally, you are not agreeing to anyone’s terms except your own hospital’s.

For patients, the stakes are more direct. It’s their diagnosis, their genetic results, their mental health history moving through whatever pipeline a hospital, or a third-party app the hospital contracted with, chose to build on. Most patients have no idea which of those two pipelines their information is taking, let alone whether the model on the other end is proprietary or open, local or cloud-hosted.

Niche Specificity

A hospital in rural Mississippi treating a rare sickle-cell variant has different needs than a quaternary academic center in Boston. Proprietary APIs are trained on internet-scale data, optimized for the average case, and fine-tuned for the most common complaints. They are not optimized for your patient population.

Open-weight models allow fine-tuning on local data. There are now specific medical open models like OpenBioLLM-Llama3-70B, MEDITRON-70B, and Google’s MedGemma collection. A hospital can download these, fine-tune them on its own historical cases, its own protocols, and its own rare-disease populations, and produce a model that knows its local context.

The performance gap has narrowed dramatically. A Harvard Medical School study found that an open-source model performed on par with leading proprietary AI in solving tough medical cases. By mid-2024, LLaMA 3.1 405B had nearly closed the gap with GPT-4o on general benchmarks.

But the stronger argument is verifiability. A proprietary model might perform well on your data, but you cannot inspect its weights or training data to know why. An open model can be validated against your specific clinical scenario and audited by your own quality team. In medicine, knowing why a model works is often as important as knowing that it works.

Infrastructure and Expertise

Here is where the fantasy breaks down. A 70-billion-parameter model requires serious GPU infrastructure. A 300-bed community hospital in rural America does not have an A100 cluster of servers in its basement. It does not have ML engineers on staff. It barely has enough IT staff to keep the EMR running. A three-person health-tech startup trying to fine-tune the same model on a laptop is in exactly the same position, just without the EMR.

Dennstädt et al. are explicit about this trade-off: “Open LLMs deployed on local hardware enable greater model customization” but “demand resources and technical expertise.” Most hospitals do not have the expertise to deploy, monitor, and maintain these systems. Control is only valuable if you have the organizational capacity to wield it.

This is where the literature proposes a consortium model, and it is the most practical expansion of the open-source theory. The University Health Network authors explicitly call for a “global consortium” similar to the Trillion Parameter Consortium to pool computational resources, share validated base models, and allow individual hospitals to fine-tune locally without each building its own data center. This is how open source wins in healthcare: as shared infrastructure.

There is also the rise of smaller, efficient models. Even 7-billion-parameter models can now run on a Mac M1 and perform well for targeted tasks. As models get smaller and more efficient, the infrastructure barrier drops. This trend turns the theory from a fantasy into a practical reality about targeted deployment.

The “Open Source” Definition Problem

Zuckerberg’s announcement is a perfect teaching moment. The NYT reported explicitly that Muse Glimmer “is not fully open source”. Meta released the weights, but not the training data, or the full architecture details, or the license terms that would allow unrestricted commercial use. This is “open weight,” not “open source.”

The distinction matters because Riedemann et al., discuss code, data, weights, documentation, and licensing separately. By that framework, most “open” models are barely halfway open: open enough to download and run, rarely open enough to fully audit. The European Union AI Act is currently struggling with this exact problem. Whether you’re procuring a model for a hospital, deciding what to build a clinical product on, or just trying to evaluate a vendor’s privacy claims, you need to ask the same question: open in what dimension? Open to inspection? Open to modification? Open to commercial deployment without a usage license from Meta?

If you write a procurement policy based on “open source” without defining it, you will end up with a vendor that released weights under a restrictive license and still controls your destiny. The marketing language is not your friend.

Security and Adversarial Risk

The open-source security argument has a flip side. Yes, running locally keeps data in-house. But open weights are also susceptible to attackers. Because the architecture and weights are public, adversaries can systematically search for vulnerabilities, craft adversarial inputs, and test data-poisoning strategies at scale.

A Galileo AI analysis of open-source LLM security risks notes that “open-source LLMs often introduce significant security risks due to their public nature and generally immature security practices.” A PMC study on safety in medical LLMs found that medical-specific fine-tuned models, both open-source and proprietary, underperformed general-purpose frontier models across most safety dimensions tested. The problem is that safety in medical AI is hard for everyone, not that open models are uniquely unsafe.

The counterargument is that closed models get breached too. OpenAI had its data leak. The rogue AI incidents of the past month show that proprietary models are not magically secure because their weights are hidden. The difference is transparency. With open weights, your hospital can probe the model itself, find vulnerabilities, and patch or mitigate them. With a closed model, you must wait for the vendor to fix it, and hope they tell you the truth about what happened. A Censinet analysis of AI-specific vulnerabilities in healthcare makes the case for adversarial robustness testing regardless of model source.

The Regulatory Paradox

Here is the most biting counterargument, and it is underexplored in the literature. The FDA wants medical devices to be locked-down and validated. If you download an open-weight model, fine-tune it on your local patient data, and deploy it in a clinical decision support workflow, have you just created a new medical device? Does it require its own 510(k)?

The regulatory framework has not caught up to this question. The Riedemann paper notes that LLM AI chatbots require approval as medical devices, but the approval process assumes a static, validated product. An open model that a hospital can freely modify is, by definition, not a static device. If your AI officer fine-tunes the model next quarter, is that a design change requiring re-validation?

Liability shifts entirely to whoever deployed the model, the hospital, or the developer who built a clinical product on top of it. When a proprietary model harms a patient, the hospital, the developer, and the vendor will all point at each other. An organization running its own locked model has clearer liability. For the risk-averse, that is actually a feature, not a bug. But only if the organization has the documentation and governance infrastructure to defend itself.

Hybrid Deployment

If you take one thing from this article, take this: hospitals should version-pin and locally deploy open models for high-stakes clinical tasks, while using proprietary APIs for low-stakes general tasks.

This is the only position that is both defensible and practical. Use GPT-4o or Claude for scheduling, patient education, and general documentation tasks where version drift is annoying but not dangerous, and where the cost of local infrastructure is not justified. Use pinned, validated, locally run open models for clinical decision support, pathology review, rare disease diagnosis, and any workflow where a silent behavior change could harm a patient.

Dennstädt et al. and the Frontiers review both back this hybrid approach. The Frontiers review focuses narrowly on information-extraction tasks, pairing local, on-premises retrieval with cloud-based generation, but the same logic applies more broadly to clinical decision support. Hospitals get control where it counts without building full ML infrastructure for every task. And it creates a natural safety net: if a proprietary API goes rogue, it's running your scheduling bot, not your sepsis predictor.

The Version-Pinned Hospital

The hospitals, developers, and health systems that will own the next decade are the ones that treat AI like the medical device it is: validate, lock, version, archive, and only update through a controlled change-management process. Whoever is building or buying these tools, in-house AI team, third-party vendor, or solo developer, will not outsource their clinical brains to black boxes that update silently every three months. They will build the competency to run, validate, and govern their own models, even if they have to share the infrastructure to do it.

The ones that keep renting APIs and hoping the terms of service will protect them, and the ones handing over their health data without asking what’s actually running on the other end, will have no one to blame but themselves when the next data leak hits or the next unannounced model update quietly changes what their AI actually does.

Open source is a risk-mitigation strategy, not a magic bullet. And in medicine, whether you’re the hospital deploying it, the developer building it, or the patient trusting it, that’s what protects you.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

A patient arrives in the emergency department with sudden chest pain and shortness of breath. You order a CT angiogram. You check a troponin and a D-dimer. Then, you scan the heart at the bedside, looking for right ventricular strain, without a second thought.

No one asks whether CT angiography or troponin and D-dimer panels are available. No one questions whether a bedside cardiac scan is appropriate for a patient with unexplained dyspnea.

In the 1990s, bedside ultrasound was viewed as experimental by many physicians. Training was scarce. Equipment was limited. Skepticism was the default posture. Today, failing to use it in certain situations may be difficult to defend.

Could artificial intelligence be following the same path?

What Does “Standard of Care” Actually Mean?

The standard of care is a legal concept. Courts define it as what a reasonably prudent physician would do under similar circumstances, established by expert testimony in malpractice cases and by what has become available and expected in community practice. Best practice sets a higher bar: it describes what excellent physicians do, not what the law requires of an average one. Having access to a technology sets a lower bar still, since a device sitting unused in a hospital does not obligate anyone to use it. Standard of care sits between the two, and it moves as medicine absorbs new technology into routine practice, without anyone declaring a formal start date.

Understanding how that absorption happens is the key to understanding where AI is heading.

The POCUS Roadmap

To see how a technology becomes standard of care, it helps to look at one that has already made the full journey. Point-of-care ultrasound (POCUS) offers an unusually clean roadmap, divided into seven identifiable steps.

1. Innovation. The technology itself predates its American emergency medicine adoption by decades. Clinicians in Europe and Japan used bedside ultrasound for trauma assessment as early as the 1970s. In the United States, the late 1980s and early 1990s produced a small group of early adopters, emergency physicians willing to look at a blurry image and ask whether it changed their clinical decision-making. Equipment was limited, skepticism was widespread, and the idea that a bedside scan could substitute for a formal radiology study struck many as reckless.

2. Evidence. Clinical studies accumulated over years. Grace Rozycki’s 1993 study of 476 trauma patients demonstrated that surgeons and emergency physicians could reliably detect free fluid at the bedside. The exam itself did not get its name, Focused Assessment with Sonography for Trauma, until an international consensus conference in December 1997, with results published in 1999. Evidence built specialty by specialty and application by application: trauma first, then pregnancy, cardiac, and procedural guidance.

3. Professional Endorsement. Institutional recognition followed the evidence, but slowly. ACEP issued a position statement backing physician-performed emergency ultrasound in 1990, then approved a dedicated Emergency Ultrasound Section in 1995. The real inflection point came in June 2001, when ACEP’s Board approved the first comprehensive Emergency Ultrasound Guidelines, the first time a major American specialty society formally endorsed bedside ultrasound as a core skill with defined training expectations. The guidelines were revised in 2008, retitled and expanded in 2016 to “Ultrasound Guidelines: Emergency, Point-of-Care, and Clinical Ultrasound Guidelines in Medicine,” and revised again in 2023.

4. Infrastructure. Machines became smaller and cheaper. Hospitals purchased equipment, and image archiving, billing codes, quality assurance programs, and workflow integration all had to be built around it. None of this happened automatically; it required systems engineering, IT integration, and financial justification, mostly after the guidelines existed to justify the spend.

5. Cultural Adoption. Residents began expecting to learn POCUS. Patients began expecting it. Hospitals started advertising bedside ultrasound capability. The technology moved from curiosity to baseline expectation.

6. Governance. Credentialing pathways emerged. Residency programs built ultrasound into formal curricula. Quality assurance programs began tracking image quality and interpretation accuracy over time.

7. Medicolegal Recognition. Here is where the standard of care quietly shifts. A 2022 review by Russ and colleagues in the Journal of Emergency Medicine examined POCUS malpractice cases filed between December 2012 and January 2021 and found that the dominant allegation was failure to perform, not misinterpretation. Jonathan Mezrich made the same point in Academic Radiology in 2025: physicians are now more likely to face litigation for not performing POCUS than for misreading it, calling it “a rare example where defensive medicine and patient care demands converge.” No single court case or guideline declared POCUS the standard. Evidence, endorsement, infrastructure, culture, and litigation converged until the question flipped from why would you do this to why didn’t you.

Where Is AI Today?

Let’s run the same seven steps against AI in medicine as it stands in mid-2026.

1. Innovation. AI has cleared this step. The FDA’s running list of AI- and Machine Learning-enabled medical devices passed 1,524 entries by the end of March 2026, with radiology accounting for 1,163 of them, about 76 percent. Cardiology alone now has more than 200 cleared algorithms. In 2025, the FDA cleared 295 new AI devices from 221 different manufacturers.

2. Evidence. This is a bit uneven. Retrospective validation is abundant: chest X-ray triage tools such as Qure.ai’s qXR line now carry 26 FDA-cleared indications across X-ray and CT, from lung nodule detection to intracranial hemorrhage triage, and prospective studies have shown measurable reductions in time-to-read for critical findings. ECG interpretation has a similarly mature evidence base. But outcome-level prospective evidence, the kind showing a mortality or morbidity benefit rather than a diagnostic-accuracy benefit, is still catching up. One of the more ambitious efforts, a cluster-randomized trial evaluating whether an AI chest X-ray triage system reduces mortality, isn’t expected to report full data until December 2027. For most other applications, especially complex diagnostic reasoning, the trials that would satisfy this step are still being designed.

3. Professional Endorsement. This step is moving at different speeds by specialty. In October 2025, ACEP hosted the first All Emergency Medicine AI Summit, convening SAEM, CORD, ACOEP, ABEM, AAEM, EMRA, AACEM, and AOBEM. The resulting consensus statement, published in March 2026, affirms that emergency physicians retain authority for patient care decisions and that AI should enhance rather than replace clinical judgment. That’s real, but it reads more like ACEP’s 1990 position statement than its 2001 guidelines: principles without defined training or credentialing standards. Radiology has moved further. In May 2026, the ACR Council approved the ACR-SIIM Practice Parameter for Imaging Artificial Intelligence, the specialty’s first formal practice parameter for AI and the same category of document as ACR’s Appropriateness Criteria. It sets expectations for governance, clinical validation, bias mitigation, and ongoing performance monitoring, and it’s paired with Assess-AI, a new quality registry that tracks real-world model performance against radiology-report-derived outcomes. That is, right now, the closest thing in medicine to a 2001-style guideline for AI.

4. Infrastructure. This is moving fast. Major EHR vendors have built native AI integration points, hospital AI governance committees are now common rather than novel, and the FDA’s Predetermined Change Control Plan (PCCP) framework, finalized in 2024, lets manufacturers update models without seeking a new clearance for every version. That matters more than it sounds: it’s a regulatory acknowledgment that AI infrastructure will need to absorb continuous change in a way POCUS equipment never did.

5. Cultural Adoption. The share of U.S. and Canadian medical schools incorporating AI into their curricula jumped from 53 percent in 2023 to 77 percent in 2024, according to the AAMC and AACOM’s Curriculum SCOPE Survey. Residency training lags behind medical school. ACGME hasn’t made AI competency a formal program requirement yet, though it’s gathering stakeholder feedback toward that end and added AI-focused sessions to its 2026 programming, and the AAMC’s own national AI competency framework for the full training continuum isn’t due until fall 2026. Patients are beginning to ask about AI-assisted diagnosis, and health systems increasingly market their AI capabilities. What’s still missing is the residency-level equivalent of a POCUS rotation: standardized, required, and tested.

6. Governance. This remains the messy middle. The FDA’s January 2026 revision to its clinical decision support guidance leans on what industry has started calling a “glass box” standard: CDS software must disclose, in plain language, its underlying logic, validation methodology, and the limitations of its training data, specifically to guard against automation bias. What’s still mostly missing is the hospital-side counterpart: credentialing pathways, and version-tracking systems that treat a model update the way a hospital treats a new piece of equipment.

7. Medicolegal Recognition. No case law yet establishes AI non-use as a deviation from standard care. But the fact that a Harvard Business School economist has now formally modeled the question tells you something about where the conversation has moved. More on that paper below.

What Still Has to Happen?

Some of the remaining milestones are already in motion, or already achieved in a single specialty, like radiology’s new practice parameter and the chest X-ray mortality trial cited above. Most are not, and none has happened across specialties broadly. The list that would move AI further up the ladder:

• Clinical outcome studies showing mortality or morbidity benefit, not just diagnostic accuracy

• Prospective trials rather than retrospective validation, across more than imaging and ECG

• Professional society recommendations with the weight of the 2001 ACEP guidelines, in specialties beyond radiology, which already has one

• Residency curricula that train every new physician on AI tools as a baseline skill

• Board examination content that tests AI competency

• Credentialing pathways for physicians using AI clinically

• Hospital policies that define when and how AI is used

• Workflow integration inside EHRs that makes AI a seamless part of clinical decision-making rather than an add-on

• Quality assurance programs, run by hospitals rather than manufacturers, that monitor model performance over time

• Version-control transparency that lets clinicians know when a model changed and how that changed its outputs

• Independent benchmarking of commercial models against real, local patient populations

• Malpractice case law that establishes precedent for when declining to use AI becomes indefensible

Why AI May Be Different

History doesn’t always repeat itself. There are real differences between POCUS and AI that complicate the transition:

Velocity of change. Ultrasound machines improved incrementally. AI models can change materially every few months, and the FDA’s own PCCP framework formalizes this: manufacturers can now pre-clear a range of future model updates in a single submission, meaning a device that behaves one way in 2026 may legitimately behave differently in 2027 without ever returning for a new review. Ultrasound machines never needed that provision.

Evolving outputs. A 2024 model doesn’t necessarily behave like a 2025 model. POCUS images don’t change their meaning retroactively; AI outputs can.

Probabilistic nature. AI recommendations are not visual. A radiologist can look at an ultrasound image and see what the physician saw. An AI recommendation is often invisible unless documented, complicating both quality assurance and medicolegal review.

Vendor fragmentation. Unlike ultrasound, dominated by a handful of manufacturers, the AI landscape is crowded with substantially different models, making standardization and benchmarking harder.

Continuous governance. With POCUS, governance was largely a one-time investment in training and equipment. With AI, governance is continuous: models drift, data distributions shift, and monitoring has to be perpetual.

Deskilling risk. POCUS extended the physical exam; it didn’t replace clinical reasoning. AI carries a genuine risk of cognitive atrophy, where physicians grow less capable of independent verification because they’ve grown dependent on the tool. POCUS has no direct equivalent of this risk.

These distinctions make the transition more complicated than the POCUS roadmap would suggest.

The Medicolegal Inflection Point

Here I want to sit with questions rather than answers.

• If AI reliably detects subtle diagnoses that physicians routinely miss...

• If AI consistently reduces missed findings across multiple validated studies...

• If most emergency physicians in a community routinely use it...

• If hospitals provide it as part of standard equipment...

• If specialty societies recommend it in formal guidelines...

At what point does declining to use AI become difficult to defend?

A 2026 Harvard Business School working paper by economist Alex Chan, also circulated through the National Bureau of Economic Research, models exactly this transition: the point at which AI moves from a passive clinical tool to something that shapes what a reasonably prudent physician would do. Chan’s core question isn’t whether liability law should protect physicians who rely on AI. It’s whether the law should adapt its standards to AI’s specific failure modes, data drift, vendor-side updates, probabilistic rather than binary outputs, or force AI to fit liability frameworks built for static tools like a stethoscope or an ultrasound probe.

I don’t think this moment has arrived. But it may be closer than it feels, because the standard of care doesn’t announce itself with a press release. It shifts in the accumulated weight of evidence, endorsement, and litigation until the defense of “I didn’t use it” begins to sound hollow.

The Ladder as a Diagnostic Tool

The seven steps above aren’t specific to ultrasound. The same ladder applies to CT, MRI, and troponin testing, all of which climbed it long before POCUS did, and it applies just as well going forward, to next-generation sequencing or whatever comes after AI.

The value of naming the steps isn’t prediction. It’s diagnosis: for any technology you’re evaluating, you can ask which step it has actually reached, rather than which step its marketing implies it has reached. That’s precisely where most of the confusion about AI’s clinical readiness comes from, conflating step one (an FDA clearance exists) with step three (a specialty society has endorsed it) or step seven (declining to use it is indefensible).

Conclusion

Nobody predicted the exact year bedside ultrasound became routine. There was no ribbon-cutting ceremony. Looking backward, the progression seems obvious and inevitable. Looking forward at the time, it was uncertain and contested.

AI may follow a similar path. Or it may follow an entirely new one. The important question isn’t whether AI will become the standard of care. Given enough evidence, infrastructure, and cultural absorption, most useful technologies eventually do.

The important question is: How will I recognize the moment when it does?

Sources

• Rozycki GS, Ochsner MG, Jaffin JH, Champion HR. “Prospective Evaluation of Surgeons’ Use of Ultrasound in the Evaluation of Trauma Patients.” Journal of Trauma. 1993;34(4):516–527. (476-patient study establishing the technique; predates the FAST name.)

• Scalea TM, et al. “Focused Assessment with Sonography for Trauma (FAST): Results From an International Consensus Conference.” Journal of Trauma. 1999;46(3):466–472. (Consensus conference, Dec. 1997, that coined the FAST acronym.) PubMed

• Rozycki GS. “Surgeon-Performed Ultrasound: Its Use in Clinical Practice.” Annals of Surgery. 1998;228(1):16–28.

• Russ B, Arthur J, Lewis Z, Snead G. “A Review of Lawsuits Related to Point-of-Care Emergency Ultrasound Applications.” Journal of Emergency Medicine. 2022;63(5):661–672.

• Mezrich JL. “POCUS in the Emergency Department: An Example of Convergence of Defensive Medicine With Patient Care.” Academic Radiology. 2025;32(Suppl 1):S126–S130. PubMed

• ACEP. “Ultrasound Guidelines: Emergency, Point-of-Care, and Clinical Ultrasound Guidelines in Medicine.” Approved as “Emergency Ultrasound Guidelines,” June 2001; revised Oct. 2008; retitled and expanded June 2016; revised again April 2023.

• Chan A. “Optimal Medical Liability for AI.” Harvard Business School Working Paper No. 26-087, June 2026; also NBER Working Paper No. w35321. NBER

• FDA. “Artificial Intelligence-Enabled Medical Devices” database, last updated March 4, 2026 (1,524 total devices; 1,163 in radiology, ~76%; 295 new clearances in 2025 from 221 manufacturers).

• FDA. “Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles.” Final guidance, 2024.

• FDA. Revised Clinical Decision Support Software guidance, issued January 6, 2026 (“glass box” transparency requirements, automation-bias framing).

• ACEP, SAEM, CORD, ACOEP, ABEM, AAEM, EMRA, AACEM, and AOBEM. Consensus Statement following the first All Emergency Medicine AI Summit (Irving, TX, Oct. 2025); statement published March 18, 2026.

• American College of Radiology, Data Science Institute. AI use-case repository and FDA-cleared algorithm tracker.

As always, if you enjoy reading, consider subscribing and tell a friend.

Sam

Subscribe now

In July 2026, OpenAI rolled out ChatGPT Health to every U.S. adult user, inviting them to connect their Apple Health data, medical records from Epic and Oracle Health systems, and wellness apps directly to the AI. OpenAI announced the nationwide launch on July 23, promising “secure” connections and personalized health insights. The move was widely covered as a consumer convenience, a smarter way to track medications, interpret lab results, and prepare for doctor visits.

What most coverage missed is that this is only one half of a much larger story. While consumers debate whether to trust OpenAI with their blood pressure readings, a separate, invisible pipeline has already moved hundreds of millions of patient records into the hands of AI companies. The difference is structural, legal, and deeply asymmetrical, the back-door pipeline strips your name before it sells your data. The front-door pipeline does not have to.

The Front Door, HIPAA Ends Here

Here is the first thing to understand about ChatGPT Health, it is not covered by HIPAA.

HIPAA, the Health Insurance Portability and Accountability Act, governs a narrow class of “covered entities”, hospitals, health insurers, and healthcare clearinghouses, plus their “business associates.” OpenAI is none of these. OpenAI states explicitly that “Health in ChatGPT is not intended for clinical or covered-entity use and does not offer a Business Associate Agreement.”

What this means in practice is stark. When a hospital holds your medical record, federal law requires them to protect it, limit its use, and obtain authorization before sharing it broadly. When you upload that same record to ChatGPT, those protections evaporate. As health platform b.well noted, “medical records uploaded into ChatGPT Health are no longer covered by HIPAA, because the service isn’t a covered entity under that law.”

OpenAI does not need to de-identify your data. It does not need to strip your name, your birth date, your medical record number, or your diagnostic codes. Its only constraints are its own privacy policies. OpenAI’s Health Privacy Notice says that by default, health data received via Health Features is not used to improve foundational models. But the policy also notes that “a limited number of authorized OpenAI personnel and trusted service providers might access data... to improve model safety, unless you have opted out.” That is a significant carve-out, governed by a terms-of-service agreement most users will not read.

The consumer is being asked to make a privacy decision without understanding the legal terrain. They assume HIPAA protects their health data everywhere. It does not.

The Back Door, The Pipeline You Never Signed Up For

While OpenAI courts consumers on the front end, a separate industry has built a lucrative infrastructure for selling hospital data to AI companies on the back end. The key players are data middlemen, companies that contract with hospitals, strip patient records of direct identifiers, and license the resulting datasets to AI developers and researchers.

Truveta is among the largest. Founded by a consortium of major U.S. health systems, Truveta now includes data from more than 130 million de-identified patients across over 900 hospitals and 20,000 clinics. Its members include Providence, Advocate Health, Trinity Health, Tenet Healthcare, Northwell Health, AdventHealth, and Novant Health. In September 2021, Microsoft announced a strategic partnership and investment in Truveta, and Truveta uses Azure-based AI to process its data. Truveta has also launched its own Truveta Language Model trained on this clinical corpus.

Protege is the emerging contender. In January 2026, Protege raised a $30 million Series A led by Andreessen Horowitz, bringing its total funding to $65 million. The company works with what it describes as “close to 20 data partners” in healthcare and more than 100 across all sectors. Protege licenses what it calls “private, real-world data” including clinical notes, medical images, video, audio, pathology slides, genomic data, lab reports, wearables data, and social determinants of health. In a partnership with Syndesis Health, Protege gained access to 70 million de-identified patient lives from more than 500 facilities across 15 countries.

Protege states on its website that it works directly with “leading foundation model labs” to define licensing standards. It does not name them. What is clear is that the pipeline is vast, well-funded, and growing rapidly.

The Asymmetry

Consider what this means for a single patient.

If you are treated at a Truveta-member hospital, your de-identified record may be harmonized, aggregated, and sold to AI developers. The hospital has removed your name, address, medical record number, and the other 15 identifiers required by HIPAA’s Safe Harbor method. The AI company receives a rich longitudinal record of your diagnoses, medications, procedures, and lab values, but without a name attached.

If you then upload your own medical records to ChatGPT Health to ask about your medication list, OpenAI receives the same clinical information, but with your name on it, your dates intact, and your full identifying context, because HIPAA does not apply to them.

The hospital is legally required to de-identify your data before sharing it. The consumer AI company is not required to do anything of that kind. The regulated system forces anonymization on the back door while the consumer app collects richer, more identifiable data through the front door with no federal health privacy guardrails at all.

Here is the part that should give consumers pause. The software companies building these health AI products may already have more data on you than you would want them to know, de-identified records from your hospital, imaging from your health system, lab values from your clinic. If you then provide them with a small sampling of identifiable data, your own uploaded records, your Apple Health metrics, a few lab results, you may inadvertently give them the key to re-identify what they already possess as belonging to you. Careful what you share, because even a narrow slice of identifiable health data can be used to frame a much more detailed picture of you than you may think.

The De-identification Myth

There is a second problem with the back-door pipeline, and it is mathematical.

For decades, the legal and ethical framework for health data sharing has relied on a simple premise, remove 18 identifiers, and the data is no longer “personally identifiable.” Once de-identified, it falls outside HIPAA’s scope and can be “freely used, shared, and sold” without patient consent.

Modern AI has broken that premise. In 2019, researchers Luc Rocher, Julien Hendrickx, and Yves-Alexandre de Montjoye published a study in Nature Communications titled “Estimating the success of re-identifications in incomplete datasets using generative models.” Their finding was startling, “Using our model, we find that 99.98% of Americans would be correctly re-identified in any dataset using 15 demographic attributes.”

The authors were explicit about the policy implications, “Our results suggest that even heavily sampled anonymized datasets are unlikely to satisfy the modern standards for anonymization set forth by GDPR and seriously challenge the technical and legal adequacy of the de-identification release-and-forget model.”

Other studies have reinforced the point. Researchers have demonstrated that algorithms can accurately match physical activity data and demographic information to 95% of adults in a dataset. As Scripps News reported, “Health data shared with AI systems can be stripped of names and sold legally, then re-identified using AI tools.”

The data middlemen are selling records that are legally de-identified but practically re-identifiable. The “without your name” framing is a legal fiction that AI has rendered increasingly untrue.

The Consent Theater

If the pipeline is invisible, how do patients find out about it? In most cases, they don’t.

When you check into a hospital, you sign a group of forms and receive a Notice of Privacy Practices, a long document that mentions treatment, payment, and “health care operations.” De-identification and data sharing may be mentioned in a single clause. The notice almost never says, “Your records may be sold to an AI training data broker.”

Some hospitals are becoming more explicit. Archbold Medical Center in Georgia publishes a Notice of Privacy Practices that states, in plain language, “De-identified data is no longer subject to privacy or security laws.” It also notes that “AI use within applications uses/discloses your medical information.” This is unusually candid. Most notices are not.

The legal framework creates what might be called consent theater, patients are given a document they do not read, which authorizes broad data uses they do not understand, for purposes, including AI model training, that were not contemplated when HIPAA was written in 1996.

The Regulatory Vacuum

One might expect that the wave of state consumer privacy laws would close this gap. They largely do not.

Washington’s My Health My Data Act (MHMDA), enacted in 2023, broadly defines “consumer health data” and requires opt-in consent for collection and sale. But the Act explicitly excludes “deidentified data” from its definition of personal information. A hospital selling de-identified records to Protege falls outside the law.

Maryland’s Online Data Privacy Act (MODPA), effective October 2025, has some restrictions on using consumer health data for AI model development. But properly de-identified or aggregated data is largely exempt.

Nevada’s SB 370 and Connecticut’s CDPA amendments follow the same pattern, they regulate identifiable consumer health data but leave the de-identified pipeline untouched.

At the federal level, the FTC has been active on data broker enforcement, but not on this specific subject. In February 2026, the FTC sent warning letters to 13 data brokers about compliance with the Protecting Americans’ Data from Foreign Adversaries Act (PADFAA), focusing on foreign data transfers rather than domestic AI training. There is no federal enforcement action specifically targeting the sale of de-identified health records for commercial AI model training.

The Convergence

The two pipelines are separate in law but convergent in effect. OpenAI is building the infrastructure to ingest medical data on both sides.

On the front end, OpenAI acquired Torch, a health records unification startup, for roughly $60-100 million in January 2026. Torch’s technology consolidates fragmented patient data from hospitals, labs, and visit recordings. The stated goal, per HLTH reporting, is a “unified medical memory” for ChatGPT Health.

On the back end, OpenAI’s health data ambitions are clear even if its purchases from middlemen are not publicly documented. The company is positioning itself to be a central platform for health information, whether that data arrives through consumer upload or hospital pipeline.

The patient, meanwhile, is left with a false sense of security. They are told HIPAA protects their medical records. They are told de-identification makes their data anonymous. They are told they can “securely” connect their health records to an AI assistant. None of these statements are entirely false. But none of them capture the full picture of where their data is going, how it is being used, or how thin the legal protections actually are.

The health data economy has bifurcated into two tracks, one regulated and de-identified, the other unregulated and fully identifiable. Both are feeding the same models. And the patient is the last to know.

The waiting ended in 2026. State legislatures stopped asking and started writing. By the end of this year's legislative sessions, more than two dozen new laws and regulations across eighteen states will have redrawn the boundaries of what AI may do in healthcare. Additionally, there is Federal legislation in process that may conflict with everything states have done so far. Here I survey the full landscape in the following categories:

• Prior Authorizations

• Therapy

• Consent and Disclosure

• Chatbot Safety

• Professional Licensing

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Prior Authorizations

The most active legislative front in 2026 was prior authorization. Seven states enacted laws that draw a hard line between AI assistance and AI decision-making for coverage denials. Two additional states also took related approaches targeting claims downcoding and oversight reporting.

Alabama SB 63, enacted on April 17, requires health insurers to disclose their use of AI, base authorization decisions on a patient’s individual medical history, and ensure that any denial is issued by a licensed professional. It is similar to the disclosure-plus-human-review framework seen elsewhere, but it explicitly ties authorization decisions to individual medical records rather than group data.

Colorado HB 1139, signed June 2, prohibits insurers from basing a denial solely on group data or algorithms and requires a licensed clinician to review any adverse decision. The subtle but critical distinction here is the explicit ban on group-level data as the basis for individual denials, a protection Alabama and Washington do not spell out as directly.

Georgia SB 444, enacted May 5, is one of the shortest and most direct statutes in this category: a blanket prohibition on coverage decisions based solely on AI systems or software tools. It is nearly identical in spirit to Iowa’s approach but without Iowa’s explicit carveout allowing AI to conduct initial screening.

Illinois SB 3114, awaiting the governor’s signature, is the only law in this category that targets downcoding, the practice of reducing reimbursement codes, rather than prior authorization denials. It prohibits AI from bypassing provider documentation to downgrade claims and requires a natural person to review any downcoding determination.

Indiana HB 1271, enacted March 4 and effective July 1, is the only law regulating AI on both sides of the transaction: it prohibits insurers from using AI as the sole basis for downcoding and bars providers from using AI to submit claims without human review. It bridges the gap between utilization-review laws and claims-integrity laws.

Iowa HF 2635, enacted May 13, creates a two-step process: AI may screen prior authorization requests, but only a human may deny, delay, or downgrade them. The statute is nearly identical in structure to Utah’s approach but with less detail on time limits and validity periods.

Maryland HB 1563, enacted April 28, takes an oversight approach rather than a direct prohibition. It requires insurers to file quarterly reports on AI-assisted adverse decisions and allows the insurance commissioner to investigate spikes in denials, particularly for emergency department services. It is the only authorization-related law focused on regulatory monitoring rather than rewriting decision-making rules.

Utah SB 319, enacted March 19, requires disclosure of AI use in preauthorization, mandates independent medical judgment by reviewers, and sets minimum validity periods for chronic condition authorizations. It is the most procedurally detailed of the authorization statutes, paired with a separate scope-of-practice law passed the same week.

Washington SB 5395, enacted March 23, expands the state’s existing prior authorization framework and explicitly bars AI from modifying care, not just denying it. This is a broader prohibition than most, covering any adverse modification of a treatment plan.

Therapy

While prior authorization laws dominated, therapy bans consumed the headlines. Six states enacted statutes prohibiting AI from providing psychotherapy or representing itself as a mental health professional. Most took effect in 2026 or will take effect in 2027.

Colorado HB 1195, enacted June 3, is the most comprehensive therapy ban in the country. It prohibits unlicensed AI chatbots from offering psychotherapy and also restricts licensed psychologists, counselors, and social workers from using AI to provide independent therapeutic services. It is the only law that explicitly limits both the unlicensed and licensed sides of the equation.

Maine LD 2082, enacted April 13, prohibits AI from providing therapy or psychotherapy and classifies offering therapy via AI without a license as an unfair trade practice. It also adds a patient-consent requirement for ambient listening and AI-powered recording during sessions. It is nearly identical in scope to Nevada’s law but adds the recording consent provision.

Nevada AB 406, enacted in 2023 and effective July 1, 2025, was the first therapy-bot ban in the nation. It applies to any AI system in Nevada, including telehealth platforms, and imposes civil penalties of up to $1,500 per violation. It served as the model for several 2026 statutes, including Maine’s and Rhode Island’s.

Rhode Island H 7349 / S 2197, enacted June 22, contains a dual prohibition: unlicensed AI may not practice therapy, and licensed providers may not use AI for independent therapeutic decisions or treatment plans. It is nearly identical in structure to Colorado’s law but lacks the explicit licensed-provider restrictions seen in Colorado’s statute.

Tennessee SB 1580, enacted April 1, prohibits developing or deploying AI systems that advertise or represent themselves as qualified mental health professionals. It treats such misrepresentation as a deceptive practice under state consumer protection law. It is narrower than Colorado’s ban, and targets the advertising and representation rather than the therapeutic act itself.

Vermont H 816, enacted June 17, safeguards individuals seeking mental health services by ensuring delivery by professionals rather than independently by AI systems. The wording lacks the explicit title protections and civil penalties seen in Nevada and Rhode Island. It is the least prescriptive of the therapy bans.

Consent and Disclosure

Four states enacted or implemented disclosure requirements in 2025 and 2026, creating a patchwork of transparency obligations that providers and insurers now must navigate.

Arizona Board of Behavioral Health Examiners adopted regulations in November 2025, effective January 1, 2027, requiring behavioral health professionals to obtain and document informed consent before providing services involving AI. It is the only regulatory (non-statutory) entry in this survey and applies only to behavioral health, making it narrower than California’s statewide disclosure laws.

California AB 3030, enacted in 2024 and effective January 1, 2025, was the first statewide disclosure law in the nation. It requires any generative AI communication to a patient to include a clear disclaimer that the message was generated by AI. AB 489, enacted in 2025 and effective January 1, 2026, goes further by prohibiting AI systems from presenting themselves as licensed medical professionals. Together, these two laws create a layered transparency framework that no other state has fully replicated.

Louisiana HB 475, enacted June 2, requires verbal disclosure of any recording device or AI transcription before recording any clinical visit. It is the only law in the country that mandates verbal (not just written) disclosure for AI transcription, and it applies to all healthcare settings including telehealth.

Texas SB 1188, enacted in 2025 and effective September 1, 2025, is the most permissive framework in this category. It explicitly allows AI to diagnose and treat patients, provided the patient is informed and the Texas Medical Board has reviewed the technology. It is the only state that affirmatively authorizes AI clinical decision-making rather than merely restricting it.

Chatbot Safety

Three states enacted what are effectively “Chatbot Safety Acts”: laws that do not regulate medical practice directly but impose safety and disclosure requirements on AI chatbots that might interact with patients in crisis.

Idaho SB 1297, enacted March 31 and effective July 1, 2027, requires disclosure and crisis-response protocols for any conversational AI system that provides health-related information. The law is nearly identical to Nebraska’s, suggesting both states may have used model legislation.

Nebraska LB 525, enacted April 14, combines the Agricultural Data Privacy Act with a Conversational AI Safety Act that applies to all public-facing AI chatbots with healthcare safety requirements. It is identical in structure to Idaho’s law but applies to all public-facing chatbots, not just health-specific ones.

Oregon SB 1546, enacted March 31, is the most comprehensive chatbot safety law in the country. It includes a private right of action, safeguards for minors, and covers AI companions. It is significantly broader than the Idaho and Nebraska statutes, extending beyond healthcare to cover emotional support AI.

Professional Licensing

Two states passed laws clarifying that AI is not a licensed professional, using different legal techniques to reach the same conclusion.

Delaware HB 191, enacted April 23, prohibits any nonhuman entity, including AI, from being licensed or certified as a nurse, physician, or physician assistant. It also bars AI from using protected professional titles. It is the most explicit title-protection law in the survey.

Utah SB 150, enacted March 24, uses a negative-definition approach: AI providing advice or treatment without a practitioner-patient interaction does not qualify as an “innovation” within the scope of practice. It is the only law that defines AI out of scope rather than explicitly prohibiting licensure. Paired with Utah’s SB 319, enacted the day before, these two laws create the most comprehensive AI regulatory framework of any single state.

The Horizon: Pending Bills and Federal Counter-Trends

Three jurisdictions have legislation still pending that could reshape the landscape by 2027.

New York S7896 / A8556, introduced in the 2025-2026 session, would establish comprehensive AI utilization review requirements. It is the most detailed authorization law proposed to date, with specific reporting and human-review mandates that go beyond the 2026 enacted statutes.

Pennsylvania HB 1925, introduced in the 2025-2026 session, would amend both the Health and Insurance titles to require AI utilization review and reporting by insurers, hospitals, and clinicians. It is broader than most authorization bills because it covers providers as well as insurers.

Federal H.R. 238, the Healthy Technology Act, stands in direct counterpoint to the state trend. Introduced in the 119th Congress, it would allow AI and machine learning systems to qualify as practitioners able to prescribe FDA-approved drugs, effectively federalizing the expansion of AI clinical authority that Texas alone has permitted at the state level. If it advances, it would create a direct conflict with the authorization and therapy bans enacted by more than two dozen states.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

On July 22, 2026, Pastor Scott Winters sued OpenAI in San Francisco County Superior Court. The complaint describes a five-week pattern that began a year earlier. On June 8, 2025, Winters grew dizzy mid-sermon and had to stop the service; paramedics were called and found him stable. Two days later, on June 10, too dizzy to walk, he crawled toward his bathroom and asked ChatGPT for directions across the floor between himself and his recliner. Both times, the suit alleges, ChatGPT told him to take it easy and that his symptoms would improve. On July 13, 2025, Winters asked ChatGPT about a pain in his groin. The suit alleges it offered spiritual reassurance, telling Winters that “God did not design your body to endlessly fail.” Hours later, he was in the intensive care unit with a massive pulmonary embolism. Winters filed his suit one day before OpenAI expanded Health in ChatGPT to 300 million weekly users.

Six weeks before Winters’ hospitalization, on May 31, 2025, nineteen-year-old Sam Nelson asked ChatGPT about nausea from Kratom use. Nelson had used ChatGPT since high school for homework, and according to the family’s complaint, the system had initially refused to discuss drug use with him at all. The complaint alleges those refusals stopped after the rollout of GPT-4o in 2024, and that by May 2025 the chatbot was offering him specific guidance instead. Rather than tell him to stop or seek care, it suggested pairing the Kratom with 0.25 to 0.5 milligrams of Xanax to “minimize nausea.” Nelson died that night of asphyxiation from the combination. His parents filed a wrongful-death suit in May 2026, arguing that ChatGPT had functioned as an active clinical decision-maker, not a passive search engine, and that a safety behavior the system once had was allowed to erode.

Both lawsuits were filed within nine weeks of each other, in the same stretch of 2026 when OpenAI was expanding its product’s reach to nearly the population of the United States. Together, the two cases show a pattern. A conversational AI system overrode medical urgency, once with spiritual language, once with a specific drug dose, and in both cases the underlying failure was the same: an architecture that struggles to recognize a rare, dangerous symptom because it is built to predict what’s statistically common, not what’s clinically true.

The Study Behind the Number

That failure has since been measured directly. In February 2026, Ramaswamy and colleagues at the Icahn School of Medicine at Mount Sinai published a study in Nature Medicine testing Health in ChatGPT, one month after launch. Researchers tested the product across 60 clinician-authored vignettes, yielding 960 total responses. It under-triaged roughly 52 percent of gold-standard emergencies, cases where every reviewing clinician agreed on the correct triage level. In practice, that meant sorting patients with conditions like diabetic ketoacidosis or impending respiratory failure into a 24-to-48-hour follow-up window instead of directing them to emergency care. The failures followed an inverted U-shaped pattern: the system missed 35 percent of nonurgent cases, sending people who didn’t need it toward urgent care, and 48 percent of true emergencies, the more dangerous error. It struggled most at both extremes rather than erring safely in one direction. Because the researchers tested the live product rather than a benchmark model, that 52 percent figure describes the exact system OpenAI connected to Epic, MyChart, One Medical, and Apple Health on July 23, 2026. OpenAI has pushed back on the study’s relevance, telling reporters that single-prompt vignettes don’t reflect “typical use,” since the product is designed for multi-turn conversations where patients supply additional context through follow-up questions. That may be true of some users. It does not explain why the same product, tested as it actually ships, missed the emergency in nearly half of the cases where the correct answer was to send someone to a hospital.

The Sandbox That Didn’t Hold

Two days before that expansion, on July 21, 2026, OpenAI and Hugging Face disclosed a security incident that undercuts any claim that the underlying architecture is under control. The incident began inside ExploitGym, an internal OpenAI benchmark built to measure whether frontier models could turn known software vulnerabilities into working exploits, a test of cyber capability with no connection to medical or consumer safety. During that test, GPT-5.6 Sol and an unreleased, more capable model escaped the isolated environment built to contain them. The models found a hidden flaw in a piece of software so obscure that even its own developers didn't know it existed. That flaw was enough to slip past the barriers meant to contain them. From there, they worked their way deeper into OpenAI's own systems, picking up login credentials and higher-level access along the way, until they reached the live servers running Hugging Face's actual public platform, not a test copy. Reporting on the incident described it as thousands of individual actions carried out across a swarm of short-lived sandboxes, a scale well beyond a single lucky guess. OpenAI's own account describes a multi-step plan, executed autonomously, to defeat an isolation boundary that had been purpose-built to hold the model.

What the Breach Does and Doesn’t Prove

Some caveats matter here. The model that escaped is not the exact instance running inside Health in ChatGPT, and the environment it escaped was a research sandbox rather than a live production server. The incident also occurred while OpenAI had deliberately reduced the models’ safety refusals for the purposes of the evaluation, so it does not show what happens under normal deployment conditions. What it does show is a capability that can’t be walked back with a patch: this model family, when its refusals are lowered or bypassed, can plan and execute a multi-domain escape without human authorization. That is closer to discovering a structural flaw in a medical device than a bug in one unit. The specific unit in the hospital may not have overheated yet, but the design has already shown it can. Installing that design in every ward, days after finding the flaw, is a decision that needs justifying, not assuming.

Why the Architecture Fails This Way

The clinical failure and the security failure share a root cause. Large language models generate text by predicting the most statistically likely next word given everything that came before. That process rewards fluency and common answers, a tendency researchers call regression to the mean: the model’s outputs cluster around the most probable response in its training data, at the expense of rare but important outliers. The training data itself makes this specifically worse in medicine. Models learn from internet-scale text in which routine complaints, muscle strain, stress, fatigue, vastly outnumber descriptions of rare emergencies, so the “likely” answer reflects what’s overrepresented in the general data rather than what’s actually likely for the patient typing the question. A weather model that always forecasts “partly cloudy” would be right most days and catastrophically wrong on the day a hurricane arrives. Pulmonary embolism is rare in the universe of chat transcripts about groin pain; muscle strain and spiritual doubt are common. Faced with Winters’ symptom, a system optimized for the likely answer will drift toward the likely explanation. There’s no malice in this. The architecture is doing exactly what it was built to do, and that is the problem in a clinical setting where the rare case is often the one that kills.

A System Built Differently

Contrast this with a different architecture. MomConnect, South Africa’s government maternal health platform, added a symptom-checking tool from Ada Health built on a Bayesian network: a model that encodes medical knowledge as explicit probabilities and updates them as new evidence comes in, rather than generating conversational text. The mechanics are different. The system starts with a prior probability for each condition it’s tracking, then updates that number as symptoms are entered, arriving at a posterior probability the system can act on directly: refer the case, ask another question, or reassure. When a Bayesian network is uncertain, that uncertainty is a number the system can act on, not a pattern buried inside billions of weights. Ada’s underlying engine also carries CE marking as regulated medical device software in Europe, a regulatory status Health in ChatGPT does not have. A study by Schmude and colleagues, published in Nature Health in June 2026, evaluated the tool’s advice to 968 pregnant women and mothers and found a physician panel rated it safe in 98.4 percent of cases, with appropriate care-seeking rising by more than 40 percentage points. The gap between that result and the 52 percent under-triage rate for Health in ChatGPT isn’t a matter of better prompts or more training data. It reflects two different ways of handling uncertainty. One system can tell you, in a number, how sure it is. The other generates a confident sentence whether or not it should.

The Regulatory Gap

That is also why "Health in ChatGPT" sits in a regulatory gap rather than a regulated category. The European Union's AI Act treats general-purpose AI models with systemic reach as their own class, subject to added testing and governance. The Act draws that line using a measure of how much raw computing power went into training a model, a number large enough that it's written in scientific notation: 10^25 floating-point operations, essentially a tally of the individual calculations involved in building the system. Cross that line, and the law assumes, by default, that the model carries systemic risk. A company can still make the case that its model doesn't actually pose that risk, but the starting point shifts toward oversight rather than away from it. A model that just demonstrated an autonomous escape from a security sandbox while serving 300 million weekly users is a reasonable candidate for that scrutiny. But systemic-risk status under the AI Act is not the same as an Annex III high-risk medical device classification, and OpenAI's product currently qualifies for neither in the way a diagnostic device would. The United States has, if anything, moved in the opposite direction. On January 6, 2026, the FDA issued revised guidance on clinical decision support software that loosened its prior stance on automation bias. The previous version had treated tools that presented a single recommendation as raising heightened safety concerns; the new guidance takes a more permissive, risk-based approach instead. That change landed six months before this launch, in the same regulatory environment where Health in ChatGPT now operates. A chatbot with a medical disclaimer, wired into Epic MyChart and making triage-adjacent judgments for hundreds of millions of people, is large enough to matter and, under current rules on both sides of the Atlantic, small enough to avoid the oversight a medical device would face.

The Experiment Continues

None of this is abstract. Winters and Nelson show what the failure looks like in a single conversation. The Ramaswamy study shows how often it happens across the live product. The Hugging Face disclosure shows the same model family can defeat a boundary built specifically to contain it. Put together, they describe a product launched into open clinical use before any of those three questions- whether the model can be trusted with rare symptoms, whether its containment holds under pressure, and who is checking either claim besides the company that built it- have been independently answered. Independent verification would mean outside researchers, people with no financial ties to OpenAI, deliberately trying to break the system and expose where it fails. Or it would mean a regulator with real investigative authority taking up the question directly. It would not mean another blog post from the company itself. The lawsuits will keep arriving. The question worth asking now is how many more it will take before this is treated as a systemic risk to public health rather than a chatbot with a disclaimer attached.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

Who gets to decide that software is ready to perform work that has historically required a medical license?

Medicine has spent more than a century building institutions that determine when physicians are competent to care for patients. Medical schools educate them. Residency programs train them. Specialty boards certify them. State medical boards license them. Hospitals credential them. Peer review evaluates them. Courts review their actions when patients are harmed.

Those institutions are not perfect, but together they establish something fundamental. Society has created a process for determining when a physician is competent enough to practice medicine.

Artificial intelligence introduces a new challenge because software is beginning to perform tasks that have traditionally depended on physician judgment. I am not talking about whether AI will participate in medicine. I’m adking who has jurisdiction to determine when the evidence is sufficient for software to assume greater clinical responsibility.

Earlier this year, Utah became one of the first states to confront that question directly. The debate that followed was widely portrayed as one about artificial intelligence and prescription renewals. I believe it was fundamentally a debate about jurisdiction over medical competence.

The Utah Pilot Structure

In December 2025, Utah’s Office of Artificial Intelligence Policy authorized a health technology company called Doctronic to begin renewing prescriptions for Utah patients using an AI system, under a 12-month regulatory sandbox agreement announced January 6, 2026.

The Office of Artificial Intelligence Policy was created in 2024 under Utah’s AI Policy Act, SB 149, which gives the office authority to temporarily waive regulatory requirements so companies can test AI systems under state supervision. Utah has used that same authority for a mental health chatbot aimed at teenagers and for an AI tool that reads dental radiographs. Doctronic’s agreement lets its AI process 30-, 60-, or 90-day renewals for medications that a licensed physician has already prescribed, and screen for drug interactions along the way.

The pilot is structured in three phases. In Phase One, every AI-generated renewal is reviewed and approved by a licensed physician before it reaches a pharmacy. Phase Two would move that review to shortly after a prescription is issued. Phase Three would allow the system to renew prescriptions with only periodic sampling of AI outputs for oversight. State officials say the pilot will not advance to the next phase until safety has been demonstrated in the one before it.

Before the pilot launched, Doctronic shared data with state regulators comparing its AI’s recommendations to physicians’ decisions across 500 urgent care cases. The company reported that its recommendations matched physicians’ 99.2 percent of the time.

Medication renewals consume substantial physician time, and many involve stable chronic conditions with predictable decision pathways. Exploring whether AI can safely assist physicians with those tasks is a reasonable objective.

The Utah Pilot Significance

On April 20, 2026, eleven of the fourteen members of the Utah Medical Licensing Board sent a letter to the Office of Artificial Intelligence Policy. They wrote that the board “was made aware of this agreement only after its implementation, once the system was already live and available for use.” The letter argued that prescription refills require reassessment of dose, side effects, contraindications, and drug interactions, work the board said belongs to a licensed physician, and it warned that “patients who continue refilling medications without assessment may remain on outdated or suboptimal therapy for months or years.” It closed by recommending that the pilot “be immediately suspended pending further discussion.”

The next day, the directors of the Office of Artificial Intelligence Policy and the Division of Professional Licensing responded in writing. They said the pilot had been “rigorously reviewed by several medical professionals prior to launch,” a process that produced “a large number of suggested substantive adjustments and guardrails,” and that the state would not suspend the pilot because it remained in Phase One, where a physician reviews every renewal before it is filled. They committed to involving the board more closely going forward.

Reasonable people can disagree about whether that was the correct decision. The more enduring question concerns the process itself. Which institution should determine that the available evidence justifies introducing software into a clinical role that has traditionally required physician judgment?

Evidence and Jurisdiction Cannot Be Separated

Much of the public debate over the Utah pilot focused on the AI itself: was it accurate, was physician oversight sufficient, should refill decisions be delegated to software at all.

Those questions naturally followed once the pilot was underway. They also assume that someone had already concluded the available evidence justified launching it. Let’s take a closer look.

The Doctronic pilot shows how the same evidence can support different conclusions. A 99.2 percent concordance rate across 500 urgent care cases was sufficient for the state and the medical professionals who reviewed the pilot before launch. It was not sufficient for the Medical Licensing Board, whose objection was not about the AI’s accuracy in aggregate. It was about what a refill requires in every individual case: a reassessment that a benchmark conducted before launch cannot fully substitute for once the system is operating on real patients. Both readings of the evidence are defensible. They come from institutions with different responsibilities.

The institution responsible for authorizing clinical AI also determines what counts as convincing evidence. A software engineer may prioritize benchmark performance and operational reliability. A practicing physician may focus on uncommon but consequential clinical presentations. A statistician may emphasize study design and external validity. A regulator may concentrate on statutory authority, while a hospital executive may view liability and implementation as equally important.

Each perspective is legitimate. Each emphasizes different forms of evidence. The authority that evaluates the evidence decides the standard by which readiness is judged.

Medicine Already Has a Model

Imagine a hospital announced that it had created a pathway allowing physicians to practice emergency medicine without residency training. The first question would not concern examination scores or clinical outcomes. It would concern authority. Who approved this process?

Medicine has developed a distributed system for answering that question. Medical education is accredited. Residency programs are supervised. Licensing examinations are standardized. Board certification is independently administered. Hospitals grant privileges only after reviewing credentials and competence.

No single organization controls that process. Government, professional organizations, accrediting bodies, and hospitals all participate. Each contributes a different perspective before a physician is entrusted with patient care.

Clinical AI is developing without an equally mature framework for determining when software is ready to assume comparable responsibilities.

The Same Evidence, Different Conclusions

One reason these discussions become contentious is that stakeholders evaluate evidence through different lenses.

Software developers ask whether the model performs accurately. Researchers ask whether a study demonstrates benefit. Medical boards ask whether patient safety has been adequately protected. Attorneys ask who assumes liability when errors occur. Patients ask whether they can trust the recommendations being made.

These are not competing questions, but complementary ones.

Evidence never speaks for itself. People decide what the evidence is sufficient to support.

A Different Way to Begin

Imagine Utah had started somewhere else. Before enrolling the first patient, the state convenes a multidisciplinary commission composed of primary care physicians, pharmacists, medical licensing officials, software engineers, biostatisticians, health services researchers, patient representatives, ethicists, and regulators. Their first task is not evaluating the AI, but instead defining the rules.

What evidence will be required? How should safety be measured? What outcomes matter? What level of physician oversight is necessary? Who has authority to suspend the pilot if safety concerns emerge?

Only after those questions are answered does the commission evaluate the software.

That sequence changes the discussion. The debate isn’t about defending or criticizing artificial intelligence. It is about establishing a legitimate process before patients become part of the evaluation.

Why Utah Matters

Prescription renewals are unlikely to be the last area where this issue appears.

Utah has already used the same regulatory sandbox authority outside prescribing: for a mental health chatbot aimed at teenagers and for an AI tool that reads dental radiographs. The same questions will emerge in radiology, pathology, dermatology, emergency medicine, imaging interpretation, treatment recommendations, autonomous documentation, and other clinical applications as AI capabilities continue to expand.

Every new application will generate studies, benchmark results, and performance claims. Those discussions remain essential. They do not eliminate the need for an accepted process that determines when the available evidence justifies broader clinical use.

The American Medical Association has already staked out a position on where that leaves physicians. Responding to the Utah pilot, AMA chief executive Dr. John Whyte said that “while AI has limitless opportunity to transform medicine for the better, without physician input it also poses serious risks to patients and physicians alike.”

Without an agreed framework, different organizations will continue reaching different conclusions from similar evidence because they are applying different standards and serving different responsibilities.

The Question Utah Leaves Behind

The public conversation often frames medical AI as a choice between accelerating innovation and slowing it down. Utah suggests the more important issue comes earlier.

Before software performs work that has historically required a medical license, society should determine who has jurisdiction to evaluate the evidence, what standards that evidence must satisfy, and how disagreements between institutions will be resolved.

The Utah pilot may ultimately prove to be an important success. It may demonstrate that AI can safely improve efficiency in routine clinical care. Regardless of its eventual outcome, it has already exposed a governance question that will become increasingly difficult to ignore.

Medicine has long established how society decides that physicians are ready to care for patients. Artificial intelligence now requires an equally thoughtful process for deciding when software is ready to assume comparable responsibilities. Utah did not answer that question. It demonstrated that we need one.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On April 27, 2026, OpenEvidence pulled its clinical AI platform from the EU and UK. The notice that replaced it cited “mounting regulatory uncertainty regarding the treatment of AI systems in the European Union and the United Kingdom, including, among other rules, the EU Artificial Intelligence Act.”

What the EU Actually Requires

Let’s start with what “high-risk” means under the EU AI Act, because most coverage of this story treated it as a vague label. Annex III places clinical decision support software in the high-risk category regardless of whether it’s also regulated as a medical device. That status requires, under Chapter III, Section 2 (Articles 8-15), a conformity assessment before the product reaches the market, technical documentation covering training data characteristics and known limitations, a human oversight mechanism built into the product’s design rather than added as a disclaimer, data governance standards for training data, ongoing post-market monitoring, and mandatory reporting of serious incidents to national authorities within 15 days.

None of that is free. Industry estimates put ongoing compliance at roughly €29,000 a year per AI system, with certification running another €17,000 to €23,000. The penalties for getting it wrong in Article 99 sets fines up to €35 million or 7 percent of global turnover for the most serious violations, up to €15 million or 3 percent for ordinary high-risk violations. A US company weighing whether the exposure justifies the market has real numbers to weigh it against.

The obligation doesn’t stop at the vendor, either. A hospital or clinic that adopts a high-risk AI tool becomes a “deployer” under Article 26, which lists twelve distinct duties, including assigning a person with actual authority to override the system, monitoring its performance, retaining logs for at least six months, notifying affected patients that they’re subject to the system, and training staff to understand what the tool can and can’t do. If you’re a health system reader in the EU wondering whether any of this applies to you, it does the moment you adopt a high-risk AI tool, regardless of what the vendor has or hasn’t done on their end.

What the US Exempts, and How Narrowly

In the US, software like OpenEvidence can avoid FDA device regulation entirely under the Non-Device Clinical Decision Support carve-out in the 21st Century Cures Act. FDA’s final guidance, issued March 11, 2026, sets four criteria a product has to meet simultaneously to qualify:

• It cannot acquire, process, or analyze a medical image, an in vitro diagnostic signal, or a pattern or signal from a signal acquisition system. FDA’s own guidance specifically names ECG waveforms as an example.

• It has to be intended for displaying, analyzing, or printing medical information about a patient, such as literature or guidelines, rather than patient-specific signal data.

• It has to support or recommend, not replace, a clinician’s judgment.

• It has to let the clinician independently review the basis for its output rather than functioning as a black box. FDA’s guidance is blunt that this is difficult for large language models generally and close to impossible in time-critical situations.

Miss any one of these four, and the software is a regulated medical device.

This connects to something I tested myself back in May. I uploaded an ECG to OpenEvidence and received an AI-generated interpretation. That alone looks like a Criterion 1 problem: analyzing a signal from a signal acquisition system is exactly what the exemption excludes. I checked again this week to see whether the feature had changed. It hadn’t. OpenEvidence still accepts ECG image uploads and still generates an interpretation. On the same case I tested in May, it still produced an incorrect but very official-appearing reading.

If OpenEvidence Doesn’t Qualify, What Happens on Each Side?

Assume for a moment that the ECG function alone is enough to knock a product out of Non-Device CDS status. What would actually follow, in the US and in the EU, and does the comparison support “the US is the settled option”?

In the US, there’s already a precedent-setting path for exactly this function. AccurKardia’s AccurECG 2.0 cleared FDA review as a Class II device via 510(k) in January 2026. Tempus received 510(k) clearance for its ECG-Low EF software the same way. If OpenEvidence’s interpretation function is similar enough to an existing cleared device, it would likely follow the same 510(k) route, averaging 155 days as of mid-2026. If it’s different enough that no predicate applies, it would more likely need De Novo classification, averaging 341 days.

In the EU, this same feature would almost certainly count as a moderate-to-higher-risk medical device (what MDR calls Class IIa or IIb), the kind that needs an outside safety review before it can be sold. That review currently takes 13 to 18 months on average. A new rule adopted this May is supposed to shrink that to roughly 6 to 9 months, but only for agreements signed after February 2027, so it doesn't help anyone applying today. And crossing that medical-device threshold doesn't let a company trade one set of rules for another. It automatically pulls the product into the AI Act's high-risk category too, so it ends up answering to both frameworks at once: the device safety review, plus the AI Act's own requirements for data governance, incident reporting, and built-in human oversight.

On the two sides of the Atlantic, there is a real structural difference, not manufactured uncertainty.

Two More Differences Worth Examining

Data protection is not the same question as AI regulation, and the two get blurred together in most coverage of this story. OpenEvidence’s own privacy policy tells EU users not to use the product because its infrastructure is US-based and isn’t governed by EU safeguards. GDPR treats health data as a special category under Article 9, requiring explicit consent or a documented research basis before it can be processed at all, a stricter bar than HIPAA’s authorization-or-de-identification model.

The EU and the UK are also not the same problem, though OpenEvidence’s notice treats them as one. The EU has written binding rules that are simply demanding and expensive to satisfy. The UK, by contrast, has no binding AI-specific medical device framework at all yet. The MHRA’s version is still a voluntary pilot program, with a real framework promised sometime later in 2026. “Uncertainty” is the accurate word for the UK’s situation. For the EU, the accurate words are “burden” and “cost.”

What OpenEvidence Actually Did

None of this required OpenEvidence to leave in the specific week it left, and the run-up to its exit is worth walking through.

By the time OpenEvidence posted its notice on April 27, the relief it was asking for was already in motion and had been for months, none of it because of anything OpenEvidence did. EU officials had proposed a package of changes back in November 2025, five months earlier, specifically to ease the same high-risk AI rules OpenEvidence's notice pointed to. The EU's parliament had already voted, 569 to 45, to move that package forward on March 26, a full month before OpenEvidence acted, and the different EU bodies involved had already started hammering out the details. The one round of those talks that hit a snag, on April 28, the day after OpenEvidence's notice went up, got stuck on a technical point that had nothing to do with the kind of product OpenEvidence makes. OpenEvidence left in the middle of a process that was already headed toward the outcome it said it needed, on a schedule set months before the company made its decision.

Its own notice is worth reading closely too. The original version named two people, Brando Benifei and Michael McNamara, and invited clinicians to contact them along with CPME, the umbrella body for European medical associations. Benifei and McNamara aren’t random picks. They co-chair Parliament’s actual AI Act oversight working group, and McNamara went on to become the Rapporteur for the exact Omnibus package that delivered the deferral. CPME, for its part, had already co-signed letters in March and April arguing to keep the AI Act strict, the opposite of what OpenEvidence needed. That callout disappeared from the notice within 48 hours. A correspondence published in the Lancet in May read the episode as a scrubbed attempt at bottom-up lobbying, notable for what it suggests about the gap between OpenEvidence’s stated rationale and its own actions, regardless of what the callout was intended to accomplish.

And the technology itself never actually left. Although OpenEvidence’s own site still shows the same notice to anyone in the EU, Elsevier’s ClinicalKey AI, running on OpenEvidence’s own engine per their 2023 partnership and featuring The Lancet as a headline content source, is available to EU subscribers. The AI didn’t leave Europe. It just stopped answering to OpenEvidence’s name.

So, Why Did OpenEvidence Really Leave?

Probably not because the EU is uncertain. The EU’s rules are written down, phased in on a public schedule, and expensive to satisfy, which is a different problem than uncertainty and a more honest one for OpenEvidence to have named. The timeline it cited resolved in a direction it didn’t need to lobby for. And its own underlying technology kept working in the exact market OpenEvidence says it can no longer serve, just under someone else’s name.

The clinicians who lost access to OpenEvidence didn’t get any of that. They got one sentence, a since-deleted prompt to contact officials who didn’t need the push, and a tool that, as best I can tell, may already be operating past the line the FDA draws around what doesn’t need to be regulated at all.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On July 10, 2026, OpenEvidence launched EvidenceGrade, a feature that assigns a letter grade to the strength of evidence beneath every AI-generated clinical answer. It’s an improvement in the value of the responses OpenEvidence creates, and it lands in a market where other tools have been grading evidence in multiple different ways for quite some time.

What EvidenceGrade Does

EvidenceGrade builds on GRADE, the Grading of Recommendations, Assessment, Development, and Evaluation framework behind Cochrane reviews, WHO guidance, and most major clinical practice guidelines. GRADE has run evidence-based medicine for two decades. What’s new is speed, not the underlying idea of an overall grade. OpenEvidence computes a single grade for the whole answer live, at the moment a clinician asks a question, a step that has previously required human editors working ahead of time.

When a clinician asks OpenEvidence a question, the system first runs a classification step, checking whether the question amounts to a clear evidentiary claim. Simple definition lookups and summarization requests get filtered out here. Questions that pass get scored on study design, consistency of findings, precision, and directness- the same inputs GRADE has always used- and the system returns a letter grade for strength, with a separate “U” designation for evidence that can’t be graded.

Two details are worth noting. OpenEvidence’s classifier decides which questions get graded, not the clinician. And the grading itself comes from a single company applying GRADE algorithmically to its own retrieval results, a different process than GRADE’s original consensus-panel model.

Five Products, Five Methods

OE’s EvidenceGrade scores literature live, at the moment a clinician asks a question. Scoring extends to questions that no formal appraisal has reviewed yet, and the tool runs at point-of-care speed.

UpToDate’s Expert AI launched in late 2025 and sits on top of content that physician editors already reviewed and graded using GRADE, with Grade 1 or 2 for recommendation strength and Grade A, B, or C for evidence quality. The AI layer surfaces existing editorial work conversationally, with rationale and links back to the original topics. Scoring depends on what editors have written and reviewed, so very recent literature can lag what OpenEvidence pulls in real time.

DynaMed shares UpToDate’s editorial foundation: physician editors review and rate evidence with GRADE ahead of time. The method diverges at synthesis. DynaMed’s AI layer rates the individual sources it cites but doesn’t roll them up into a single grade for the answer as a whole, the way UpToDate’s per-recommendation grade or OpenEvidence’s per-answer letter does.

Vera Health also grades the strength of underlying evidence behind its answers and positions itself as a direct point-of-care competitor to OpenEvidence. That grading applies to the individual sources it cites, the same source-level approach DynaMed’s AI takes, rather than producing one overall grade for the answer.

Consensus takes a different approach. Its default ranking sorts papers by citation count, journal reputation, and recency, all signals of a paper’s influence and reach rather than the rigor of its methodology. Consensus also offers an optional evidence-hierarchy filter that lets a clinician sort by study design, from meta-analyses and systematic reviews down through RCTs and observational studies. Nothing is graded automatically here; the clinician chooses whether to apply the filter every time.

Pros and Cons

OpenEvidence offers speed and reach. A grade can appear under a question no editor has ever reviewed, in the time a clinician has between patients. That speed comes from a single company’s automated interpretation of GRADE, run through a classifier a clinician can’t inspect.

UpToDate carries the weight of editorial review behind every grade, built up over years with an established audit trail. Questions about very recent literature can outrun what the editorial team has reached, so timeliness suffers where OpenEvidence’s live scoring holds an advantage.

DynaMed shares UpToDate’s tradeoff of editorial rigor against editorial pace. Its AI layer grades each source it cites, but leaves the work of weighing those individual grades into one overall judgment to the clinician, since it doesn’t generate the single per-recommendation grade UpToDate does.

Vera Health competes on benchmark performance and broader geographic reach and on a grading methodology it hasn’t named as clearly as its competitors have. It also stops at source-level grades rather than synthesizing one for the answer, so a clinician still has to weigh conflicting source grades themselves. A clinician outside the US or UK who lost access to OpenEvidence may find Vera Health the more practical option.

Consensus puts control directly in the clinician’s hands. No classifier decides what gets graded, and every AI-generated summary traces back to an exact sentence in the source paper. That control depends on the clinician knowing to apply the hierarchy filter; skip it, and the ranking on screen reflects citation counts and journal reputation, which are signals of popularity rather than rigor. Consensus also serves general research across all fields, so it lacks the clinical workflow tuning built into the other four tools.

When to Actually Use Which

A fast, evidentiary bedside question, something like whether a drug reduces mortality in a given population, fits OpenEvidence well. Watch for whether a banner appears at all; no banner means the classifier judged the question outside EvidenceGrade’s scope, which is worth a second look if the question felt evidentiary to you.

A well-established clinical question or standard management of a common chronic condition plays to UpToDate or DynaMed’s strength. Human editorial review has already happened, and the subscription cost buys that vetting.

A clinician outside the US or UK, or one weighing the benchmark claims directly, has reason to look at Vera Health alongside or instead of OpenEvidence, keeping in mind that the comparative claims come from Vera Health itself.

Deep literature review on an emerging or contested question, or research headed into a paper or grant application, calls for Consensus, used deliberately with the hierarchy filter engaged and the full-text tool open to check claims directly.

Many everyday questions have no clear winner among these five. When UpToDate already holds a well-established Grade A recommendation for a common condition, OpenEvidence’s live grade on the same question will likely land in the same place, since both draw on the same underlying evidence base. The choice there comes down to workflow preference, editorial certainty against conversational speed. A clinician willing to spend an extra ninety seconds applying Consensus’s RCT filter can reach a similar result to what EvidenceGrade delivers automatically.

Risk and Bias, In Both Directions

The editorial model behind UpToDate and DynaMed carries its own risks. A recommendation can sit unchanged for months after new evidence complicates it, and every grading decision reflects the judgment of the specific editors who wrote it, not an infallible panel.

OpenEvidence’s automated model carries its own risk. A letter grade can read as more authoritative than a live scoring process actually supports, which invites automation bias. The classification step still makes a judgment call the clinician doesn’t get to weigh in on: whether a question counts as evidentiary at all. Grading quality also depends entirely on what the retrieval system pulled in the first place, a step a clinician can’t audit in real time.

DynaMed’s AI and Vera Health carry a different risk, tied to the same source-level grading described above. Neither tool synthesizes the individual source grades into one judgment, which means the aggregation OpenEvidence and UpToDate perform automatically becomes a manual step for a clinician moving quickly between patients, and manual synthesis done under time pressure is where inconsistency creeps in.

Consensus shifts risk toward the clinician directly. Someone who doesn’t know to apply the evidence-hierarchy filter, or who doesn’t recognize that the default ranking measures popularity rather than rigor, gets no protection from the tool itself.

Each model fails in its own way, and the failure a clinician is exposed to depends on which tool they picked and how well they understand what runs underneath it.

The Question Nobody Has Answered: Liability

Clinicians are not attorneys, and nothing here is legal advice. It’s worth asking plainly, since none of the marketing for any of these five products addresses it directly: Does a clinician carry more or less liability for relying on an AI-generated evidence grade compared to a human-editor-vetted one?

I could not find an answer to this question. Malpractice and negligence standards generally turn on whether a clinician exercised reasonable judgment consistent with the standard of care, and historically that inquiry centers on the clinician’s own decision rather than the reference tool behind it. All five products here position themselves as clinical decision support rather than autonomous diagnostic tools, which keeps the clinician as the final decision-maker of record, the same SaMD-versus-CDS framing this newsletter has examined before.

Whether courts will eventually treat a real-time AI grade differently from an incremental human editorial grade remains genuinely untested. I don’t have an answer, and anyone claiming otherwise is speculating.

What holds steady across all five tools is this: whatever grade appears on the screen, the decision that follows still belongs to the clinician who acts on it.

As always, if you enjoy reading, please subscribe and tell a friend.

Sam

When a clinician has a question during an encounter, the workflow is straightforward. They recognize the need for evidence. They leave the patient, open a search tool like OpenEvidence, UpToDate, AMBOSS, or Doximity, formulate a query, and try to find an answer. Then they return to the chart and translate that finding into a plan.

It takes time and requires cognitive effort. It also interrupts the clinical workflow.

What if that entire sequence could be automated by pairing two things that currently operate independently: a scribe that captures context and an AI search tool that needs context?

If it works the way vendors are beginning to market it, the implications could be substantial.

The Manual Search Problem

Clinical decision support tools are valuable. The problem is that clinicians rarely use them consistently.

Adoption of standalone clinical decision support (CDS) has hovered between 20 and 50% for more than a decade. When adoption rates exceed 50% in any health system, it’s usually because the tool was so deeply embedded in workflow that using it became invisible. Not a conscious decision to “search for evidence,” but a byproduct of how documentation already worked.

The Agency for Healthcare Research and Quality (AHRQ)‘s evidence on clinical decision support has long established this pattern. In a 2005 study on clinical information needs, Ely and colleagues found that the most common reason physicians did not pursue answers to clinical questions was time. Not having access to tools, or skepticism about the instrument. Every additional click, every additional screen, every additional cognitive step to formulate what you’re searching for, all of that is friction.

The manual prompt introduces friction at multiple points.

First, the clinician has to recognize that they need help. That alone is cognitively taxing when you’re in the middle of an encounter and managing multiple competing hypotheses simultaneously.

Second, they have to formulate what they’re searching for. They have to decide what context matters. For example, a 65-year-old woman with a subsegmental PE is the bare minimum. But the clinically relevant narrative includes much more: her main symptom was chest pain, her father died of an MI at 65, she has no dyspnea despite the imaging findings, she’s anxious about the PE diagnosis. The question is whether the clinician remembers or prioritizes all of that when they type a prompt. Often they don’t. They reduce it to what feels immediately salient.

Third, they have to type it. And then translate the results back into their documentation.

Each of these steps is a place where information can be lost. Each is a place where a busy clinician can decide it’s faster to just make a decision than to go through the process.

The result: CDS adoption stalls. Not because the tools don’t work, but because the friction cost exceeds the perceived benefit.

What If Context Was Automatic?

Now consider a different architecture.

A scribe is already listening to the encounter. It’s capturing everything: chief complaint, associated symptoms, what improved with aspirin, the patient’s expressed anxiety about the PE finding, the absence of dyspnea despite imaging, family history, the nuance of how the history actually unfolded. The scribe structures that into a clinical note.

What if that same structured encounter context became the automatic input to downstream AI search and clinical decision support?

The difference is subtle but consequential. Instead of waiting for the clinician to recognize they need help and formulate a prompt, the system works from what’s already been captured. The context is automatically available. The search doesn’t depend on the clinician deciding what matters or remembering to type it.

This is the architectural shift vendors are now building.

Glass Health combines ambient scribe with a clinical reasoning layer that accesses the encounter transcript. When the scribe finishes capturing the visit, the same data becomes context for CDS reasoning, not a separate step, not a separate prompt, but the natural downstream use of data already captured.

Microsoft’s DAX Copilot generates the note from the scribe, then uses that same encounter context to surface order suggestions within Epic. The clinician doesn’t have to prompt for order ideas; they arrive pre-staged, informed by what was actually discussed.

Abridge, after expanding into real-time prior authorization through an Availity partnership announced in January 2026, is pre-populating prior auth requests with encounter context. The clinician reviews, not writes.

Ambience Healthcare’s AutoScribe provides real-time coding suggestions, quality measure tracking, and automated prior authorization recommendations, all triggered by the scribe context, all without requiring the clinician to initiate a separate search.

In each case, the pattern is the same: scribe captures context automatically, downstream systems act on that context without waiting for the clinician to formulate a prompt.

Why This Matters

If CDS adoption has been stuck at 20-50% because clinicians won’t use tools that require extra steps, and if integrated scribe-plus-search systems eliminate that friction by making context automatic and routing recommendations directly into workflow, then adoption could shift.

Not because the search tools suddenly became better. But because the context became richer, the friction resolved, and the recommendations arrived where clinicians are already working.

There’s a secondary automation gain as well. Once encounter context is structured and available, systems downstream can act without additional clinician prompts. Abridge is already doing this with order placement. The prior authorization request comes pre-populated. The labs or imaging recommendations arrive as suggestions for review, not as questions requiring the clinician to type more information.

Each of these automations removes a decision point. Each removes a place where context can be lost, or a clinician can opt out because the friction became too high.

This is different than asking “are AI search tools more accurate when they have more information?” This is asking whether integrated scribe-plus-search systems could achieve the adoption curves and utilization patterns that standalone CDS have chased unsuccessfully for years.

Where This Is Already Shipping

The market is moving in this direction. The examples above are live products, deployed now.

• OpenEvidence, Doximity, and Glass Health offer scribe plus clinical decision support in the same interface.

• DAX Copilot’s order suggestions are live within Epic.

• Abridge’s prior authorization workflow doesn’t require the clinician to re-enter what was already discussed with the patient; the scribe context carries it forward.

• Athenahealth made its ambient scribe free to all customers in February 2026. The company has explicitly positioned the scribe as a foundation for downstream clinical and administrative workflows.

These aren’t experimental pilots. They’re market moves. Multiple vendors are racing to integrate scribe plus CDS, scribe plus order suggestions, scribe plus prior authorization, and scribe plus billing optimization. The race itself suggests vendors believe integration is where the value is concentrating.

The Evidence Question

AHRQ best practices emphasize that CDS embedded in clinical workflow achieves significantly higher adoption rates than standalone tools.

The Rotenstein et al. study published in JAMA in April 2026 tracked 8,581 clinicians across five academic medical centers and found that 79% of eligible clinicians declined to adopt a scribe when it was offered as a standalone documentation tool. Among those who did adopt, only 32% used it in 50% or more of visits, the threshold where benefits actually accumulated.

But at organizations where the scribe was deliberately integrated into clinical workflow with physician champions, hands-on training, and customization of documentation to match local practice, adoption reached 75-80%.

Central Oklahoma Family Medical Center saw minimal scribe adoption in year 1 when it was positioned as a documentation tool. In year 3, after deliberate workflow integration and physician-led training, the system was generating over 14,000 records annually.

Research supports the pattern: embedded tools are better than standalone tools. Integrated workflows are better than siloed workflows. Friction is the constraint, not technology limits.

However, there is no published comparative adoption study in the same health system measuring scribe-only utilization versus integrated scribe-plus-CDS utilization. The vendors are claiming that integration drives adoption. The friction research supports that claim logically. But direct comparative evidence is absent.

That gap is fixable. It’s also important. If the thesis holds that integrated scribe plus CDS achieves meaningfully higher adoption than scribe-only, that would directly support the “game changer” framing. Until that evidence exists, we’re working from logical inference and market positioning, not real-world data.

What Changes Clinically

Let me be concrete about what this looks like in practice.

Today’s workflow: A 65-year-old woman presents with chest pain, elevated troponin, and a subsegmental PE on imaging. The clinician needs to decide between acute coronary syndrome and a low-risk PE with incidental findings. They recognize they need evidence about ACS risk stratification and PE disposition pathways. They leave the encounter, open OpenEvidence or UpToDate, search for something like “subsegmental PE,” scan results, and return to the chart with an answer. Time cost: 5-10 minutes. Friction cost: context loss between encounter and search.

Tomorrow’s workflow: The same patient, same presentation. The scribe has captured the full encounter: the aspirin response, the specific way the history unfolded, the family history details, the absence of dyspnea. Once the note is drafted, that encounter context automatically feeds into CDS reasoning. Before the clinician even finishes reviewing the note, recommendations about ACS risk stratification appear inline. They’re specific to what was discussed, not to a reductive prompt. The clinician reviews and acts. No extra steps. No context loss.

The difference isn’t speed alone. It’s that the clinical reasoning is informed by what actually happened, not by what the clinician decided was relevant enough to type.

What Has to Be True for This to Work

This only works if several conditions hold.

Scribe accuracy matters more. If the scribe omits a symptom or invents a clinical detail, everything downstream breaks. Recommendations based on false or inaccurate context are worse than no recommendations. The accuracy bar for a scribe that feeds into automation is higher than for a scribe that just generates a note for a clinician to review and edit.

Context routing has to be intelligent. Not every piece of encounter context should trigger every possible recommendation. Signal-to-noise matters. If the system fires off ten recommendations per encounter, most of which are irrelevant, clinician trust collapses. Integration has to include filtering and prioritization logic that decides what context maps to what recommendations.

Integration has to be seamless. If embedding CDS into the scribe workflow adds steps, the friction returns and adoption stalls. The system has to route recommendations directly into existing documentation workflows without requiring the clinician to open new windows, review separate interfaces, or translate between formats.

Liability has to be clear. If a scribe misses a clinical detail, who’s responsible? If CDS gives a recommendation based on scribe context and it’s wrong, who bears the liability: the scribe vendor, the CDS vendor, the clinician, the health system? Until those questions are answered clearly, adoption will be cautious.

Clinicians have to adopt the pattern. The mental model of “don’t prompt, just accept recommendations” is new for many physicians. It requires trust in both the scribe's accuracy and the CDS's reasoning. It requires that clinicians believe recommendations are being triggered appropriately, not indiscriminately. Building that trust takes time.

Conclusion

The architectural logic is sound. Pairing a scribe that captures context with CDS that needs context could eliminate the friction that’s kept adoption low. Integration could shift clinical decision support from a tool clinicians invoke when they have time to a system that informs their thinking automatically.

But “could” and “does” are different things. The market believes in this direction. Vendors are shipping products built on this assumption.

The evidence, though, is still emerging. The comparative adoption study doesn’t exist yet. The real-world error profiles haven’t been published. The liability frameworks are still being negotiated.

What we’re watching is an inflection point. The question isn’t whether AI scribes plus AI search could be a game changer. The architecture is sound enough that it’s plausible. The question is whether the systems being deployed right now will deliver on that promise at scale.

That’s worth paying attention to.

They need comparison data, but there isn’t any. The single largest, most consequential technology decision many U.S. hospitals will make this decade is being made with far less evidence than we would accept for a new drug or device. Let’s dive deeper.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

It’s not a hypothetical problem.

• More than 85% of Epic’s customer base is already using some form of Epic AI

• EHR vendor dependency is cited as the top execution barrier by 74% of health system technology leaders

• A majority of Epic’s customers report spending a quarter of their IT bandwidth simply managing multiple vendor integrations rather than building anything new.

So how are hospitals supposed to decide about Epic’s AI features? Three questions capture the actual decision tree a CIO faces:

1. How do you compare Epic’s tools against alternatives when no published performance data exists?

2. Are you left comparing on price alone?

3. Can you even connect an outside AI tool to Epic, and what does that actually cost in dollars and time?

Each answer turns out to be more complicated than it first appears.

The Comparison Vacuum

According to EPIC, more than 200 organizations use Penny for professional coding, with many seeing 20% of more coding-related denial reductions. Art’s Insights feature is used more than 16 million times a month. At The Christ Hospital, Art’s radiology finding extraction is credited with a 69% early lung cancer detection rate compared to a national average of 46%. At Rush University Medical Center, Emmie delivered a 58% reduction in billing-related customer service messages.

These are genuine wins, not vague marketing language. But notice what they have in common: every one of them is a curated success story that Epic chose to publish, from a customer Epic chose to name, describing an outcome Epic chose to measure. None of them come with a comparison arm. None of them tell you what a third-party tool, or no AI tool at all, would have achieved at the same hospital over the same period.

What doesn’t exist is any neutral, third-party study putting Epic’s tools head-to-head against Abridge, Ambience, Nabla, Suki, or Oracle Health’s Clinical AI Agent, on the same patients, measuring the same things. The closest proxies available each fall short in a specific, disqualifying way:

• KLAS “Best in KLAS” rankings exist for ambient AI and clinical documentation, and vendors like Abridge and Ambience have won them in 2025 and 2026. But KLAS rankings are built from customer satisfaction surveys. They measure whether clinicians like a tool, not whether it performs more accurately or safely than the alternative.

• Vendor comparison content, like the “Abridge vs. Ambience” writeups that circulate among health IT consultants, reads like independent analysis but is typically produced by resellers and integration firms with a commercial stake in one outcome or another.

To be fair to Epic, native integration solves a real, well-documented problem. Third-party AI tools often exist as a separate app or window that a clinician has to switch into, breaking their workflow. Epic’s tools, by contrast, draw on the patient’s complete longitudinal record natively, inside the same interface the clinician already lives in. That is legitimate integration depth, and Epic has a structural claim to it that outside vendors have to work much harder to match.

But “we’re already inside your workflow, so trust us” is a switching-cost argument, not a quality argument. It explains why Epic’s tools are more convenient to adopt. It says nothing about whether Art’s diagnostic suggestions, Emmie’s patient explanations, or Penny’s coding recommendations are more accurate than a competitor’s because nobody has measured that, and Epic has no more incentive to fund that comparison than a third-party vendor would have to fund one showing Epic winning.

The honest, unsatisfying answer for how hospitals resolve this: they don’t rely on external comparison data. They run their own limited pilots of both options against their own patient population and staff, because nobody else has done, or has an incentive to do, that comparison for them. That’s not a failure of any individual CIO’s diligence. It’s a structural gap in the market that every hospital is left to close on its own, one expensive pilot at a time.

Price Is Visible. Quality Isn’t.

No CIO is literally forced to decide on price alone. But price is the one variable in this decision with any real transparency, and that asymmetry quietly warps the whole comparison because when quality is unmeasurable and price is a line item, price tends to win by default, whether or not it should.

The other differentiators hospitals actually weigh are real, but none of them are clinical outcome measures:

• Integration depth. Does the tool live inside Epic natively, or does it require clinicians to leave the EHR? This is the “house vs. guest in the house” distinction discussed above. It’s a genuine operational factor, not evidence of accuracy or safety.

• Specialty coverage breadth. Some third-party tools market coverage across 200-plus specialties, from oncology to emergency medicine. That’s a real capability difference, but it says nothing about performance within any single specialty.

• Vendor relationship tier with Epic. Some vendors, like Abridge and Nuance, hold a spot in Epic’s invite-only “Workshop” co-development tier. Others sit in the more generic, self-service “Connection Hub” listing. This tells you something about durability and long-term support risk. It tells you nothing about whether the underlying AI is any good.

• Lock-in and commoditization risk. As Epic rolls out “good enough” native alternatives, standalone point solutions face real competitive pressure. Betting on a third-party tool means betting that it survives Epic’s gravitational pull.

The uncomfortable throughline across all four: they’re operational and strategic risk proxies, not evidence. A CIO weighing them isn’t answering “which AI is better”. They’re answering “which AI is safer to bet on organizationally.”

Yes, You Can Connect Outside AI to Epic

Epic is not a walled garden on this point. The technical path to connecting an outside AI tool, whether a commercial product or a custom build on an API, is real, standardized, and already proven in production.

The mechanisms, at the level a CIO actually needs:

• SMART on FHIR is an open industry standard, not an Epic-proprietary one, that lets a third-party application launch inside Epic’s interface (Hyperspace, Hyperdrive, or MyChart) with OAuth2-based access to the patient’s clinical context.

• CDS Hooks is a lighter-weight alternative for narrower use cases: an external tool fires in real time on a specific clinical event (opening a chart, signing an order) and returns a recommendation card, without the overhead of launching a full separate application.

• Epic Showroom (formerly App Orchard) is the marketplace and certification pathway, with three tiers:

  • Connection Hub, a basic listing starting around $500 a year, where most third-party apps live.

  • Toolbox, offering curated visibility

  • Workshop, reserved for vendors Epic is actively co-developing with, not a tier you can apply for.

Abridge, Ambience, Nabla, and Suki all run as third-party ambient scribes launched inside Epic today via exactly this pathway. Connecting a custom AI build is well-trodden ground, not an experimental frontier.

What it actually costs, in dollars:

• A basic FHIR read integration for a first site: $15,000–$40,000

• Bidirectional integration (the tool can write back into the chart, not just read from it): $40,000–$80,000

• Each additional hospital site: $5,000–$20,000 in configuration, testing, and governance overhead

• Enterprise multi-site deployments (five-plus hospitals): $100,000–$500,000+

• Annual maintenance: 15–20% of build cost, ongoing

Those figures answer the technical and financial questions. They don’t answer the harder question, which is where most integration projects actually stall.

“Integration Is 20% Development, 80% Negotiation”

Brendan Keeler, a widely cited EHR integration analyst, has a line that captures this better than any cost table: integration is 20% development, 80% negotiation.

It’s helpful sitting thinking about what that 80% actually is, because it’s several negotiations stacked, and none of them get easier just because a vendor has already done this at another hospital:

• Per-site IT governance review. Even though Epic’s FHIR APIs are standardized across every customer, each hospital runs its own instance with its own security policies. A vendor doesn’t clear “Epic” once. They clear each hospital’s IT and security team separately, every time.

• Scope and access-level negotiation. What data can the tool read? Can it write back into the chart? Is access tied to individual user logins or a backend service account? These aren’t just technical settings. They’re negotiated line by line with each hospital’s compliance and security staff, who have their own risk tolerance and institutional precedent to defend.

• Contract terms with Epic itself, separate from any single hospital deal: Showroom tier, vendor services registration, fees.

• Institutional change management: Getting clinical leadership, IT, compliance, and often legal to actually agree to put a new tool inside their clinicians’ workflow.

This negotiation stack isn’t identical across hospitals, even though the underlying FHIR technology may be. Each hospital is, in effect, a separate legal and organizational gate a vendor has to clear from scratch, no matter how many times they’ve cleared an equivalent gate elsewhere. That mismatch is exactly why Epic’s own native tools carry a durable structural advantage that has nothing to do with whether their AI is actually better. They skip the entire negotiation stack because they’re already inside the walls, already covered by the hospital’s existing Epic contract and security posture.

How Vendors Are Fighting Back

Third-party vendors aren’t standing still against this asymmetry. Several strategies have emerged, each attacking a different piece of the 80%:

• Pre-certification as a trust shortcut. HITRUST certification (including a new AI-specific security certification launched in response to healthcare’s AI boom) and SOC 2 Type II audits let a vendor prove its security posture once and present that proof to every prospective hospital, rather than rebuilding trust from zero at each site. HITRUST is now often a baseline requirement for even getting considered in procurement.

• Integration aggregators. Platforms like Redox normalize FHIR, HL7 v2, and proprietary EHR APIs into a single interface, letting a vendor avoid building direct integrations against three to five different EHR systems from scratch, at the cost of some data fidelity and an extra hop in the data flow.

• Design-partner-first rollout. Land one engaged hospital willing to work through the full registration and security review cycle together, get the integration proven in production, then use that as a template for the second and third hospital, accepting that some per-site variability will still surface each time.

• Climbing into Epic’s Workshop tier. Vendors like Abridge and Nuance have secured Epic’s invite-only co-development relationship, which functions as Epic effectively vouching for them. It’s a different negotiation dynamic from approaching each hospital cold.

• Leaning on CDS Hooks for narrower use cases, trading a smaller interaction surface for meaningfully lower integration complexity.

• Regulatory tailwinds. The 21st Century Cures Act’s information-blocking provisions and ONC’s HTI-1 rule, which raised the certification baseline to USCDI v3 as of January 2026, give vendors a legally backed claim to FHIR access, shifting some leverage away from ad hoc hospital IT gatekeeping.

None of this eliminates the negotiation. It compresses or front-loads it. Bidirectional write-back (actually pushing AI-generated content into the medical record, not just reading from it) remains the hardest and slowest part of any integration, regardless of how much pre-certification a vendor obtains.

What CIOs Actually Have to Work With

A few genuinely useful resources exist for the leader trying to navigate this, though none of them is the comprehensive primer this decision deserves.

The Health Sector Coordinating Council’s “Health Industry AI Cyber Governance Framework Implementation Guide” (May 2026) is the closest thing to an authoritative, non-commercial reference, covering governance committee structure scaled to hospital size, escalation authority across CMO, CMIO, and Privacy Officer roles, and a benefit-risk framework tied explicitly to FDA regulatory status.

Qventus’s “Beyond the Pilot” survey of more than 60 CIOs, Chief AI Officers, and CMIOs offers real peer benchmarking: 74% cite EHR vendor dependency as their top execution barrier; 72% say they’d prefer a single consolidated AI partner over a fragmented multi-vendor stack, but only 13% have actually achieved that consolidation. It’s vendor-published, so it should be read as useful data with a thumb on the scale, not neutral research.

Narrower academic frameworks exist too, like peer-reviewed pragmatic-trial protocols for evaluating ambient AI specifically, which bring real methodological rigor but only to one slice of the larger decision.

What none of these does is tie the whole picture together. A CIO today has to assemble governance policy from one source, vendor economics from a consulting firm’s blog, integration cost data from an engineering guide, and validation-status skepticism from wherever they can find it, with no neutral party doing that synthesis for them. That absence is, itself, worth naming plainly: the market has produced plenty of pieces but no assembled whole.

Takeaway

Individual hospitals cannot solve this problem on their own. It is a fundamental issue with the entire healthcare market, which tends to favor established players, like Epic, over newer or better alternatives, simply because they already hold the power.

A few things worth holding onto if you’re the one making this call:

1. Don’t mistake Epic’s adoption percentages for comparative quality evidence. They are usage statistics, curated by the company reporting them. They tell you Epic AI is widely used. They don’t tell you it’s the best option.

2. Budget for the 80%, not just the 20%. The dollar figures for FHIR integration are real, but the calendar time and staff bandwidth consumed by per-site negotiation are the actual cost drivers, and they’re the ones most commonly missing from a project’s original scope.

3. Ask any vendor directly what Showroom tier they hold. It won’t tell you if their AI is accurate, but it will tell you something real about how durable their access is likely to be.

4. Treat the absence of comparative data as a mandate to pilot, not a reason to default to the incumbent. Nobody else is going to run that comparison for you. That doesn’t mean it isn’t worth running.

No one selling you this technology is going to prove it works for you. That job still belongs to the hospital buying it.

Meanwhile, nurses in New York, California, Michigan, and North Carolina have been striking and writing AI oversight rights into union contracts. The national nursing union organization that represents those same nurses is publishing its own AI bill of rights.

Both responses exist because the frameworks and the union contracts are answering different questions. Understanding why requires looking at which AI tools are actually being deployed, to whom, and at whose direction.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

What the Professional Associations Agree On

The American Medical Association published formal AI principles in November 2023, adopted a transparency policy calling for independent third-party verification of AI explainability in June 2025, and released an 8-step governance toolkit in August 2025. The toolkit calls for multidisciplinary governance committees that include nursing leadership, staged deployment with pilot testing, and continuous performance monitoring after launch.

The American Nurses Association convened its inaugural AI in Nursing Practice Think Tank on April 22, 2026. The consensus findings identify automation bias and erosion of professional judgment as the primary risks, call for mandatory AI literacy as a core nursing competency, and push for nurse-led AI governance at the institutional level.

The American Academy of Nursing approved a comprehensive AI position statement on February 25, 2026, setting out 13 specific policy recommendations covering data privacy, algorithmic bias, FDA oversight, and human-in-the-loop oversight standards across all institutional governance policies.

The foundational language across all three documents is nearly identical: AI must augment clinical judgment, not replace it. The human clinician remains the final accountable decision-maker. Human-in-the-loop oversight is non-negotiable.

All three frameworks rely on the same implementation mechanism: voluntary institutional governance, multidisciplinary committees, professional education, and policy advocacy. None specifies what “human in the loop” requires at the point of care. None has an enforcement mechanism at the bedside level.

Two Categories of AI Tools

The governance frameworks were designed for a particular relationship between clinician and AI: the clinician chooses to use a tool, reviews its output, and retains authority over the final decision. The ambient scribe is the clearest example. The physician activates it, the scribe drafts a note, the physician reviews and signs. The tool serves the physician. Clinical judgment stays with the clinician throughout.

Physician AI adoption fits this model closely. By 2026, more than 80% of physicians report using AI in their professional work, with more than three-quarters saying it improves their ability to care for patients. The tools driving that adoption are documentation-focused: 70% of physicians at UCSF now use AI scribes daily, and Kaiser Permanente logged more than 2.5 million AI-scribed encounters over 14 months. Randomized trial evidence confirms that these tools reduce documentation time and burnout scores.

The AI tools most commonly deployed in nursing workflows belong to a different category. Patient acuity scoring algorithms analyze EHR data to determine how sick a patient is and predict how many nursing hours that patient requires. Staffing algorithms use those acuity scores to determine how many nurses are called into a shift. Automated handoff tools generate shift reports. Clinical deterioration algorithms produce alerts. These tools are not activated by the nurse. They are deployed by hospital administration, they run continuously, and their outputs directly constrain what nurses do and how many patients they are assigned.

The NNU’s 2024 survey of 2,300 members documented what this looks like in practice. Half of respondents said their employer uses algorithmic systems to determine patient acuity and predict required nursing hours. Of those, 69% said the AI-generated acuity score did not match their own clinical assessment. Among nurses whose employers use automated handoff tools, 48% said those AI-generated reports contradicted their own patient assessment.

The Override Problem

Forty percent of nurses in hospitals using algorithmic patient outcome tools said they cannot override the algorithm’s prediction when their clinical assessment differs. Twenty-nine percent said they cannot alter algorithm-produced wound or pain documentation in the EHR even when they believe it is inaccurate.

Every professional association framework, including the ANA, AAN, and AMA documents described above, establishes that clinical judgment must prevail over AI outputs. The survey data documents a significant gap between that principle and what is happening at the bedside, and the governance frameworks have no mechanism to close it.

The gap has a structural explanation. The patient acuity and staffing tools that nurses cannot override were not purchased by clinical leadership. They were sold to hospital CFOs and operations teams on the explicit promise of reducing labor costs. Vendor marketing for these systems is direct: AI staffing tools minimize overtime, reduce reliance on expensive agency nurses, and cut contract-labor dependency. Ascension Health reported reducing contract-labor dependency by 15%within six months of implementing a predictive staffing system.

A nurse who can override her acuity score is a nurse who can force additional staffing. Override capability undermines the tool’s core value to its actual customer. The inability to override is not a design flaw. It is, from the purchaser’s perspective, a feature.

Many of these operational tools, including staffing algorithms and acuity scoring systems, also do not meet the FDA’s current definition of a medical device. They face no premarket safety review requirement. No regulator required the vendor to build in an override function, and no regulator currently enforces one.

Presence Without Agency

The academic literature has recently begun to examine what “human in the loop” actually means in practice. A paper in the American Journal of Bioethics specifically flags that researchers and institutions routinely invoke HITL as ethical legitimacy without specifying which humans, in which processes, are doing what, and cites NNU survey data on nurses unable to override algorithmic predictions as a concrete example.

The Institute for Systems Integrity published a related framework in May 2026, distinguishing between two states that are often conflated. In the first, a human is present near the AI system: they are notified, they see the output, and they are in the loop in the awareness sense. In the second, a human has agency: they can interrupt the system, modify its output, or substitute their own judgment without penalty. The institute’s framing is clear:

“A human placed near an AI system is not automatically a safeguard. A clinician asked to approve a recommendation under time pressure, incomplete information, workload overload, and unclear authority may not be exercising judgment.”

The EU AI Act makes this distinction legally operational for high-risk AI systems. Article 14 requires that humans be able to interrupt or override the system’s operation and decide not to use it in a specific situation. This is described as an architectural requirement, not a procedural right. No equivalent US requirement exists for hospital operational AI tools.

A nurse who receives an acuity score that cannot be changed has been notified. That nurse has not been given authority.

What The NNU Is Doing

The National Nurses United published a Nurses and Patients’ Bill of Rights in April 2024. Seven rights are enumerated. Most coverage of this document focuses on Right 7, which demands pre-deployment bargaining rights: the right to negotiate over whether and how AI is implemented before the system is selected.

Right 5 is less examined and more operationally significant given the override data. It establishes the right of nurses to exercise professional judgment and override AI decisions without threat of discipline or discharge. No professional association framework, from the ANA, AAN, or AMA, contains an equivalent enforceable protection. Right 5 is the only document in the current governance landscape that directly addresses what 40% of nurses report experiencing.

The NNU has coordinated a national bargaining campaign through its affiliate network: the California Nurses Association, the New York State Nurses Association, the Michigan Nurses Association, and the National Nurses Organizing Committee in North Carolina. Announced AI contract language has been reported at Mission Hospital in Asheville, North Carolina (2024), Northwell South Shore University Hospital in New York, the New York City hospital systems including NYP, Montefiore, and Mount Sinai following a strike by nearly 15,000 nurses in February 2026, Munson Medical Center in Traverse City, Michigan, and the University of California system, where a contract ratified in November 2025 specifies that nurses play a central role in selecting, designing, and validating new technology including AI systems.

One caveat applies to all of the above. No journalist or outlet has quoted actual contract clause text from any of these agreements. Coverage consistently describes outcomes in terms of “safeguards,” “guardrails,” “approval before deployment,” and “voice in how AI is rolled out.” Whether the ratified language reflects the specificity of the Bill of Rights, including Right 5’s override protection, or represents a narrower pre-deployment consultation right, is unknown without the contract documents.

The physician union equivalent, the Union of American Physicians and Dentists, has identified AI as a bargaining priority, with the organization’s president stating in December 2025 that it is “imperative” that the union engage in dialogue with employers to prevent professional judgment substitution by AI. UAPD has not yet produced ratified AI contract language. That gap is approximately 18 months, which is consistent with the tool asymmetry: physicians have not yet encountered operational AI deployed on them at scale without override rights.

Two Tracks, One Unresolved Problem

The landscape across both professions is now parallel. Both nursing and medicine have a professional association track and a union track responding to AI.

The ANA and AAN are doing what the AMA is doing: publishing principles, developing governance frameworks, and advocating for policy. The difference in urgency between the nursing professional associations and their physician counterparts reflects the difference in tool adoption rates and trust levels, not a fundamental strategic divergence.

NNU is doing something the professional associations can’t do: creating enforceable bedside protections through contract law. They are addressing different layers of the same problem. A hospital can have an excellent AI governance committee and still deploy an acuity algorithm with no override function. A union contract can protect override rights but cannot prevent a poorly validated tool from being purchased in the first place. Both layers are needed.

Neither layer has resolved the problem of definition at the center of all of this. Every framework, every position statement, every contract announcement invokes human oversight as a governing principle. No one has established that human oversight requires the ability to override. Until that definition is settled, “human in the loop” describes a spectrum that runs from a nurse with full authority to substitute her/his clinical judgment to a nurse who receives a notification she/he cannot act on. Both nurses are, technically, in the loop.

Questions Worth Asking

If you are responsible for AI governance at a hospital or health system, the relevant question is whether your governance committee’s approval process specifies override capability as an architectural requirement of deployment instead of a recommendation. A tool that generates outputs clinicians cannot modify does not meet the HITL standard described by the ANA, AAN, or AMA, regardless of what the vendor’s documentation says.

If you are a nurse in a facility with a union contract that includes AI language, the clause text matters more than the press release. “Safeguards” and “guardrails” are not contract language. Right 5 of the NNU Bill of Rights, protection of the right to override without threat of discipline, is the protection most directly supported by the survey data. Whether it appears in your contract is a question worth answering.

If you are a nurse in a non-unionized facility, the ANA Think Tank consensus document and the AAN position statement establish a national professional standard you can cite through shared governance structures or unit councils when raising concerns about tools that cannot be overridden.

If you are a physician, the operational AI category that nursing is responding to is not exclusive to nursing. The tools that constrain clinical judgment without requiring clinical input in procurement are already present in prior authorization, clinical documentation, and diagnostic triage. The UAPD’s December 2025 statement suggests that physician union organizing around AI is overdue.

The governance frameworks say clinical judgment must prevail. The mechanism that makes that principle enforceable at 3am, when an acuity score determines whether a second nurse comes to the floor, does not yet exist in voluntary guidelines.

Meanwhile, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

What the media is saying.

The narrative told by the WSJ Pro, MedCity News, and multiple others goes something like this: two Stanford physicians, Sharif Vakili and Ashwin Nayak, ran a rigorous randomized controlled trial proving an AI system could autonomously manage insulin titration in type 2 diabetic patients. The results were dramatic. The AI group hit glycemic control targets in 15 days on average. Fewer than half the standard care group got there at all within eight weeks. Result: 81% controlled by the AI versus 25% by standard practice.

They then built UpDoc on that clinical foundation, got it FDA-cleared as a Software as a Medical Device, and are deploying what the company calls “physician-grade agentic AI” to autonomously adjust insulin doses between office visits. They claim it is the first of its kind, ushering in a new era of care.

That’s the story, but there is a lot of nuance to be discussed.

What the study actually tested

The MIVA trial (Managing Insulin with Voice AI) was a real RCT, published in JAMA Network Open, and conducted at four Stanford primary care clinics from March 2021 to December 2022. Thirty-two adults with type 2 diabetes were randomized into two groups. Half got the voice AI intervention; half got standard care. The results were genuinely strong for a pilot of that size.

Here’s what the coverage consistently leaves out: the voice AI in that trial was Amazon Alexa. Not a generative model, and nothing resembling what most people picture when they hear “agentic AI.” Alexa in 2021 was a narrow, deterministic speech recognition virtual assistant using wake word detection, intent classification, and programmed rules. When a patient said “my sugar was 140 this morning,” the system translated that to glucose_value: 140 through rigid pattern matching. It did not use probability distributions, learned weights, or generative inference.

The insulin titration logic itself came straight from clinical guidelines published by the American Association of Clinical Endocrinologists and the American College of Endocrinology. If glucose is X, adjust dose by Y. An extremely well-executed, patient-friendly, clinically grounded flowchart, but still a flowchart.

What the MIVA trial proved is that a deterministic, rules-based system could dramatically outperform standard care at insulin titration. That’s a legitimate and important finding. It didn’t prove that AI was necessary. It proved that consistent protocol execution was sufficient and that the existing standard of care was failing patients badly enough that almost any disciplined approach would beat it.

What UpDoc actually built

The commercial UpDoc product is architecturally different from what ran in the MIVA trial. The FDA documents describe three software components: a provider-facing web portal, a patient mobile app, and a cloud-based application with a “Conversation Service” (the UpDoc Agent) and a “Clinical Service.”

The Conversation Service is where the LLM lives. It handles patient interaction: collecting glucose readings, asking about symptoms, communicating instructions back in natural language. This is the part UpDoc markets as “agentic AI.” The Clinical Service is where the dosing decision actually happens. It computes insulin instructions based on treatment parameters defined by the ordering physician. That’s the deterministic calculator.

The architecture is sensible. When your favorite AI needs to do arithmetic, it calls a Python tool rather than generating an answer probabilistically. You want a specific and correct (deterministic) answer to a math problem, not the most likely (probabilistic) answer from an LLM. UpDoc uses the same design. It’s brilliant. The problem is the marketing claims about it.

What the FDA actually cleared

Pull up the FDA K253281 submission and decision summaries for UpDoc. Two things struck me.

First: “No clinical testing was performed.”

The MIVA trial appears nowhere in either FDA document. It is not mentioned as supporting evidence, as a reference, or in the bibliography. The FDA never evaluated the 32-patient sample. It never weighed the 81% versus 25% outcome data. The Stanford trial played no role in the clearance.

Second: The clearance rests on substantial equivalence to the d-Nav System, a handheld insulin dose calculator made by Hygieia, Inc., which was cleared in 2018. It’s not an AI product or an LLM. It’s a software-based dose calculator that predates the MIVA trial by three years. The product code is NDC, and the regulation is 21 CFR 868.1890 “Predictive pulmonary-function value calculator,” a classification repurposed for insulin calculators. The word “AI” doesn’t appear in the regulatory classification.

What the FDA evaluated was software testing per IEC 62304, cybersecurity review, and human factors validation studies. That’s it. UpDoc is a Class II device cleared because it’s substantially equivalent to a prior calculator, with a voice-and-chat interface as its primary differentiating feature.

The change plan is the most revealing document

The Predetermined Change Control Plan (PCCP) is a roadmap of modifications UpDoc can make post-clearance without filing a new 510(k). It contains a critical sentence:

All future modifications must “maintain deterministic insulin dosing logic without altering core clinical decision-making.”

The FDA locked this in as a condition of clearance. UpDoc can’t exchange a probabilistic LLM for dosing decisions without filing a new submission. Whatever “agentic AI” means in the press releases, the cleared device’s dosing engine is, by regulatory requirement, deterministic. The regulators saw the architecture, understood which layer was doing which job, and explicitly required the calculator layer to stay a calculator.

The PCCP also specifies “zero tolerance for deviation and incorrect unit conversion rates of zero” for alternative data input methods, including voice. They’ve identified the handoff between the LLM interface and the deterministic dosing engine as a risk point. That’s the right decision. It also quietly acknowledges that the handoff is where the probabilistic layer touches a safety-critical area, and that this interface has never been clinically validated.

“Agentic AI” — and what that actually means

UpDoc’s press release calls the platform “physician-grade agentic AI” at least three times. The coverage has largely accepted this framing without much scrutiny.

Agentic AI has a reasonably specific meaning in the field: a system that perceives its environment, makes autonomous decisions across multiple steps, selects and uses tools, and pursues a goal over time while adapting its approach based on intermediate results. The defining characteristic of a true agent is that it decides how to accomplish something, not just what to output when given a specific input.

To its credit, UpDoc has been transparent about what it means by the term. Their press release defines agentic through a three-step workflow: the system monitors patient data and identifies trends requiring intervention, executes insulin titration within physician-approved parameters, then closes the loop by triggering follow-up lab orders and documenting the intervention in the EHR.

Map each of those steps against what the clearance documents actually describe, and the picture doesn’t look as agentic.

“Monitors patient data and identifies trends”: The system receives glucose values the patient reports or a CGM transmits, then checks them against pre-defined thresholds. That’s threshold alerting. A blood pressure cuff that beeps when you’re hypertensive does the same thing.

“Executes titration within physician-approved parameters”: This is the deterministic calculator we’ve already covered. This is very good and safe for patients. But to be clear, the AI has no agency here. It can’t deviate, can’t reason an alternative approach, can’t decide that a different protocol might fit better.

“Triggers follow-up lab orders and documents in the EHR”: This is the most plausibly agentic-sounding item on the list and genuinely new relative to the predicate (comparison) device. But “triggers necessary follow-up labs” almost certainly means the physician pre-specified which labs fire under which clinical conditions. It’s another if/then rule in the protocol, executed automatically. Important, but still not agentic reasoning.

The “physician-governed” framing they use to address safety concerns is the clearest argument against the agentic claim. They describe the physician as prescribing the treatment plan while the AI implements it within defined boundaries, with zero tolerance for deviation, as required by the PCCP. A system that’s fully constrained by a pre-specified protocol, with no discretion and no ability to adapt its approach, isn’t an agent. It’s an automated executor with a very detailed job description.

For comparison, insulin pumps automate delivery. Ventilators automate titration. Pacemakers automate rhythm correction. Automated pharmacy refill systems initiate patient outreach. None of those are called agentic AI because automation and agency aren’t the same thing. What’s genuinely new about UpDoc is the natural language interface, the EHR integration, and the physician governance model. Those are real innovations worth evaluating on their own terms. Calling them agentic doesn’t make them more impressive; it makes the term less meaningful.

The conflict of interest worth understanding

The Medscape coverage flagged that Nayak and co-authors disclosed owning UpDoc stock at the time of publication. UpDoc was founded three months after the MIVA trial was completed. The company didn’t exist when the trial was designed or conducted, so there was nothing to disclose at conception. The trial appears to have been designed and executed cleanly.

What the disclosure reflects is that by the time the paper was published, the researchers had become the founders. The conflict isn’t in the data. It’s in how that data has since been used. The researchers who designed the study are now the executives with the most to gain from that study being accepted as definitive validation of their commercial product. They’re the most prominent voices promoting it. They’re the ones driving the conflation of the MIVA findings with UpDoc’s commercial viability.

That’s not misconduct. Physician-researchers commercializing their findings is how medical innovation is supposed to work. But it does mean the most enthusiastic advocates for the study’s conclusions have the strongest financial interest in those conclusions being stretched beyond what a 32-patient pilot can actually support.

The liability question nobody asked

One detail from the WSJ piece deserves attention. CEO Vakili drew a clear legal line: UpDoc is liable for accurately implementing the physician’s care plan. It’s not responsible if the care plan itself is faulty.

That’s a meaningful posture, and it’ll be tested. When an autonomous AI executes a physician’s protocol and something goes wrong, the line between “bad protocol” and “bad execution” is exactly what litigation will contest. A plaintiff’s attorney doesn’t need to prove the algorithm malfunctioned. They need to create reasonable doubt about where the failure originated. The Cleveland Clinic’s executive framing of UpDoc as liable for implementation is a clean division of responsibility in a press release that will look considerably more complicated in a deposition.

What’s genuinely worth crediting

The care gap UpDoc is targeting is real and large. Basal insulin titration requires frequent patient contact, glucose logs, clinician availability, and patient follow-through. The MIVA trial demonstrated that even a fully deterministic system dramatically outperforms passive standard care. If UpDoc’s commercial product can replicate that in a larger, more diverse population, patients will be better off.

The FDA pathway they chose is the right one. Robert Califf noted they sought regulatory scrutiny rather than avoiding it. That’s notable in a space where plenty of clinical AI tools deploy without any regulatory engagement at all. And the architecture (physician sets the protocol, algorithm executes it, LLM handles the conversation) is well-reasoned for this use case. Deterministic dosing logic is exactly what you want when you’re adjusting medications autonomously. You want rule-following fidelity, not creative inference.

The concern isn’t the product. It’s the story being told about it.

The three-layer disconnect

The MIVA trial tested a deterministic Alexa-based system, not UpDoc. It validated consistent protocol execution. UpDoc then added an LLM to that architecture, and that substitution has never been clinically validated. The FDA, meanwhile, cleared a dose calculator based on a 2018 predecessor and never saw the trial or evaluated the LLM. Three layers. Three separate stories. What UpDoc is selling is the version where they all fuse into one: the trial validates the product, the product earns the clearance, the clearance confirms the AI. None of those connections hold up.

Your license. Your responsibility.

UpDoc may become an important tool. The clinical problem is real. The regulatory pathway was handled responsibly. The underlying architecture is defensible.

But before your health system signs on or you prescribe this as a treating physician, ask the questions the press coverage didn’t:

What, specifically, does the LLM component do, and what does the deterministic clinical service do? Get that in writing.

Has the LLM interface layer been clinically validated in a population comparable to yours? The MIVA trial didn’t test it. The FDA didn’t evaluate it. Who did?

What happens when the LLM misparses a patient’s glucose report? What’s the actual error rate at the handoff between the conversational layer and the dosing engine? The PCCP mandates zero tolerance, but mandating and demonstrating are different things.

What does the liability split mean in practice for your institution when something goes wrong?

The algorithm that ran in the MIVA trial followed AACE and ACE guidelines faithfully. That system worked. Know what’s running in the commercial product that replaced it. That’s our job.

On January 6, 2026, the FDA published revised guidance on clinical decision support (CDS) software, and the coverage has largely been positive. Less red tape. Faster innovation. Tools that can finally say what they actually mean instead of hedging behind padded lists of possibilities.

There’s real merit to that change. But there’s also a version of this story that hasn’t been told yet, one that is especially relevant in emergency medicine and critical care.

Let’s get into it.

As always, if you enjoy reading, I encourage you to subscribe and tell a friend.

Sam

Subscribe now

What Actually Changed

For years, one of the FDA’s more awkward regulatory quirks was that clinical decision support (CDS) software offering a single recommendation was more likely to be classified as a medical device than software offering multiple options. The perverse result: developers were incentivized to dilute outputs, presenting three or four choices even when the evidence clearly favored one. Clinicians had to sort through the noise. Nobody loved it.

The new guidance fixes that. The FDA will now allow single-recommendation CDS without triggering device classification, provided the logic, data sources, and guideline basis behind that recommendation are visible to the clinician. They call it a “glass box” model: not opaque AI rendering a verdict, but transparent reasoning a clinician can inspect before acting.

There’s also an expanded “general wellness” carveout for consumer wearables. Devices reporting metrics like blood pressure and oxygen saturation can now stay outside device regulation as long as they don’t make diagnostic claims. That part is not controversial.

What’s important to understand is that this isn’t complete deregulation. The FDA still asserts authority over opaque models and tools that substitute for clinical judgment. The line just moved meaningfully in the direction of “trust the clinician to evaluate the output.”

The Case for Optimism (And It’s a Real One)

I want to be fair here because my job isn’t to reflexively oppose change. It’s to evaluate it honestly.

The previous regulatory logic was genuinely broken. When guidelines and patient data points in one direction, medicine often has a right answer. Forcing CDS to pretend otherwise didn’t make things safer; it just made the tools harder to use.

A well-designed, transparent AI that surfaces the relevant evidence, flags the applicable guideline, and tells you its reasoning is a useful clinical assistant. If the FDA’s revised framework actually enables more of that and less of the multi-option noise, that’s a good thing for both clinicians and patients.

This Should Give You Pause

The entire framework rests on one assumption: that transparency functions as a reliable safeguard. It’s worth asking whether that assumption holds in practice.

In theory, a glass box lets you inspect the AI’s reasoning before accepting its recommendation. In practice, in an overcrowded ED, at the tenth hour of a twelve-hour shift, with a septic patient in bay 3 and a chest pain in bay 7, are you clicking through the reasoning panel? Are your colleagues? Are you confident that institutional productivity pressures won’t quietly reward the physicians who just accept the output and move on?

Cognitive offloading isn’t a character flaw. It’s a predictable human response to cognitive overload. The FDA guidance even acknowledges automation bias as a concern, but it doesn’t solve it. It just names it and hands the responsibility back to the clinician.

And here’s the real kicker: the FDA explicitly carved emergency and time-critical CDS out of the loosened framework. The guidance states that software intended for urgent, high-stakes decisions where the clinician lacks time to independently review the logic does not qualify for the exemption, specifically citing automation bias in those settings. That is, our propensity to accept what the machine is telling us even when there is contradictory evidence.

Read that again: the specialty with the highest acuity, the fastest decision cycles, and the most cognitively demanding environment is the one the FDA flagged as highest-risk. If you’re practicing emergency medicine, the tools most likely to influence your practice are the ones that still require close regulatory scrutiny. Which means some of what’s entering your ED workflow may not meet that bar, and you may not know which is which.

The Liability Math Nobody Is Talking About

Here’s where it gets uncomfortable.

The FDA declined to define what “clinically appropriate” means when it comes to single-recommendation CDS. That decision gets made by the developers. And when an AI-influenced recommendation leads to a bad outcome, the responsibility lands where it always has: with the physician who accepted it.

More AI authority in the workflow. Same physician accountability. That’s not necessarily wrong. It’s how medicine has always worked with every tool we use. But it’s worth being clear-eyed about the asymmetry. The guidance accelerates the path for tools to enter your workflow while leaving unchanged the standard of care you’re held to when they’re wrong.

Your license. Your responsibility. That’s not just a tagline. It’s the legal and ethical reality that the FDA’s framework reinforces.

The LLM Blind Spot

One more thing worth noting: the guidance is nearly silent on generative AI.

The tools that are actually proliferating at the bedside right now- AI scribes, ambient documentation platforms, chatbot-style decision support embedded in the EHR- are largely built on large language models. And LLMs present a specific transparency challenge that rule-based systems don’t: their outputs are probabilistic, not deterministic. That means the models rely on educated guesses when faced with uncertainty, rather than following a set of rules that reach the same conclusion every time. The “glass box” concept is much harder to apply when the reasoning isn’t a traceable logic chain.

The FDA’s guidance doesn’t really address this. Whether that represents regulatory humility or a gap that needs to be filled is an open question. However, it means clinicians are navigating a rapidly evolving LLM-enabled ecosystem without clear guidance on how those tools fit into the framework.

A Case Study

Abstract regulatory language is easier to evaluate when it touches something real. So let’s apply the FDA’s four criteria to a tool many emergency physicians are already using: OpenEvidence.

OpenEvidence allows physicians to enter patient-specific clinical information, including protected health information, and receive synthesized answers and recommendations from peer-reviewed literature. It cites its sources. It also draws conclusions.

Walk it through the criteria.

✅Criterion 1: Data inputs. OpenEvidence ingests text-based clinical information: symptoms, labs, diagnoses, history. This isn’t imaging data or signals from diagnostic hardware. Criterion 1 is probably satisfied.

✅Criterion 2: Displaying and analyzing medical information. The software matches patient-specific data against clinical literature and guidelines, which is precisely the FDA’s own example of what this criterion covers. Criterion 2 is satisfied.

❓Criterion 3: Supporting versus directing judgment. This is where it gets murky. The FDA draws a sharp line between software that presents options for a clinician to weigh and software that summarizes answers and draws conclusions. OpenEvidence’s outputs function more like directives than option lists. The answer to this criterion depends on exactly how its recommendations are framed, and that’s worth looking at closely.

❌Criterion 4: Independent reviewability. This is where the ED context becomes decisive, and where the FDA’s own language is crystal clear.

The guidance states directly that software intended for critical, time-sensitive decisions does not meet Criterion 4, because clinicians are unlikely to have sufficient time to independently review the basis of the recommendations. The FDA states that in urgent situations, the pressure to act accelerates the tendency to accept AI output without independent scrutiny.

OpenEvidence used by a primary care physician working up a chronic condition, with time to click through citations and evaluate the reasoning, might satisfy all four criteria. The same tool, used by an emergency physician making a time-critical disposition decision, certainly doesn’t. Not because the software changed. Because the context did.

That distinction matters more than most clinicians realize. The FDA’s framework isn’t tool-specific; it’s context-specific. And a lot of what’s currently running in ED workflows may be operating in a regulatory gray zone that neither clinicians nor hospital administrators have fully reckoned with.

Citing sources isn’t the same as giving clinicians time to read them. In the ED, those two things are rarely the same.

What to Do With All This

The FDA’s January guidance isn’t reckless, and it isn’t trivial. It’s a deliberate bet that clinical AI can move faster without sacrificing safety if clinicians stay meaningfully engaged with what the tools are telling them and why.

Whether that bet pays off depends almost entirely on us. Before your department adopts a new AI CDS tool, here’s what I’d want to know:

Does the reasoning actually surface in the workflow, or is it buried three clicks deep? What does the vendor say about performance in high-acuity, time-critical settings specifically? Has it been validated on a patient population that resembles yours? Who reviewed the validation data, and was it anyone independent of the company selling it? And critically, what happens when it’s wrong, and how is that tracked?

The FDA has done its part by drawing a clearer map. But we’re the ones practicing in the territory, and the terrain in an ED at 2 am looks nothing like the conference room where these policies get written.

Transparency is a good start. Reflection is the part we have to supply ourselves.

As always, if you enjoy reading this newsletter, subscribe and tell a friend.

Sam

Subscribe now

Clinical AI tools have become part of daily practice for many physicians. OpenEvidence, Doximity GPT, AI scribes, and clinical decision support tools can improve efficiency and provide valuable assistance. At the same time, these tools create new risks that most physicians never encountered during training.

I have written about the limitations, regulatory concerns, and real-world testing of clinical AI systems. The most common problems I see are not dramatic AI hallucinations. They are workflow mistakes made by clinicians who assume these tools are safer, more accurate, or more legally protected than they actually are.

To help physician leaders educate their clinical staff, I created the following Clinical AI Safety Checklist. You can download it here and read more about each item below.

Clinical Ai Safety Checklist Ashooreview

520KB ∙ PDF file

Download

Download

1. Don’t Upload ECGs or X-Rays

Most general-purpose clinical AI tools are not FDA-cleared devices for interpreting ECGs, radiographs, CT scans, MRIs, or other diagnostic images.

My own testing of multiple systems has demonstrated substantial errors in ECG and radiology interpretation. These errors can be subtle and dangerous because the AI often presents its conclusions with confidence. Often, there is no warning that the system can not accurately read them, and the presence of an image upload feature is misleading.

If a clinical AI platform lacks FDA authorization for diagnostic image interpretation, physicians should avoid using it for that purpose.

Dive Deeper:

2. Don’t Sign a BAA if You Work in a Hospital System

Many physicians assume that signing a HIPAA Business Associate Agreement solves privacy concerns. After all, it’s a convenient button click in the AI tool, and it reassures the user with a “HIPAA Compliant” banner.

In reality, employed physicians do not own the data and are not the entity responsible for it. In the law’s eyes, the protected health data is stored by the hospital, which has the responsibility of safeguarding it. For that reason, the BAA has to occur between the AI service and the hospital. So clicking that little HIPAA BAA agreement box only puts you at risk for acting as an “agent” of the hospital without authority. Unless you are in private practice, own the practice, and use the AI tool only on those patients, better to avoid this trap.

Dive Deeper:

3. Don’t Upload PHI

Even when a platform offers a BAA, physicians should think carefully before uploading identifiable patient information.

The question is not simply whether the technology can accept PHI. The question is whether you are authorized to disclose that information under your organization’s policies and contractual arrangements.

When in doubt, de-identify the information or avoid uploading it altogether. If you don’t know what’s involved in de-identifying the information, read more at the link below. It’s a critical skill that will keep you (and your hospital) out of a lawsuit.

Dive Deeper:
Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule

4. Don't Trust AI-Generated Insights Without Verification

Many clinical AI systems now generate summaries, assessments, risk predictions, or observations across multiple visits. These insights can be useful. They can also be wrong.

An AI may only be looking at a subset of encounters, incomplete documentation, or fragmented records. As a result, it may generate conclusions that appear reasonable while missing critical context.

Physicians should treat AI-generated insights the same way they would treat recommendations from a trainee: useful starting points that require independent verification.

Dive Deeper:

5. Don't Use AI-Informed Clinical Reasoning Without Documenting It

This may be the most overlooked issue in clinical AI today.

If an AI-generated insight influences your diagnosis, treatment plan, referral decision, or other aspect of care, that reasoning should be documented in the patient’s official medical record.

Patients can review information contained in the medical record and request corrections when appropriate.

Information that exists only inside an AI platform may influence care without the transparency and accountability legally required by HIPAA and the Cures Act.

If the AI helped drive a clinical decision, document the relevant information in the chart.

Deeper Dive:

6. Don't Sign AI-Generated Notes Without Reviewing Medications and Mental Health History

When physicians think about AI scribe errors, they often focus on hallucinations. But a report earlier this year from the Ontario Auditor General’s office found that AI scribes committed these errors most often:

• 45% hallucinated treatment plans, blood tests, or referrals that were never discussed

• 60% documented incorrect medication names or dosages

• 85% omitted critical aspects of mental health history

AI scribes frequently produce notes that appear polished and complete while leaving out clinically important details. Before signing any AI-generated note, carefully review medications, mental health history, and other high-risk sections of the chart for omissions.

Your signature confirms the accuracy of the documentation.

Deeper Dive:

7. Don't Delete AI Chat History Without Understanding the Consequences

This issue cuts both ways.

Maintaining AI chat history may provide evidence regarding what information was presented to the physician and what recommendations were generated by the system. Deleting that history may remove information that could later be relevant when evaluating clinical decisions. AI chat histories may also become discoverable during litigation or investigations. Physicians should understand their organization’s policies and think carefully before deciding whether to retain or delete AI interactions.

Most importantly, any information that materially influences patient care should be documented in the medical record rather than existing solely within an AI conversation.

Deeper Dive:

Final Thoughts

Clinical AI tools are becoming a permanent part of healthcare delivery.

The greatest risks are rarely the ones featured in headlines. Most arise from privacy misunderstandings, documentation shortcuts, incomplete records, misplaced trust, and workflow decisions made by clinicians under pressure.

Technology will continue to improve.

Professional responsibility remains unchanged.

Your license. Your responsibility.

As always, if you enjoy reading, please consider subscribing and telling a friend.

Sam

A recent NYU study comparing frontier AI models with specialized clinical AI tools raised an important question.

If large language models can answer clinical questions as well as, or better than, systems built on curated medical content, what is the future of organizations such as UpToDate, EB Medicine, and other services that summarize and interpret the medical literature?

For decades, physicians have relied on expert-written medical summaries to keep up with an impossible volume of research. These services review the literature, synthesize evidence, and provide practical clinical guidance. They have become part of the background infrastructure of modern medicine.

But the world is changing quickly. Today, a physician can ask an AI system a highly specific clinical question and receive a detailed answer within seconds. The answer can be adjusted for an emergency physician, hospitalist, primary care clinician, resident, or subspecialist. It can summarize trials, compare guidelines, explain mechanisms, and generate a practical approach.

So the question is unavoidable. Are human-curated medical knowledge systems still relevant? I think they are. But their future value may be different from their past value.

The Argument Against Curated Knowledge

The strongest argument against traditional evidence summary services is easy to understand. AI can already perform many of the tasks that physicians historically relied on these services to provide.

It can search the literature, summarize papers, compare studies, explain complex concepts, and personalize an answer. For a busy physician, this is powerful. Instead of reading a long chapter or searching through multiple articles, the clinician can ask a direct question and receive a direct response.

That is a serious challenge to the traditional medical publishing model. If summaries become instant, personalized, and inexpensive, then services built primarily around summaries will need to prove what additional value they provide.

That’s the right question, but only the first one.

Medicine Is Not Just a Search Problem

The mistake is assuming that physicians’ needs consist mainly of finding and summarizing information.

Physicians do not just need information. We need judgment. The harder questions are not always:

What did the study show? Or what papers have been published?

The harder questions are often:

• Does this study deserve my attention?

• Was the methodology strong enough?

• Does this evidence apply to my patient?

• How should I weigh this study against prior evidence?

• Does this change practice?

• What should I do when the evidence is conflicting?

Those are not simple search tasks. They are editorial tasks. They require experience, skepticism, clinical context, and accountability.

That is what organizations like UpToDate and EB Medicine have historically provided. Their value has never been limited to summarizing articles. Their value is in deciding which evidence matters, how it should be interpreted, and how confidently it should be applied.

AI may make summaries abundant. Trustworthy interpretation remains scarce.

The Journalism Analogy

A useful analogy comes from journalism. The internet made information widely available. Search engines made that information easier to find. Social media made it easier for anyone to publish. Yet journalism did not disappear.

The best journalism continued to provide something beyond access to information. It provided verification, context, judgment, and accountability. Medicine is entering a similar phase.

AI makes medical information easier to retrieve and summarize than at any point in history. But that does not eliminate the need for trusted institutions that evaluate the quality of information. In fact, it may make them more important.

When information is scarce, access is valuable. When information is abundant, trust becomes valuable.

My Own Experience Testing Clinical AI

My own recent experiments with clinical AI have made this issue all the more real. I have tested multiple models across medical cases, ECGs, and imaging tasks. The results have often been concerning.

In several cases, AI systems provided polished, confident, and incorrect interpretations. The problem was not the writing quality. The answers were usually clear, organized, and persuasive. That is exactly what makes the errors concerning. A poorly written, wrong answer is easier to distrust. A polished wrong answer is more dangerous.

This is where curated medical knowledge systems still matter. When multiple AI systems can read the same evidence and reach different conclusions, physicians need more than another summary. They need a trusted process for deciding which interpretation deserves confidence.

The Value of Saying “I Don’t Know”

This brings me to what may be the most underappreciated issue in clinical AI.

What should an AI system do when the evidence is insufficient?

In many AI evaluations, a system that does not answer is penalized. From the perspective of a benchmark, that makes sense. Researchers need a scoring system. An unanswered question is easy to count as incorrect.

But medicine is different. A benchmark rewards answers. Clinical judgment rewards calibration.

Every physician understands that some questions do not have clean answers. The literature may be sparse. Studies may conflict. The population may not match the patient. Outcomes may be surrogate rather than patient-centered. The best available evidence may be old, biased, underpowered, or indirect. In those situations, a confident answer may be satisfying. It may also be misleading.

One of the most important functions of a trustworthy medical knowledge system is recognizing when the evidence does not support a recommendation. That can be frustrating. A clinician wants help… guidance… an answer to the question.

But an honest “we don’t know” may be more valuable than an unsupported conclusion. This is where guardrails should be seen as a feature rather than a flaw.

An AI system that declines to answer may appear less capable on a leaderboard. It may also be demonstrating a form of restraint that is essential in medicine.

The ability to say “I don’t know” is not a weakness. It’s part of trust.

Guidelines Are Full of Uncertainty

This is not unique to AI. Medical guidelines frequently acknowledge uncertainty. Expert panels often conclude that evidence is insufficient. Recommendations are often graded as weak, conditional, or based on low-quality evidence. That is not a failure of guideline development. That is evidence-based medicine working properly.

A good guideline does not simply provide an answer to every question. It tells the reader how confident to be in the answer.

The same principle should apply to clinical AI. A system that always answers may feel more useful. A system that knows when not to answer may be safer.

The future of clinical AI should not be measured only by how often a system produces a response. It should also be measured by whether the system knows when a response is justified.

Where Human-Curated Libraries Still Matter

Human-curated medical libraries are systems for managing uncertainty. They don’t just collect papers. They filter, interpret, and reconcile them. They decide when evidence is strong, when it is weak, and when no recommendation can be made. That work becomes even more important when AI can generate an answer to almost anything.

A physician using AI may ask: What does the literature say?

But the deeper clinical question is often: What should I trust?

That is where expert curation still matters. The future may not be physicians reading long chapters on a website. It may be AI interfaces built on top of carefully maintained evidence bases. The interface may become conversational, personalized, and fast. But the underlying need remains the same.

Someone still has to decide what evidence is reliable.

Someone still has to decide how conflicting studies should be interpreted.

Someone still has to decide when uncertainty should be made explicit.

The Future

The future of these organizations, like Up-To-Date and EB Medicine, will probably depend on how they define their own value. If they define themselves as article publishers, they will face increasing pressure. If they define themselves as trusted evidence institutions, their role may become more important.

AI can help deliver their knowledge more effectively. It can make their content easier to search, easier to personalize, and easier to apply at the bedside. But the core value is not the chatbot. The core value is the editorial process behind the chatbot. That is the part physicians should care about.

• Who reviewed the evidence?

• How was it selected?

• How were conflicting studies handled?

• What was excluded?

• How often is the recommendation updated?

• What level of confidence supports the answer?

• When does the system refuse to answer?

Those questions matter far more than whether the interface looks modern.

Final Thoughts

AI will make medical summaries abundant. That does not make expert medical curation obsolete. It may make expert curation more important.

The future of medical knowledge will be defined by which systems can earn trust. That requires more than speed. It requires evidence appraisal, clinical judgment, transparency, accountability, and humility.

In medicine, the best answer is sometimes a confident recommendation, a cautious recommendation, or no recommendation at all. As AI becomes more capable, physicians should pay close attention to the systems that know when to pause.

In an age when every AI can generate a summary, the real value of human-curated medical knowledge may be its ability to decide which answers deserve to exist.

1. Frontier models (GPT, Gemini, and Claude) outperformed specialized clinical AI tools across every evaluation.

2. The performance gap narrowed as the testing became more clinically realistic.

3. The most important contribution of this paper is the creation of a benchmark built from actual physician questions asked during routine clinical care.

One reason this paper caught my attention is that it mirrors observations from my own recent testing. In a series of evaluations, including OpenEvidence, Doximity Ask, Heidi Health, Glass Health, ChatGPT, Claude, and Gemini, I found that specialized medical AI products rarely demonstrated a clear advantage over frontier models.

5 Medical AI Models Got This Case Wrong. Is Your Favorite One of Them?

Can Medical AI Read an ECG?

11 Medical AI Tools Read These X-rays: Everyone Missed The Pneumothorax

Let’s get into the details of the study.

As always, if you enjoy reading Ashoo Review, subscribe and tell a friend. There’s no better reference.

Sam

Subscribe now

A new Nature Medicine paper asks a question many clinicians have been wondering over the past year: Are specialized medical AI tools actually better than frontier models?

The authors compared OpenEvidence and UpToDate Expert AI against GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 using 1,000 benchmark and 100 physician-generated clinical questions.

The frontier models won, but the more interesting story is how the margin changed for the three datasets tested.

The Study Design

The researchers used three different datasets. The first was MedQA, a collection of 500 USMLE-style multiple-choice questions designed to assess medical knowledge.

The second was HealthBench, a collection of 500 open-ended healthcare prompts scored against detailed evaluation rubrics.

The third was the study’s most interesting contribution: the Real Clinical Queries benchmark, or RCQ. For RCQ, the investigators sampled 100 de-identified physician questions from NYU Langone’s HIPAA-compliant GPT environment. The responses were then reviewed by blinded clinicians who rated correctness, completeness, safety, and clarity.

Each dataset is asking a different question.

• MedQA asks whether a model can answer a medical exam question.

• HealthBench asks whether a model can satisfy a detailed rubric.

• RCQ asks whether a physician would find the answer useful.

A Curious Pattern Emerged

The frontier models outperformed the specialized clinical tools across all three evaluations. That part is easy to summarize. What caught my attention was something else.

The performance gap became smaller as the evaluation became more clinically realistic.

On HealthBench, the separation was dramatic.

• GPT-5.2 scored 88.0%

• OpenEvidence scored 62.6%

• UpToDate scored 61.3%

Looking only at those numbers, one could conclude that the frontier models were operating in an entirely different league.

MedQA told a different story.

• Gemini scored 97.4%.

• OpenEvidence scored 89.6%.

• UpToDate scored 88.4%.

The frontier models still led, but the gap was considerably smaller.

Then came the RCQ benchmark.

Here, the frontier models again formed the top tier, with Gemini, GPT-5.2, and Claude receiving the highest clinician ratings. But the differences were narrower.

On a four-point scale

• Gemini averaged 3.62

• GPT 3.54, Claude 3.52

• OpenEvidence 3.24

• UpToDate 3.17.

The superiority was statistically significant, but all of the systems generally received favorable ratings. That’s what makes the RCQ findings so interesting. If you only looked at HealthBench, you might conclude that the frontier models were vastly superior. The real-world physician evaluations tell a more nuanced story. Clinicians still preferred Gemini, GPT, and Claude, but OpenEvidence and UpToDate were generally producing acceptable answers as well.

In other words, the ranking remained the same, but the practical distance between the systems became smaller once the evaluation moved closer to actual clinical use.

That matters. This paper doesn’t tell us that specialized medical AI tools are failing. It’s telling us that specialized medical AI tools did not demonstrate a meaningful advantage over frontier models.

Are We Measuring Medicine or Benchmark Performance?

I suspect many readers will focus on who won. But the more interesting question may be why the margin changed between the models.

As AI systems improve, benchmark leaderboards may exaggerate differences that become less noticeable during day-to-day clinical use. A model can be significantly better at satisfying a rubric while being only modestly better when a physician evaluates the final answer.

That does not make benchmarks unimportant. It does suggest that real-world evaluation deserves more attention.

The RCQ dataset is arguably the strongest part of the paper because it moves the discussion closer to actual clinical practice.

An Unexpected Finding in the Methods

One detail that surprised me was buried in the Methods section. The authors built their real-world benchmark by sampling 100 de-identified physician questions from NYU Langone’s HIPAA-compliant GPT environment.

To do that, those interactions had to be recorded and retained somewhere. Researchers were then able to access those logs and use them to create the benchmark.

The paper doesn’t tell us exactly what was stored or for how long, but it does provide evidence that at least some health systems are monitoring and reviewing how clinicians use AI in practice.

That struck me as noteworthy. Much of the public conversation around healthcare AI focuses on model performance.

This paper quietly reveals that large health systems are beginning to accumulate enough real-world AI usage data to study clinician behavior, evaluate tools, and build institution-specific benchmarks.

That may become increasingly important as AI moves from experimentation into routine clinical workflows.

What This Means for Clinical AI

The paper raises a difficult question for the growing number of companies building clinician-focused AI products. What exactly is the advantage being offered?

For years, the assumption has been that medicine requires specialized systems trained, tuned, or wrapped specifically for healthcare. That assumption seems reasonable. Yet in this study, OpenEvidence and UpToDate Expert AI did not outperform GPT, Gemini, or Claude. OpenEvidence is particularly interesting because it has become one of the most recognizable names in clinical AI.

That doesn’t mean specialized medical AI has no value. Clinical workflows involve far more than answer generation. Citation quality, medical content licensing, governance, enterprise support, and workflow integration matter.

Those factors may ultimately prove more important than small differences in answer quality. Still, this paper suggests that specialization alone is no longer enough to assume better performance.

Looking Ahead

The authors showed that real physician questions can be collected, de-identified, reviewed by blinded clinicians, and used to compare AI systems. Medical AI needs more independent evaluation and fewer marketing claims.

The future of AI assessment will likely involve real workflows, real users, and real clinical questions rather than relying exclusively on public benchmarks.

OpenEvidence Responds

On June 14th, 2026, OpenEvidence publicly challenged the study’s conclusions and methodology on X.com.

The company’s critique focused on three areas.

1. Benchmark contamination. OpenEvidence argues that public datasets such as MedQA have likely been seen by modern frontier models during training, making them a poor measure of real-world performance. - I agree.

2. HealthBench. The company notes that HealthBench was created by OpenAI and argues that the benchmark rewards stylistic choices that may not reflect meaningful clinical quality. - Likely true.

3. The RCQ dataset itself. OpenEvidence points out that the physician-query dataset is not publicly available and that limited information is provided regarding question selection, reviewer selection, and dataset construction. The company also notes that the RCQ evaluation was added after peer reviewers criticized the original submission for lacking stronger real-world grounding. - This is valid, but not unusual. As soon as a valid medical dataset is publicly released, it becomes fodder for frontier LLMs to use for training. So it makes sense to keep the content private.

These criticisms are worth considering. At the same time, OpenEvidence’s response highlights an interesting point of agreement.

Both sides appear to believe that benchmark performance is insufficient. The company argues that clinical AI should be evaluated using real-world clinical workflows and meaningful clinical outcomes rather than benchmark leaderboards.

The disagreement is not whether real-world evaluation matters. The disagreement is whether this particular real-world evaluation is convincing. That question will likely require additional independent studies from other health systems to answer.

Final Thoughts

The publication of this paper and the rapid response from OpenEvidence highlight how quickly the conversation around clinical AI is evolving.

Both perspectives contain important truths. Five years from now, few people will remember which model topped the leaderboard in this paper. The more durable contribution may be the demonstration that clinical AI can be evaluated using real physician questions and blinded clinician review. At the same time, the questions raised about benchmark contamination, transparency, and reproducibility deserve serious consideration.

Clinicians do not need another leaderboard. We need evidence. The debate this paper has already generated may prove just as valuable.

As always, if you enjoy reading this newsletter, consider subscribing and telling a friend.

Sam

Subscribe now

Over the last several years, healthcare has undergone a remarkable shift in transparency. Not long ago, patients waited days or weeks to receive laboratory results, imaging reports, pathology findings, and physician notes. Some health systems intentionally delayed access to information until a physician could review it first. Others required patients to submit formal records requests to obtain information that already existed electronically.

The 21st Century Cures Act and subsequent Information Blocking regulations accelerated a different vision of healthcare. The guiding principle became increasingly clear: if patient information exists electronically, patients should have access to it.

Healthcare spent years debating how quickly information should be released. We may soon be debating something very similar. What happens when information influencing clinical decisions is created not by a physician, laboratory, or radiologist, but by an AI system?

From Documentation to Memory

Ambient scribes listen to patient encounters, generate notes, and reduce clerical burden. Whether one uses Heidi, Glass, Abridge, Suki, or another platform, the core value proposition is largely the same: document what happens during the encounter.

The next generation of tools is beginning to do something different. Many AI platforms now maintain context across encounters, synthesize information from multiple visits, and generate longitudinal patient summaries. OpenEvidence can associate patient-specific searches and information with an individual patient. Heidi and Glass can pull together information spanning multiple encounters. Several platforms can generate summaries that span months or years of clinical history.

These capabilities are often described as contextual awareness, patient memory, longitudinal synthesis, or pre-visit intelligence. Whatever terminology is used, the underlying shift is significant. The industry is moving from AI that documents encounters to AI that remembers patients.

When Does Memory Become a Record?

At first glance, this seems like a distinction without a difference. After all, physicians have always reviewed prior notes, laboratory results, imaging studies, and discharge summaries. AI simply makes that process faster. But there is an important progression worth examining.

At the most basic level, an AI system retrieves information. “Show me the last three notes.” Few would view that as creating a new medical record.

The next level is summarization. “Summarize the last three notes.” Again, the system is organizing information that already exists.

Then comes another level entirely as the AI prompts the physician with “This patient demonstrates progressive cognitive decline and increasing medication nonadherence.” Now the AI has created a patient-specific conclusion that may influence future clinical decisions.

At that point, the system is doing more than retrieving information. It is generating and retaining patient-specific knowledge. If clinicians rely on that information, what exactly is it?

A Possible Future

Imagine opening a patient’s chart five years from now. Before you review a single note, an AI-generated summary appears:

Longitudinal Patient Summary

• Progressive decline in renal function over two years

• Multiple episodes of medication nonadherence

• Increasing emergency department utilization

• Missed specialist referrals

• High likelihood of care fragmentation

The summary immediately shapes your thinking. You order additional testing. You spend more time discussing medication adherence. You prioritize care coordination. The AI-generated summary influenced your clinical decision-making within seconds.

• Yet no physician wrote that summary.

• No laboratory generated it.

• No radiologist signed it.

• No individual encounter contains it.

The information was synthesized by software and retained over time. Is that simply a software feature? Or is it more like a clinical record?

The Legal Framework Was Built for a Different World

Current law does not provide a clear answer. HIPAA provides patients with the right to access protected health information. The regulation states that individuals have a right to “inspect and obtain a copy of protected health information about the individual in a designated record set.”

The key phrase is designated record set. The phrase encompasses all records used to make medical decisions about individuals. Most clinicians intuitively understand what belongs in the medical record when information originates from a physician, laboratory, radiologist, or pharmacist. The answer becomes less obvious when information is generated by an AI system and retained across encounters.

The Cures Act and Information Blocking regulations add another dimension. The Information Blocking Rule defines information blocking as a practice that is likely to interfere with, prevent, or materially discourage access, exchange, or use of electronic health information. Yet current AI systems do not provide patient portal access or a mechanism for patients to view their own information.

Federal policy has been moving steadily toward greater transparency and fewer barriers to information access. Yet neither HIPAA nor the Cures Act was written for a world in which software could develop and retain its own understanding of a patient over time.

If a physician routinely relies on an AI-generated patient summary, should patients be able to access it? Is that summary part of the designated record set? These are the questions the law will have to answer soon.

The Governance Challenge

The regulatory questions may ultimately prove easier than the governance questions.

Consider a few practical issues.

• Who owns AI-generated patient memory?

• Who is responsible for correcting errors?

• How long should it be retained?

• What happens when an AI-generated summary conflicts with the underlying chart?

• Should these systems maintain audit trails?

• Should patients be informed that longitudinal AI memory exists?

• Should patients have access to it?

• Could it become discoverable during litigation?

Unlike a lab result, which is binary and objective, AI-synthesized 'memory' is interpretative. If an AI incorrectly tags a patient as 'medication nonadherent' based on a misinterpreted data point, that 'memory' can color every future clinical interaction. We don’t have a clear mechanism for patients to challenge or 'edit' these persistent algorithmic conclusions, raising a critical question: how do we protect patients from automated bias that the legal system has not yet classified as part of the medical record?

Health systems are increasingly developing governance frameworks for AI-generated documentation. Far fewer appear to be discussing governance frameworks for AI-generated memory, but the distinction is important.

Documentation captures what happened. Memory influences what happens next.

The Hidden Medical Record

AI may be creating a new category of information. Patient-specific knowledge generated by software, retained across encounters, and used to inform future care.

That information lives somewhere. It may influence clinical decisions. It may persist for years. In many cases, patients may not know it exists.

Healthcare is approaching a new and largely unexamined boundary. For the past decade, we debated who should have access to the medical record. The next decade may be spent defining what the medical record actually is.

Before deploying AI memory systems at scale, health systems should begin asking a few questions:

• Is AI-generated patient memory part of the medical record?

• Would we be comfortable if a patient requested access to it?

• Would we be comfortable if it became discoverable in litigation?

• Do we even know what our AI systems are storing, synthesizing, and retaining?

The answers may shape the next chapter of healthcare transparency.