As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On April 27, 2026, OpenEvidence pulled its clinical AI platform from the EU and UK. The notice that replaced it cited “mounting regulatory uncertainty regarding the treatment of AI systems in the European Union and the United Kingdom, including, among other rules, the EU Artificial Intelligence Act.”

What the EU Actually Requires

Let’s start with what “high-risk” means under the EU AI Act, because most coverage of this story treated it as a vague label. Annex III places clinical decision support software in the high-risk category regardless of whether it’s also regulated as a medical device. That status requires, under Chapter III, Section 2 (Articles 8-15), a conformity assessment before the product reaches the market, technical documentation covering training data characteristics and known limitations, a human oversight mechanism built into the product’s design rather than added as a disclaimer, data governance standards for training data, ongoing post-market monitoring, and mandatory reporting of serious incidents to national authorities within 15 days.

None of that is free. Industry estimates put ongoing compliance at roughly €29,000 a year per AI system, with certification running another €17,000 to €23,000. The penalties for getting it wrong in Article 99 sets fines up to €35 million or 7 percent of global turnover for the most serious violations, up to €15 million or 3 percent for ordinary high-risk violations. A US company weighing whether the exposure justifies the market has real numbers to weigh it against.

The obligation doesn’t stop at the vendor, either. A hospital or clinic that adopts a high-risk AI tool becomes a “deployer” under Article 26, which lists twelve distinct duties, including assigning a person with actual authority to override the system, monitoring its performance, retaining logs for at least six months, notifying affected patients that they’re subject to the system, and training staff to understand what the tool can and can’t do. If you’re a health system reader in the EU wondering whether any of this applies to you, it does the moment you adopt a high-risk AI tool, regardless of what the vendor has or hasn’t done on their end.

What the US Exempts, and How Narrowly

In the US, software like OpenEvidence can avoid FDA device regulation entirely under the Non-Device Clinical Decision Support carve-out in the 21st Century Cures Act. FDA’s final guidance, issued March 11, 2026, sets four criteria a product has to meet simultaneously to qualify:

• It cannot acquire, process, or analyze a medical image, an in vitro diagnostic signal, or a pattern or signal from a signal acquisition system. FDA’s own guidance specifically names ECG waveforms as an example.

• It has to be intended for displaying, analyzing, or printing medical information about a patient, such as literature or guidelines, rather than patient-specific signal data.

• It has to support or recommend, not replace, a clinician’s judgment.

• It has to let the clinician independently review the basis for its output rather than functioning as a black box. FDA’s guidance is blunt that this is difficult for large language models generally and close to impossible in time-critical situations.

Miss any one of these four, and the software is a regulated medical device.

This connects to something I tested myself back in May. I uploaded an ECG to OpenEvidence and received an AI-generated interpretation. That alone looks like a Criterion 1 problem: analyzing a signal from a signal acquisition system is exactly what the exemption excludes. I checked again this week to see whether the feature had changed. It hadn’t. OpenEvidence still accepts ECG image uploads and still generates an interpretation. On the same case I tested in May, it still produced an incorrect but very official-appearing reading.

If OpenEvidence Doesn’t Qualify, What Happens on Each Side?

Assume for a moment that the ECG function alone is enough to knock a product out of Non-Device CDS status. What would actually follow, in the US and in the EU, and does the comparison support “the US is the settled option”?

In the US, there’s already a precedent-setting path for exactly this function. AccurKardia’s AccurECG 2.0 cleared FDA review as a Class II device via 510(k) in January 2026. Tempus received 510(k) clearance for its ECG-Low EF software the same way. If OpenEvidence’s interpretation function is similar enough to an existing cleared device, it would likely follow the same 510(k) route, averaging 155 days as of mid-2026. If it’s different enough that no predicate applies, it would more likely need De Novo classification, averaging 341 days.

In the EU, this same feature would almost certainly count as a moderate-to-higher-risk medical device (what MDR calls Class IIa or IIb), the kind that needs an outside safety review before it can be sold. That review currently takes 13 to 18 months on average. A new rule adopted this May is supposed to shrink that to roughly 6 to 9 months, but only for agreements signed after February 2027, so it doesn't help anyone applying today. And crossing that medical-device threshold doesn't let a company trade one set of rules for another. It automatically pulls the product into the AI Act's high-risk category too, so it ends up answering to both frameworks at once: the device safety review, plus the AI Act's own requirements for data governance, incident reporting, and built-in human oversight.

On the two sides of the Atlantic, there is a real structural difference, not manufactured uncertainty.

Two More Differences Worth Examining

Data protection is not the same question as AI regulation, and the two get blurred together in most coverage of this story. OpenEvidence’s own privacy policy tells EU users not to use the product because its infrastructure is US-based and isn’t governed by EU safeguards. GDPR treats health data as a special category under Article 9, requiring explicit consent or a documented research basis before it can be processed at all, a stricter bar than HIPAA’s authorization-or-de-identification model.

The EU and the UK are also not the same problem, though OpenEvidence’s notice treats them as one. The EU has written binding rules that are simply demanding and expensive to satisfy. The UK, by contrast, has no binding AI-specific medical device framework at all yet. The MHRA’s version is still a voluntary pilot program, with a real framework promised sometime later in 2026. “Uncertainty” is the accurate word for the UK’s situation. For the EU, the accurate words are “burden” and “cost.”

What OpenEvidence Actually Did

None of this required OpenEvidence to leave in the specific week it left, and the run-up to its exit is worth walking through.

By the time OpenEvidence posted its notice on April 27, the relief it was asking for was already in motion and had been for months, none of it because of anything OpenEvidence did. EU officials had proposed a package of changes back in November 2025, five months earlier, specifically to ease the same high-risk AI rules OpenEvidence's notice pointed to. The EU's parliament had already voted, 569 to 45, to move that package forward on March 26, a full month before OpenEvidence acted, and the different EU bodies involved had already started hammering out the details. The one round of those talks that hit a snag, on April 28, the day after OpenEvidence's notice went up, got stuck on a technical point that had nothing to do with the kind of product OpenEvidence makes. OpenEvidence left in the middle of a process that was already headed toward the outcome it said it needed, on a schedule set months before the company made its decision.

Its own notice is worth reading closely too. The original version named two people, Brando Benifei and Michael McNamara, and invited clinicians to contact them along with CPME, the umbrella body for European medical associations. Benifei and McNamara aren’t random picks. They co-chair Parliament’s actual AI Act oversight working group, and McNamara went on to become the Rapporteur for the exact Omnibus package that delivered the deferral. CPME, for its part, had already co-signed letters in March and April arguing to keep the AI Act strict, the opposite of what OpenEvidence needed. That callout disappeared from the notice within 48 hours. A correspondence published in the Lancet in May read the episode as a scrubbed attempt at bottom-up lobbying, notable for what it suggests about the gap between OpenEvidence’s stated rationale and its own actions, regardless of what the callout was intended to accomplish.

And the technology itself never actually left. Although OpenEvidence’s own site still shows the same notice to anyone in the EU, Elsevier’s ClinicalKey AI, running on OpenEvidence’s own engine per their 2023 partnership and featuring The Lancet as a headline content source, is available to EU subscribers. The AI didn’t leave Europe. It just stopped answering to OpenEvidence’s name.

So, Why Did OpenEvidence Really Leave?

Probably not because the EU is uncertain. The EU’s rules are written down, phased in on a public schedule, and expensive to satisfy, which is a different problem than uncertainty and a more honest one for OpenEvidence to have named. The timeline it cited resolved in a direction it didn’t need to lobby for. And its own underlying technology kept working in the exact market OpenEvidence says it can no longer serve, just under someone else’s name.

The clinicians who lost access to OpenEvidence didn’t get any of that. They got one sentence, a since-deleted prompt to contact officials who didn’t need the push, and a tool that, as best I can tell, may already be operating past the line the FDA draws around what doesn’t need to be regulated at all.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

On July 10, 2026, OpenEvidence launched EvidenceGrade, a feature that assigns a letter grade to the strength of evidence beneath every AI-generated clinical answer. It’s an improvement in the value of the responses OpenEvidence creates, and it lands in a market where other tools have been grading evidence in multiple different ways for quite some time.

What EvidenceGrade Does

EvidenceGrade builds on GRADE, the Grading of Recommendations, Assessment, Development, and Evaluation framework behind Cochrane reviews, WHO guidance, and most major clinical practice guidelines. GRADE has run evidence-based medicine for two decades. What’s new is speed, not the underlying idea of an overall grade. OpenEvidence computes a single grade for the whole answer live, at the moment a clinician asks a question, a step that has previously required human editors working ahead of time.

When a clinician asks OpenEvidence a question, the system first runs a classification step, checking whether the question amounts to a clear evidentiary claim. Simple definition lookups and summarization requests get filtered out here. Questions that pass get scored on study design, consistency of findings, precision, and directness- the same inputs GRADE has always used- and the system returns a letter grade for strength, with a separate “U” designation for evidence that can’t be graded.

Two details are worth noting. OpenEvidence’s classifier decides which questions get graded, not the clinician. And the grading itself comes from a single company applying GRADE algorithmically to its own retrieval results, a different process than GRADE’s original consensus-panel model.

Five Products, Five Methods

OE’s EvidenceGrade scores literature live, at the moment a clinician asks a question. Scoring extends to questions that no formal appraisal has reviewed yet, and the tool runs at point-of-care speed.

UpToDate’s Expert AI launched in late 2025 and sits on top of content that physician editors already reviewed and graded using GRADE, with Grade 1 or 2 for recommendation strength and Grade A, B, or C for evidence quality. The AI layer surfaces existing editorial work conversationally, with rationale and links back to the original topics. Scoring depends on what editors have written and reviewed, so very recent literature can lag what OpenEvidence pulls in real time.

DynaMed shares UpToDate’s editorial foundation: physician editors review and rate evidence with GRADE ahead of time. The method diverges at synthesis. DynaMed’s AI layer rates the individual sources it cites but doesn’t roll them up into a single grade for the answer as a whole, the way UpToDate’s per-recommendation grade or OpenEvidence’s per-answer letter does.

Vera Health also grades the strength of underlying evidence behind its answers and positions itself as a direct point-of-care competitor to OpenEvidence. That grading applies to the individual sources it cites, the same source-level approach DynaMed’s AI takes, rather than producing one overall grade for the answer.

Consensus takes a different approach. Its default ranking sorts papers by citation count, journal reputation, and recency, all signals of a paper’s influence and reach rather than the rigor of its methodology. Consensus also offers an optional evidence-hierarchy filter that lets a clinician sort by study design, from meta-analyses and systematic reviews down through RCTs and observational studies. Nothing is graded automatically here; the clinician chooses whether to apply the filter every time.

Pros and Cons

OpenEvidence offers speed and reach. A grade can appear under a question no editor has ever reviewed, in the time a clinician has between patients. That speed comes from a single company’s automated interpretation of GRADE, run through a classifier a clinician can’t inspect.

UpToDate carries the weight of editorial review behind every grade, built up over years with an established audit trail. Questions about very recent literature can outrun what the editorial team has reached, so timeliness suffers where OpenEvidence’s live scoring holds an advantage.

DynaMed shares UpToDate’s tradeoff of editorial rigor against editorial pace. Its AI layer grades each source it cites, but leaves the work of weighing those individual grades into one overall judgment to the clinician, since it doesn’t generate the single per-recommendation grade UpToDate does.

Vera Health competes on benchmark performance and broader geographic reach and on a grading methodology it hasn’t named as clearly as its competitors have. It also stops at source-level grades rather than synthesizing one for the answer, so a clinician still has to weigh conflicting source grades themselves. A clinician outside the US or UK who lost access to OpenEvidence may find Vera Health the more practical option.

Consensus puts control directly in the clinician’s hands. No classifier decides what gets graded, and every AI-generated summary traces back to an exact sentence in the source paper. That control depends on the clinician knowing to apply the hierarchy filter; skip it, and the ranking on screen reflects citation counts and journal reputation, which are signals of popularity rather than rigor. Consensus also serves general research across all fields, so it lacks the clinical workflow tuning built into the other four tools.

When to Actually Use Which

A fast, evidentiary bedside question, something like whether a drug reduces mortality in a given population, fits OpenEvidence well. Watch for whether a banner appears at all; no banner means the classifier judged the question outside EvidenceGrade’s scope, which is worth a second look if the question felt evidentiary to you.

A well-established clinical question or standard management of a common chronic condition plays to UpToDate or DynaMed’s strength. Human editorial review has already happened, and the subscription cost buys that vetting.

A clinician outside the US or UK, or one weighing the benchmark claims directly, has reason to look at Vera Health alongside or instead of OpenEvidence, keeping in mind that the comparative claims come from Vera Health itself.

Deep literature review on an emerging or contested question, or research headed into a paper or grant application, calls for Consensus, used deliberately with the hierarchy filter engaged and the full-text tool open to check claims directly.

Many everyday questions have no clear winner among these five. When UpToDate already holds a well-established Grade A recommendation for a common condition, OpenEvidence’s live grade on the same question will likely land in the same place, since both draw on the same underlying evidence base. The choice there comes down to workflow preference, editorial certainty against conversational speed. A clinician willing to spend an extra ninety seconds applying Consensus’s RCT filter can reach a similar result to what EvidenceGrade delivers automatically.

Risk and Bias, In Both Directions

The editorial model behind UpToDate and DynaMed carries its own risks. A recommendation can sit unchanged for months after new evidence complicates it, and every grading decision reflects the judgment of the specific editors who wrote it, not an infallible panel.

OpenEvidence’s automated model carries its own risk. A letter grade can read as more authoritative than a live scoring process actually supports, which invites automation bias. The classification step still makes a judgment call the clinician doesn’t get to weigh in on: whether a question counts as evidentiary at all. Grading quality also depends entirely on what the retrieval system pulled in the first place, a step a clinician can’t audit in real time.

DynaMed’s AI and Vera Health carry a different risk, tied to the same source-level grading described above. Neither tool synthesizes the individual source grades into one judgment, which means the aggregation OpenEvidence and UpToDate perform automatically becomes a manual step for a clinician moving quickly between patients, and manual synthesis done under time pressure is where inconsistency creeps in.

Consensus shifts risk toward the clinician directly. Someone who doesn’t know to apply the evidence-hierarchy filter, or who doesn’t recognize that the default ranking measures popularity rather than rigor, gets no protection from the tool itself.

Each model fails in its own way, and the failure a clinician is exposed to depends on which tool they picked and how well they understand what runs underneath it.

The Question Nobody Has Answered: Liability

Clinicians are not attorneys, and nothing here is legal advice. It’s worth asking plainly, since none of the marketing for any of these five products addresses it directly: Does a clinician carry more or less liability for relying on an AI-generated evidence grade compared to a human-editor-vetted one?

I could not find an answer to this question. Malpractice and negligence standards generally turn on whether a clinician exercised reasonable judgment consistent with the standard of care, and historically that inquiry centers on the clinician’s own decision rather than the reference tool behind it. All five products here position themselves as clinical decision support rather than autonomous diagnostic tools, which keeps the clinician as the final decision-maker of record, the same SaMD-versus-CDS framing this newsletter has examined before.

Whether courts will eventually treat a real-time AI grade differently from an incremental human editorial grade remains genuinely untested. I don’t have an answer, and anyone claiming otherwise is speculating.

What holds steady across all five tools is this: whatever grade appears on the screen, the decision that follows still belongs to the clinician who acts on it.

As always, if you enjoy reading, please subscribe and tell a friend.

Sam

When a clinician has a question during an encounter, the workflow is straightforward. They recognize the need for evidence. They leave the patient, open a search tool like OpenEvidence, UpToDate, AMBOSS, or Doximity, formulate a query, and try to find an answer. Then they return to the chart and translate that finding into a plan.

It takes time and requires cognitive effort. It also interrupts the clinical workflow.

What if that entire sequence could be automated by pairing two things that currently operate independently: a scribe that captures context and an AI search tool that needs context?

If it works the way vendors are beginning to market it, the implications could be substantial.

The Manual Search Problem

Clinical decision support tools are valuable. The problem is that clinicians rarely use them consistently.

Adoption of standalone clinical decision support (CDS) has hovered between 20 and 50% for more than a decade. When adoption rates exceed 50% in any health system, it’s usually because the tool was so deeply embedded in workflow that using it became invisible. Not a conscious decision to “search for evidence,” but a byproduct of how documentation already worked.

The Agency for Healthcare Research and Quality (AHRQ)‘s evidence on clinical decision support has long established this pattern. In a 2005 study on clinical information needs, Ely and colleagues found that the most common reason physicians did not pursue answers to clinical questions was time. Not having access to tools, or skepticism about the instrument. Every additional click, every additional screen, every additional cognitive step to formulate what you’re searching for, all of that is friction.

The manual prompt introduces friction at multiple points.

First, the clinician has to recognize that they need help. That alone is cognitively taxing when you’re in the middle of an encounter and managing multiple competing hypotheses simultaneously.

Second, they have to formulate what they’re searching for. They have to decide what context matters. For example, a 65-year-old woman with a subsegmental PE is the bare minimum. But the clinically relevant narrative includes much more: her main symptom was chest pain, her father died of an MI at 65, she has no dyspnea despite the imaging findings, she’s anxious about the PE diagnosis. The question is whether the clinician remembers or prioritizes all of that when they type a prompt. Often they don’t. They reduce it to what feels immediately salient.

Third, they have to type it. And then translate the results back into their documentation.

Each of these steps is a place where information can be lost. Each is a place where a busy clinician can decide it’s faster to just make a decision than to go through the process.

The result: CDS adoption stalls. Not because the tools don’t work, but because the friction cost exceeds the perceived benefit.

What If Context Was Automatic?

Now consider a different architecture.

A scribe is already listening to the encounter. It’s capturing everything: chief complaint, associated symptoms, what improved with aspirin, the patient’s expressed anxiety about the PE finding, the absence of dyspnea despite imaging, family history, the nuance of how the history actually unfolded. The scribe structures that into a clinical note.

What if that same structured encounter context became the automatic input to downstream AI search and clinical decision support?

The difference is subtle but consequential. Instead of waiting for the clinician to recognize they need help and formulate a prompt, the system works from what’s already been captured. The context is automatically available. The search doesn’t depend on the clinician deciding what matters or remembering to type it.

This is the architectural shift vendors are now building.

Glass Health combines ambient scribe with a clinical reasoning layer that accesses the encounter transcript. When the scribe finishes capturing the visit, the same data becomes context for CDS reasoning, not a separate step, not a separate prompt, but the natural downstream use of data already captured.

Microsoft’s DAX Copilot generates the note from the scribe, then uses that same encounter context to surface order suggestions within Epic. The clinician doesn’t have to prompt for order ideas; they arrive pre-staged, informed by what was actually discussed.

Abridge, after expanding into real-time prior authorization through an Availity partnership announced in January 2026, is pre-populating prior auth requests with encounter context. The clinician reviews, not writes.

Ambience Healthcare’s AutoScribe provides real-time coding suggestions, quality measure tracking, and automated prior authorization recommendations, all triggered by the scribe context, all without requiring the clinician to initiate a separate search.

In each case, the pattern is the same: scribe captures context automatically, downstream systems act on that context without waiting for the clinician to formulate a prompt.

Why This Matters

If CDS adoption has been stuck at 20-50% because clinicians won’t use tools that require extra steps, and if integrated scribe-plus-search systems eliminate that friction by making context automatic and routing recommendations directly into workflow, then adoption could shift.

Not because the search tools suddenly became better. But because the context became richer, the friction resolved, and the recommendations arrived where clinicians are already working.

There’s a secondary automation gain as well. Once encounter context is structured and available, systems downstream can act without additional clinician prompts. Abridge is already doing this with order placement. The prior authorization request comes pre-populated. The labs or imaging recommendations arrive as suggestions for review, not as questions requiring the clinician to type more information.

Each of these automations removes a decision point. Each removes a place where context can be lost, or a clinician can opt out because the friction became too high.

This is different than asking “are AI search tools more accurate when they have more information?” This is asking whether integrated scribe-plus-search systems could achieve the adoption curves and utilization patterns that standalone CDS have chased unsuccessfully for years.

Where This Is Already Shipping

The market is moving in this direction. The examples above are live products, deployed now.

• OpenEvidence, Doximity, and Glass Health offer scribe plus clinical decision support in the same interface.

• DAX Copilot’s order suggestions are live within Epic.

• Abridge’s prior authorization workflow doesn’t require the clinician to re-enter what was already discussed with the patient; the scribe context carries it forward.

• Athenahealth made its ambient scribe free to all customers in February 2026. The company has explicitly positioned the scribe as a foundation for downstream clinical and administrative workflows.

These aren’t experimental pilots. They’re market moves. Multiple vendors are racing to integrate scribe plus CDS, scribe plus order suggestions, scribe plus prior authorization, and scribe plus billing optimization. The race itself suggests vendors believe integration is where the value is concentrating.

The Evidence Question

AHRQ best practices emphasize that CDS embedded in clinical workflow achieves significantly higher adoption rates than standalone tools.

The Rotenstein et al. study published in JAMA in April 2026 tracked 8,581 clinicians across five academic medical centers and found that 79% of eligible clinicians declined to adopt a scribe when it was offered as a standalone documentation tool. Among those who did adopt, only 32% used it in 50% or more of visits, the threshold where benefits actually accumulated.

But at organizations where the scribe was deliberately integrated into clinical workflow with physician champions, hands-on training, and customization of documentation to match local practice, adoption reached 75-80%.

Central Oklahoma Family Medical Center saw minimal scribe adoption in year 1 when it was positioned as a documentation tool. In year 3, after deliberate workflow integration and physician-led training, the system was generating over 14,000 records annually.

Research supports the pattern: embedded tools are better than standalone tools. Integrated workflows are better than siloed workflows. Friction is the constraint, not technology limits.

However, there is no published comparative adoption study in the same health system measuring scribe-only utilization versus integrated scribe-plus-CDS utilization. The vendors are claiming that integration drives adoption. The friction research supports that claim logically. But direct comparative evidence is absent.

That gap is fixable. It’s also important. If the thesis holds that integrated scribe plus CDS achieves meaningfully higher adoption than scribe-only, that would directly support the “game changer” framing. Until that evidence exists, we’re working from logical inference and market positioning, not real-world data.

What Changes Clinically

Let me be concrete about what this looks like in practice.

Today’s workflow: A 65-year-old woman presents with chest pain, elevated troponin, and a subsegmental PE on imaging. The clinician needs to decide between acute coronary syndrome and a low-risk PE with incidental findings. They recognize they need evidence about ACS risk stratification and PE disposition pathways. They leave the encounter, open OpenEvidence or UpToDate, search for something like “subsegmental PE,” scan results, and return to the chart with an answer. Time cost: 5-10 minutes. Friction cost: context loss between encounter and search.

Tomorrow’s workflow: The same patient, same presentation. The scribe has captured the full encounter: the aspirin response, the specific way the history unfolded, the family history details, the absence of dyspnea. Once the note is drafted, that encounter context automatically feeds into CDS reasoning. Before the clinician even finishes reviewing the note, recommendations about ACS risk stratification appear inline. They’re specific to what was discussed, not to a reductive prompt. The clinician reviews and acts. No extra steps. No context loss.

The difference isn’t speed alone. It’s that the clinical reasoning is informed by what actually happened, not by what the clinician decided was relevant enough to type.

What Has to Be True for This to Work

This only works if several conditions hold.

Scribe accuracy matters more. If the scribe omits a symptom or invents a clinical detail, everything downstream breaks. Recommendations based on false or inaccurate context are worse than no recommendations. The accuracy bar for a scribe that feeds into automation is higher than for a scribe that just generates a note for a clinician to review and edit.

Context routing has to be intelligent. Not every piece of encounter context should trigger every possible recommendation. Signal-to-noise matters. If the system fires off ten recommendations per encounter, most of which are irrelevant, clinician trust collapses. Integration has to include filtering and prioritization logic that decides what context maps to what recommendations.

Integration has to be seamless. If embedding CDS into the scribe workflow adds steps, the friction returns and adoption stalls. The system has to route recommendations directly into existing documentation workflows without requiring the clinician to open new windows, review separate interfaces, or translate between formats.

Liability has to be clear. If a scribe misses a clinical detail, who’s responsible? If CDS gives a recommendation based on scribe context and it’s wrong, who bears the liability: the scribe vendor, the CDS vendor, the clinician, the health system? Until those questions are answered clearly, adoption will be cautious.

Clinicians have to adopt the pattern. The mental model of “don’t prompt, just accept recommendations” is new for many physicians. It requires trust in both the scribe's accuracy and the CDS's reasoning. It requires that clinicians believe recommendations are being triggered appropriately, not indiscriminately. Building that trust takes time.

Conclusion

The architectural logic is sound. Pairing a scribe that captures context with CDS that needs context could eliminate the friction that’s kept adoption low. Integration could shift clinical decision support from a tool clinicians invoke when they have time to a system that informs their thinking automatically.

But “could” and “does” are different things. The market believes in this direction. Vendors are shipping products built on this assumption.

The evidence, though, is still emerging. The comparative adoption study doesn’t exist yet. The real-world error profiles haven’t been published. The liability frameworks are still being negotiated.

What we’re watching is an inflection point. The question isn’t whether AI scribes plus AI search could be a game changer. The architecture is sound enough that it’s plausible. The question is whether the systems being deployed right now will deliver on that promise at scale.

That’s worth paying attention to.

They need comparison data, but there isn’t any. The single largest, most consequential technology decision many U.S. hospitals will make this decade is being made with far less evidence than we would accept for a new drug or device. Let’s dive deeper.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

It’s not a hypothetical problem.

• More than 85% of Epic’s customer base is already using some form of Epic AI

• EHR vendor dependency is cited as the top execution barrier by 74% of health system technology leaders

• A majority of Epic’s customers report spending a quarter of their IT bandwidth simply managing multiple vendor integrations rather than building anything new.

So how are hospitals supposed to decide about Epic’s AI features? Three questions capture the actual decision tree a CIO faces:

1. How do you compare Epic’s tools against alternatives when no published performance data exists?

2. Are you left comparing on price alone?

3. Can you even connect an outside AI tool to Epic, and what does that actually cost in dollars and time?

Each answer turns out to be more complicated than it first appears.

The Comparison Vacuum

According to EPIC, more than 200 organizations use Penny for professional coding, with many seeing 20% of more coding-related denial reductions. Art’s Insights feature is used more than 16 million times a month. At The Christ Hospital, Art’s radiology finding extraction is credited with a 69% early lung cancer detection rate compared to a national average of 46%. At Rush University Medical Center, Emmie delivered a 58% reduction in billing-related customer service messages.

These are genuine wins, not vague marketing language. But notice what they have in common: every one of them is a curated success story that Epic chose to publish, from a customer Epic chose to name, describing an outcome Epic chose to measure. None of them come with a comparison arm. None of them tell you what a third-party tool, or no AI tool at all, would have achieved at the same hospital over the same period.

What doesn’t exist is any neutral, third-party study putting Epic’s tools head-to-head against Abridge, Ambience, Nabla, Suki, or Oracle Health’s Clinical AI Agent, on the same patients, measuring the same things. The closest proxies available each fall short in a specific, disqualifying way:

• KLAS “Best in KLAS” rankings exist for ambient AI and clinical documentation, and vendors like Abridge and Ambience have won them in 2025 and 2026. But KLAS rankings are built from customer satisfaction surveys. They measure whether clinicians like a tool, not whether it performs more accurately or safely than the alternative.

• Vendor comparison content, like the “Abridge vs. Ambience” writeups that circulate among health IT consultants, reads like independent analysis but is typically produced by resellers and integration firms with a commercial stake in one outcome or another.

To be fair to Epic, native integration solves a real, well-documented problem. Third-party AI tools often exist as a separate app or window that a clinician has to switch into, breaking their workflow. Epic’s tools, by contrast, draw on the patient’s complete longitudinal record natively, inside the same interface the clinician already lives in. That is legitimate integration depth, and Epic has a structural claim to it that outside vendors have to work much harder to match.

But “we’re already inside your workflow, so trust us” is a switching-cost argument, not a quality argument. It explains why Epic’s tools are more convenient to adopt. It says nothing about whether Art’s diagnostic suggestions, Emmie’s patient explanations, or Penny’s coding recommendations are more accurate than a competitor’s because nobody has measured that, and Epic has no more incentive to fund that comparison than a third-party vendor would have to fund one showing Epic winning.

The honest, unsatisfying answer for how hospitals resolve this: they don’t rely on external comparison data. They run their own limited pilots of both options against their own patient population and staff, because nobody else has done, or has an incentive to do, that comparison for them. That’s not a failure of any individual CIO’s diligence. It’s a structural gap in the market that every hospital is left to close on its own, one expensive pilot at a time.

Price Is Visible. Quality Isn’t.

No CIO is literally forced to decide on price alone. But price is the one variable in this decision with any real transparency, and that asymmetry quietly warps the whole comparison because when quality is unmeasurable and price is a line item, price tends to win by default, whether or not it should.

The other differentiators hospitals actually weigh are real, but none of them are clinical outcome measures:

• Integration depth. Does the tool live inside Epic natively, or does it require clinicians to leave the EHR? This is the “house vs. guest in the house” distinction discussed above. It’s a genuine operational factor, not evidence of accuracy or safety.

• Specialty coverage breadth. Some third-party tools market coverage across 200-plus specialties, from oncology to emergency medicine. That’s a real capability difference, but it says nothing about performance within any single specialty.

• Vendor relationship tier with Epic. Some vendors, like Abridge and Nuance, hold a spot in Epic’s invite-only “Workshop” co-development tier. Others sit in the more generic, self-service “Connection Hub” listing. This tells you something about durability and long-term support risk. It tells you nothing about whether the underlying AI is any good.

• Lock-in and commoditization risk. As Epic rolls out “good enough” native alternatives, standalone point solutions face real competitive pressure. Betting on a third-party tool means betting that it survives Epic’s gravitational pull.

The uncomfortable throughline across all four: they’re operational and strategic risk proxies, not evidence. A CIO weighing them isn’t answering “which AI is better”. They’re answering “which AI is safer to bet on organizationally.”

Yes, You Can Connect Outside AI to Epic

Epic is not a walled garden on this point. The technical path to connecting an outside AI tool, whether a commercial product or a custom build on an API, is real, standardized, and already proven in production.

The mechanisms, at the level a CIO actually needs:

• SMART on FHIR is an open industry standard, not an Epic-proprietary one, that lets a third-party application launch inside Epic’s interface (Hyperspace, Hyperdrive, or MyChart) with OAuth2-based access to the patient’s clinical context.

• CDS Hooks is a lighter-weight alternative for narrower use cases: an external tool fires in real time on a specific clinical event (opening a chart, signing an order) and returns a recommendation card, without the overhead of launching a full separate application.

• Epic Showroom (formerly App Orchard) is the marketplace and certification pathway, with three tiers:

  • Connection Hub, a basic listing starting around $500 a year, where most third-party apps live.

  • Toolbox, offering curated visibility

  • Workshop, reserved for vendors Epic is actively co-developing with, not a tier you can apply for.

Abridge, Ambience, Nabla, and Suki all run as third-party ambient scribes launched inside Epic today via exactly this pathway. Connecting a custom AI build is well-trodden ground, not an experimental frontier.

What it actually costs, in dollars:

• A basic FHIR read integration for a first site: $15,000–$40,000

• Bidirectional integration (the tool can write back into the chart, not just read from it): $40,000–$80,000

• Each additional hospital site: $5,000–$20,000 in configuration, testing, and governance overhead

• Enterprise multi-site deployments (five-plus hospitals): $100,000–$500,000+

• Annual maintenance: 15–20% of build cost, ongoing

Those figures answer the technical and financial questions. They don’t answer the harder question, which is where most integration projects actually stall.

“Integration Is 20% Development, 80% Negotiation”

Brendan Keeler, a widely cited EHR integration analyst, has a line that captures this better than any cost table: integration is 20% development, 80% negotiation.

It’s helpful sitting thinking about what that 80% actually is, because it’s several negotiations stacked, and none of them get easier just because a vendor has already done this at another hospital:

• Per-site IT governance review. Even though Epic’s FHIR APIs are standardized across every customer, each hospital runs its own instance with its own security policies. A vendor doesn’t clear “Epic” once. They clear each hospital’s IT and security team separately, every time.

• Scope and access-level negotiation. What data can the tool read? Can it write back into the chart? Is access tied to individual user logins or a backend service account? These aren’t just technical settings. They’re negotiated line by line with each hospital’s compliance and security staff, who have their own risk tolerance and institutional precedent to defend.

• Contract terms with Epic itself, separate from any single hospital deal: Showroom tier, vendor services registration, fees.

• Institutional change management: Getting clinical leadership, IT, compliance, and often legal to actually agree to put a new tool inside their clinicians’ workflow.

This negotiation stack isn’t identical across hospitals, even though the underlying FHIR technology may be. Each hospital is, in effect, a separate legal and organizational gate a vendor has to clear from scratch, no matter how many times they’ve cleared an equivalent gate elsewhere. That mismatch is exactly why Epic’s own native tools carry a durable structural advantage that has nothing to do with whether their AI is actually better. They skip the entire negotiation stack because they’re already inside the walls, already covered by the hospital’s existing Epic contract and security posture.

How Vendors Are Fighting Back

Third-party vendors aren’t standing still against this asymmetry. Several strategies have emerged, each attacking a different piece of the 80%:

• Pre-certification as a trust shortcut. HITRUST certification (including a new AI-specific security certification launched in response to healthcare’s AI boom) and SOC 2 Type II audits let a vendor prove its security posture once and present that proof to every prospective hospital, rather than rebuilding trust from zero at each site. HITRUST is now often a baseline requirement for even getting considered in procurement.

• Integration aggregators. Platforms like Redox normalize FHIR, HL7 v2, and proprietary EHR APIs into a single interface, letting a vendor avoid building direct integrations against three to five different EHR systems from scratch, at the cost of some data fidelity and an extra hop in the data flow.

• Design-partner-first rollout. Land one engaged hospital willing to work through the full registration and security review cycle together, get the integration proven in production, then use that as a template for the second and third hospital, accepting that some per-site variability will still surface each time.

• Climbing into Epic’s Workshop tier. Vendors like Abridge and Nuance have secured Epic’s invite-only co-development relationship, which functions as Epic effectively vouching for them. It’s a different negotiation dynamic from approaching each hospital cold.

• Leaning on CDS Hooks for narrower use cases, trading a smaller interaction surface for meaningfully lower integration complexity.

• Regulatory tailwinds. The 21st Century Cures Act’s information-blocking provisions and ONC’s HTI-1 rule, which raised the certification baseline to USCDI v3 as of January 2026, give vendors a legally backed claim to FHIR access, shifting some leverage away from ad hoc hospital IT gatekeeping.

None of this eliminates the negotiation. It compresses or front-loads it. Bidirectional write-back (actually pushing AI-generated content into the medical record, not just reading from it) remains the hardest and slowest part of any integration, regardless of how much pre-certification a vendor obtains.

What CIOs Actually Have to Work With

A few genuinely useful resources exist for the leader trying to navigate this, though none of them is the comprehensive primer this decision deserves.

The Health Sector Coordinating Council’s “Health Industry AI Cyber Governance Framework Implementation Guide” (May 2026) is the closest thing to an authoritative, non-commercial reference, covering governance committee structure scaled to hospital size, escalation authority across CMO, CMIO, and Privacy Officer roles, and a benefit-risk framework tied explicitly to FDA regulatory status.

Qventus’s “Beyond the Pilot” survey of more than 60 CIOs, Chief AI Officers, and CMIOs offers real peer benchmarking: 74% cite EHR vendor dependency as their top execution barrier; 72% say they’d prefer a single consolidated AI partner over a fragmented multi-vendor stack, but only 13% have actually achieved that consolidation. It’s vendor-published, so it should be read as useful data with a thumb on the scale, not neutral research.

Narrower academic frameworks exist too, like peer-reviewed pragmatic-trial protocols for evaluating ambient AI specifically, which bring real methodological rigor but only to one slice of the larger decision.

What none of these does is tie the whole picture together. A CIO today has to assemble governance policy from one source, vendor economics from a consulting firm’s blog, integration cost data from an engineering guide, and validation-status skepticism from wherever they can find it, with no neutral party doing that synthesis for them. That absence is, itself, worth naming plainly: the market has produced plenty of pieces but no assembled whole.

Takeaway

Individual hospitals cannot solve this problem on their own. It is a fundamental issue with the entire healthcare market, which tends to favor established players, like Epic, over newer or better alternatives, simply because they already hold the power.

A few things worth holding onto if you’re the one making this call:

1. Don’t mistake Epic’s adoption percentages for comparative quality evidence. They are usage statistics, curated by the company reporting them. They tell you Epic AI is widely used. They don’t tell you it’s the best option.

2. Budget for the 80%, not just the 20%. The dollar figures for FHIR integration are real, but the calendar time and staff bandwidth consumed by per-site negotiation are the actual cost drivers, and they’re the ones most commonly missing from a project’s original scope.

3. Ask any vendor directly what Showroom tier they hold. It won’t tell you if their AI is accurate, but it will tell you something real about how durable their access is likely to be.

4. Treat the absence of comparative data as a mandate to pilot, not a reason to default to the incumbent. Nobody else is going to run that comparison for you. That doesn’t mean it isn’t worth running.

No one selling you this technology is going to prove it works for you. That job still belongs to the hospital buying it.

Meanwhile, nurses in New York, California, Michigan, and North Carolina have been striking and writing AI oversight rights into union contracts. The national nursing union organization that represents those same nurses is publishing its own AI bill of rights.

Both responses exist because the frameworks and the union contracts are answering different questions. Understanding why requires looking at which AI tools are actually being deployed, to whom, and at whose direction.

As always, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

What the Professional Associations Agree On

The American Medical Association published formal AI principles in November 2023, adopted a transparency policy calling for independent third-party verification of AI explainability in June 2025, and released an 8-step governance toolkit in August 2025. The toolkit calls for multidisciplinary governance committees that include nursing leadership, staged deployment with pilot testing, and continuous performance monitoring after launch.

The American Nurses Association convened its inaugural AI in Nursing Practice Think Tank on April 22, 2026. The consensus findings identify automation bias and erosion of professional judgment as the primary risks, call for mandatory AI literacy as a core nursing competency, and push for nurse-led AI governance at the institutional level.

The American Academy of Nursing approved a comprehensive AI position statement on February 25, 2026, setting out 13 specific policy recommendations covering data privacy, algorithmic bias, FDA oversight, and human-in-the-loop oversight standards across all institutional governance policies.

The foundational language across all three documents is nearly identical: AI must augment clinical judgment, not replace it. The human clinician remains the final accountable decision-maker. Human-in-the-loop oversight is non-negotiable.

All three frameworks rely on the same implementation mechanism: voluntary institutional governance, multidisciplinary committees, professional education, and policy advocacy. None specifies what “human in the loop” requires at the point of care. None has an enforcement mechanism at the bedside level.

Two Categories of AI Tools

The governance frameworks were designed for a particular relationship between clinician and AI: the clinician chooses to use a tool, reviews its output, and retains authority over the final decision. The ambient scribe is the clearest example. The physician activates it, the scribe drafts a note, the physician reviews and signs. The tool serves the physician. Clinical judgment stays with the clinician throughout.

Physician AI adoption fits this model closely. By 2026, more than 80% of physicians report using AI in their professional work, with more than three-quarters saying it improves their ability to care for patients. The tools driving that adoption are documentation-focused: 70% of physicians at UCSF now use AI scribes daily, and Kaiser Permanente logged more than 2.5 million AI-scribed encounters over 14 months. Randomized trial evidence confirms that these tools reduce documentation time and burnout scores.

The AI tools most commonly deployed in nursing workflows belong to a different category. Patient acuity scoring algorithms analyze EHR data to determine how sick a patient is and predict how many nursing hours that patient requires. Staffing algorithms use those acuity scores to determine how many nurses are called into a shift. Automated handoff tools generate shift reports. Clinical deterioration algorithms produce alerts. These tools are not activated by the nurse. They are deployed by hospital administration, they run continuously, and their outputs directly constrain what nurses do and how many patients they are assigned.

The NNU’s 2024 survey of 2,300 members documented what this looks like in practice. Half of respondents said their employer uses algorithmic systems to determine patient acuity and predict required nursing hours. Of those, 69% said the AI-generated acuity score did not match their own clinical assessment. Among nurses whose employers use automated handoff tools, 48% said those AI-generated reports contradicted their own patient assessment.

The Override Problem

Forty percent of nurses in hospitals using algorithmic patient outcome tools said they cannot override the algorithm’s prediction when their clinical assessment differs. Twenty-nine percent said they cannot alter algorithm-produced wound or pain documentation in the EHR even when they believe it is inaccurate.

Every professional association framework, including the ANA, AAN, and AMA documents described above, establishes that clinical judgment must prevail over AI outputs. The survey data documents a significant gap between that principle and what is happening at the bedside, and the governance frameworks have no mechanism to close it.

The gap has a structural explanation. The patient acuity and staffing tools that nurses cannot override were not purchased by clinical leadership. They were sold to hospital CFOs and operations teams on the explicit promise of reducing labor costs. Vendor marketing for these systems is direct: AI staffing tools minimize overtime, reduce reliance on expensive agency nurses, and cut contract-labor dependency. Ascension Health reported reducing contract-labor dependency by 15%within six months of implementing a predictive staffing system.

A nurse who can override her acuity score is a nurse who can force additional staffing. Override capability undermines the tool’s core value to its actual customer. The inability to override is not a design flaw. It is, from the purchaser’s perspective, a feature.

Many of these operational tools, including staffing algorithms and acuity scoring systems, also do not meet the FDA’s current definition of a medical device. They face no premarket safety review requirement. No regulator required the vendor to build in an override function, and no regulator currently enforces one.

Presence Without Agency

The academic literature has recently begun to examine what “human in the loop” actually means in practice. A paper in the American Journal of Bioethics specifically flags that researchers and institutions routinely invoke HITL as ethical legitimacy without specifying which humans, in which processes, are doing what, and cites NNU survey data on nurses unable to override algorithmic predictions as a concrete example.

The Institute for Systems Integrity published a related framework in May 2026, distinguishing between two states that are often conflated. In the first, a human is present near the AI system: they are notified, they see the output, and they are in the loop in the awareness sense. In the second, a human has agency: they can interrupt the system, modify its output, or substitute their own judgment without penalty. The institute’s framing is clear:

“A human placed near an AI system is not automatically a safeguard. A clinician asked to approve a recommendation under time pressure, incomplete information, workload overload, and unclear authority may not be exercising judgment.”

The EU AI Act makes this distinction legally operational for high-risk AI systems. Article 14 requires that humans be able to interrupt or override the system’s operation and decide not to use it in a specific situation. This is described as an architectural requirement, not a procedural right. No equivalent US requirement exists for hospital operational AI tools.

A nurse who receives an acuity score that cannot be changed has been notified. That nurse has not been given authority.

What The NNU Is Doing

The National Nurses United published a Nurses and Patients’ Bill of Rights in April 2024. Seven rights are enumerated. Most coverage of this document focuses on Right 7, which demands pre-deployment bargaining rights: the right to negotiate over whether and how AI is implemented before the system is selected.

Right 5 is less examined and more operationally significant given the override data. It establishes the right of nurses to exercise professional judgment and override AI decisions without threat of discipline or discharge. No professional association framework, from the ANA, AAN, or AMA, contains an equivalent enforceable protection. Right 5 is the only document in the current governance landscape that directly addresses what 40% of nurses report experiencing.

The NNU has coordinated a national bargaining campaign through its affiliate network: the California Nurses Association, the New York State Nurses Association, the Michigan Nurses Association, and the National Nurses Organizing Committee in North Carolina. Announced AI contract language has been reported at Mission Hospital in Asheville, North Carolina (2024), Northwell South Shore University Hospital in New York, the New York City hospital systems including NYP, Montefiore, and Mount Sinai following a strike by nearly 15,000 nurses in February 2026, Munson Medical Center in Traverse City, Michigan, and the University of California system, where a contract ratified in November 2025 specifies that nurses play a central role in selecting, designing, and validating new technology including AI systems.

One caveat applies to all of the above. No journalist or outlet has quoted actual contract clause text from any of these agreements. Coverage consistently describes outcomes in terms of “safeguards,” “guardrails,” “approval before deployment,” and “voice in how AI is rolled out.” Whether the ratified language reflects the specificity of the Bill of Rights, including Right 5’s override protection, or represents a narrower pre-deployment consultation right, is unknown without the contract documents.

The physician union equivalent, the Union of American Physicians and Dentists, has identified AI as a bargaining priority, with the organization’s president stating in December 2025 that it is “imperative” that the union engage in dialogue with employers to prevent professional judgment substitution by AI. UAPD has not yet produced ratified AI contract language. That gap is approximately 18 months, which is consistent with the tool asymmetry: physicians have not yet encountered operational AI deployed on them at scale without override rights.

Two Tracks, One Unresolved Problem

The landscape across both professions is now parallel. Both nursing and medicine have a professional association track and a union track responding to AI.

The ANA and AAN are doing what the AMA is doing: publishing principles, developing governance frameworks, and advocating for policy. The difference in urgency between the nursing professional associations and their physician counterparts reflects the difference in tool adoption rates and trust levels, not a fundamental strategic divergence.

NNU is doing something the professional associations can’t do: creating enforceable bedside protections through contract law. They are addressing different layers of the same problem. A hospital can have an excellent AI governance committee and still deploy an acuity algorithm with no override function. A union contract can protect override rights but cannot prevent a poorly validated tool from being purchased in the first place. Both layers are needed.

Neither layer has resolved the problem of definition at the center of all of this. Every framework, every position statement, every contract announcement invokes human oversight as a governing principle. No one has established that human oversight requires the ability to override. Until that definition is settled, “human in the loop” describes a spectrum that runs from a nurse with full authority to substitute her/his clinical judgment to a nurse who receives a notification she/he cannot act on. Both nurses are, technically, in the loop.

Questions Worth Asking

If you are responsible for AI governance at a hospital or health system, the relevant question is whether your governance committee’s approval process specifies override capability as an architectural requirement of deployment instead of a recommendation. A tool that generates outputs clinicians cannot modify does not meet the HITL standard described by the ANA, AAN, or AMA, regardless of what the vendor’s documentation says.

If you are a nurse in a facility with a union contract that includes AI language, the clause text matters more than the press release. “Safeguards” and “guardrails” are not contract language. Right 5 of the NNU Bill of Rights, protection of the right to override without threat of discipline, is the protection most directly supported by the survey data. Whether it appears in your contract is a question worth answering.

If you are a nurse in a non-unionized facility, the ANA Think Tank consensus document and the AAN position statement establish a national professional standard you can cite through shared governance structures or unit councils when raising concerns about tools that cannot be overridden.

If you are a physician, the operational AI category that nursing is responding to is not exclusive to nursing. The tools that constrain clinical judgment without requiring clinical input in procurement are already present in prior authorization, clinical documentation, and diagnostic triage. The UAPD’s December 2025 statement suggests that physician union organizing around AI is overdue.

The governance frameworks say clinical judgment must prevail. The mechanism that makes that principle enforceable at 3am, when an acuity score determines whether a second nurse comes to the floor, does not yet exist in voluntary guidelines.

Meanwhile, if you enjoy reading, subscribe and tell a friend.

Sam

Subscribe now

What the media is saying.

The narrative told by the WSJ Pro, MedCity News, and multiple others goes something like this: two Stanford physicians, Sharif Vakili and Ashwin Nayak, ran a rigorous randomized controlled trial proving an AI system could autonomously manage insulin titration in type 2 diabetic patients. The results were dramatic. The AI group hit glycemic control targets in 15 days on average. Fewer than half the standard care group got there at all within eight weeks. Result: 81% controlled by the AI versus 25% by standard practice.

They then built UpDoc on that clinical foundation, got it FDA-cleared as a Software as a Medical Device, and are deploying what the company calls “physician-grade agentic AI” to autonomously adjust insulin doses between office visits. They claim it is the first of its kind, ushering in a new era of care.

That’s the story, but there is a lot of nuance to be discussed.

What the study actually tested

The MIVA trial (Managing Insulin with Voice AI) was a real RCT, published in JAMA Network Open, and conducted at four Stanford primary care clinics from March 2021 to December 2022. Thirty-two adults with type 2 diabetes were randomized into two groups. Half got the voice AI intervention; half got standard care. The results were genuinely strong for a pilot of that size.

Here’s what the coverage consistently leaves out: the voice AI in that trial was Amazon Alexa. Not a generative model, and nothing resembling what most people picture when they hear “agentic AI.” Alexa in 2021 was a narrow, deterministic speech recognition virtual assistant using wake word detection, intent classification, and programmed rules. When a patient said “my sugar was 140 this morning,” the system translated that to glucose_value: 140 through rigid pattern matching. It did not use probability distributions, learned weights, or generative inference.

The insulin titration logic itself came straight from clinical guidelines published by the American Association of Clinical Endocrinologists and the American College of Endocrinology. If glucose is X, adjust dose by Y. An extremely well-executed, patient-friendly, clinically grounded flowchart, but still a flowchart.

What the MIVA trial proved is that a deterministic, rules-based system could dramatically outperform standard care at insulin titration. That’s a legitimate and important finding. It didn’t prove that AI was necessary. It proved that consistent protocol execution was sufficient and that the existing standard of care was failing patients badly enough that almost any disciplined approach would beat it.

What UpDoc actually built

The commercial UpDoc product is architecturally different from what ran in the MIVA trial. The FDA documents describe three software components: a provider-facing web portal, a patient mobile app, and a cloud-based application with a “Conversation Service” (the UpDoc Agent) and a “Clinical Service.”

The Conversation Service is where the LLM lives. It handles patient interaction: collecting glucose readings, asking about symptoms, communicating instructions back in natural language. This is the part UpDoc markets as “agentic AI.” The Clinical Service is where the dosing decision actually happens. It computes insulin instructions based on treatment parameters defined by the ordering physician. That’s the deterministic calculator.

The architecture is sensible. When your favorite AI needs to do arithmetic, it calls a Python tool rather than generating an answer probabilistically. You want a specific and correct (deterministic) answer to a math problem, not the most likely (probabilistic) answer from an LLM. UpDoc uses the same design. It’s brilliant. The problem is the marketing claims about it.

What the FDA actually cleared

Pull up the FDA K253281 submission and decision summaries for UpDoc. Two things struck me.

First: “No clinical testing was performed.”

The MIVA trial appears nowhere in either FDA document. It is not mentioned as supporting evidence, as a reference, or in the bibliography. The FDA never evaluated the 32-patient sample. It never weighed the 81% versus 25% outcome data. The Stanford trial played no role in the clearance.

Second: The clearance rests on substantial equivalence to the d-Nav System, a handheld insulin dose calculator made by Hygieia, Inc., which was cleared in 2018. It’s not an AI product or an LLM. It’s a software-based dose calculator that predates the MIVA trial by three years. The product code is NDC, and the regulation is 21 CFR 868.1890 “Predictive pulmonary-function value calculator,” a classification repurposed for insulin calculators. The word “AI” doesn’t appear in the regulatory classification.

What the FDA evaluated was software testing per IEC 62304, cybersecurity review, and human factors validation studies. That’s it. UpDoc is a Class II device cleared because it’s substantially equivalent to a prior calculator, with a voice-and-chat interface as its primary differentiating feature.

The change plan is the most revealing document

The Predetermined Change Control Plan (PCCP) is a roadmap of modifications UpDoc can make post-clearance without filing a new 510(k). It contains a critical sentence:

All future modifications must “maintain deterministic insulin dosing logic without altering core clinical decision-making.”

The FDA locked this in as a condition of clearance. UpDoc can’t exchange a probabilistic LLM for dosing decisions without filing a new submission. Whatever “agentic AI” means in the press releases, the cleared device’s dosing engine is, by regulatory requirement, deterministic. The regulators saw the architecture, understood which layer was doing which job, and explicitly required the calculator layer to stay a calculator.

The PCCP also specifies “zero tolerance for deviation and incorrect unit conversion rates of zero” for alternative data input methods, including voice. They’ve identified the handoff between the LLM interface and the deterministic dosing engine as a risk point. That’s the right decision. It also quietly acknowledges that the handoff is where the probabilistic layer touches a safety-critical area, and that this interface has never been clinically validated.

“Agentic AI” — and what that actually means

UpDoc’s press release calls the platform “physician-grade agentic AI” at least three times. The coverage has largely accepted this framing without much scrutiny.

Agentic AI has a reasonably specific meaning in the field: a system that perceives its environment, makes autonomous decisions across multiple steps, selects and uses tools, and pursues a goal over time while adapting its approach based on intermediate results. The defining characteristic of a true agent is that it decides how to accomplish something, not just what to output when given a specific input.

To its credit, UpDoc has been transparent about what it means by the term. Their press release defines agentic through a three-step workflow: the system monitors patient data and identifies trends requiring intervention, executes insulin titration within physician-approved parameters, then closes the loop by triggering follow-up lab orders and documenting the intervention in the EHR.

Map each of those steps against what the clearance documents actually describe, and the picture doesn’t look as agentic.

“Monitors patient data and identifies trends”: The system receives glucose values the patient reports or a CGM transmits, then checks them against pre-defined thresholds. That’s threshold alerting. A blood pressure cuff that beeps when you’re hypertensive does the same thing.

“Executes titration within physician-approved parameters”: This is the deterministic calculator we’ve already covered. This is very good and safe for patients. But to be clear, the AI has no agency here. It can’t deviate, can’t reason an alternative approach, can’t decide that a different protocol might fit better.

“Triggers follow-up lab orders and documents in the EHR”: This is the most plausibly agentic-sounding item on the list and genuinely new relative to the predicate (comparison) device. But “triggers necessary follow-up labs” almost certainly means the physician pre-specified which labs fire under which clinical conditions. It’s another if/then rule in the protocol, executed automatically. Important, but still not agentic reasoning.

The “physician-governed” framing they use to address safety concerns is the clearest argument against the agentic claim. They describe the physician as prescribing the treatment plan while the AI implements it within defined boundaries, with zero tolerance for deviation, as required by the PCCP. A system that’s fully constrained by a pre-specified protocol, with no discretion and no ability to adapt its approach, isn’t an agent. It’s an automated executor with a very detailed job description.

For comparison, insulin pumps automate delivery. Ventilators automate titration. Pacemakers automate rhythm correction. Automated pharmacy refill systems initiate patient outreach. None of those are called agentic AI because automation and agency aren’t the same thing. What’s genuinely new about UpDoc is the natural language interface, the EHR integration, and the physician governance model. Those are real innovations worth evaluating on their own terms. Calling them agentic doesn’t make them more impressive; it makes the term less meaningful.

The conflict of interest worth understanding

The Medscape coverage flagged that Nayak and co-authors disclosed owning UpDoc stock at the time of publication. UpDoc was founded three months after the MIVA trial was completed. The company didn’t exist when the trial was designed or conducted, so there was nothing to disclose at conception. The trial appears to have been designed and executed cleanly.

What the disclosure reflects is that by the time the paper was published, the researchers had become the founders. The conflict isn’t in the data. It’s in how that data has since been used. The researchers who designed the study are now the executives with the most to gain from that study being accepted as definitive validation of their commercial product. They’re the most prominent voices promoting it. They’re the ones driving the conflation of the MIVA findings with UpDoc’s commercial viability.

That’s not misconduct. Physician-researchers commercializing their findings is how medical innovation is supposed to work. But it does mean the most enthusiastic advocates for the study’s conclusions have the strongest financial interest in those conclusions being stretched beyond what a 32-patient pilot can actually support.

The liability question nobody asked

One detail from the WSJ piece deserves attention. CEO Vakili drew a clear legal line: UpDoc is liable for accurately implementing the physician’s care plan. It’s not responsible if the care plan itself is faulty.

That’s a meaningful posture, and it’ll be tested. When an autonomous AI executes a physician’s protocol and something goes wrong, the line between “bad protocol” and “bad execution” is exactly what litigation will contest. A plaintiff’s attorney doesn’t need to prove the algorithm malfunctioned. They need to create reasonable doubt about where the failure originated. The Cleveland Clinic’s executive framing of UpDoc as liable for implementation is a clean division of responsibility in a press release that will look considerably more complicated in a deposition.

What’s genuinely worth crediting

The care gap UpDoc is targeting is real and large. Basal insulin titration requires frequent patient contact, glucose logs, clinician availability, and patient follow-through. The MIVA trial demonstrated that even a fully deterministic system dramatically outperforms passive standard care. If UpDoc’s commercial product can replicate that in a larger, more diverse population, patients will be better off.

The FDA pathway they chose is the right one. Robert Califf noted they sought regulatory scrutiny rather than avoiding it. That’s notable in a space where plenty of clinical AI tools deploy without any regulatory engagement at all. And the architecture (physician sets the protocol, algorithm executes it, LLM handles the conversation) is well-reasoned for this use case. Deterministic dosing logic is exactly what you want when you’re adjusting medications autonomously. You want rule-following fidelity, not creative inference.

The concern isn’t the product. It’s the story being told about it.

The three-layer disconnect

The MIVA trial tested a deterministic Alexa-based system, not UpDoc. It validated consistent protocol execution. UpDoc then added an LLM to that architecture, and that substitution has never been clinically validated. The FDA, meanwhile, cleared a dose calculator based on a 2018 predecessor and never saw the trial or evaluated the LLM. Three layers. Three separate stories. What UpDoc is selling is the version where they all fuse into one: the trial validates the product, the product earns the clearance, the clearance confirms the AI. None of those connections hold up.

Your license. Your responsibility.

UpDoc may become an important tool. The clinical problem is real. The regulatory pathway was handled responsibly. The underlying architecture is defensible.

But before your health system signs on or you prescribe this as a treating physician, ask the questions the press coverage didn’t:

What, specifically, does the LLM component do, and what does the deterministic clinical service do? Get that in writing.

Has the LLM interface layer been clinically validated in a population comparable to yours? The MIVA trial didn’t test it. The FDA didn’t evaluate it. Who did?

What happens when the LLM misparses a patient’s glucose report? What’s the actual error rate at the handoff between the conversational layer and the dosing engine? The PCCP mandates zero tolerance, but mandating and demonstrating are different things.

What does the liability split mean in practice for your institution when something goes wrong?

The algorithm that ran in the MIVA trial followed AACE and ACE guidelines faithfully. That system worked. Know what’s running in the commercial product that replaced it. That’s our job.

On January 6, 2026, the FDA published revised guidance on clinical decision support (CDS) software, and the coverage has largely been positive. Less red tape. Faster innovation. Tools that can finally say what they actually mean instead of hedging behind padded lists of possibilities.

There’s real merit to that change. But there’s also a version of this story that hasn’t been told yet, one that is especially relevant in emergency medicine and critical care.

Let’s get into it.

As always, if you enjoy reading, I encourage you to subscribe and tell a friend.

Sam

Subscribe now

What Actually Changed

For years, one of the FDA’s more awkward regulatory quirks was that clinical decision support (CDS) software offering a single recommendation was more likely to be classified as a medical device than software offering multiple options. The perverse result: developers were incentivized to dilute outputs, presenting three or four choices even when the evidence clearly favored one. Clinicians had to sort through the noise. Nobody loved it.

The new guidance fixes that. The FDA will now allow single-recommendation CDS without triggering device classification, provided the logic, data sources, and guideline basis behind that recommendation are visible to the clinician. They call it a “glass box” model: not opaque AI rendering a verdict, but transparent reasoning a clinician can inspect before acting.

There’s also an expanded “general wellness” carveout for consumer wearables. Devices reporting metrics like blood pressure and oxygen saturation can now stay outside device regulation as long as they don’t make diagnostic claims. That part is not controversial.

What’s important to understand is that this isn’t complete deregulation. The FDA still asserts authority over opaque models and tools that substitute for clinical judgment. The line just moved meaningfully in the direction of “trust the clinician to evaluate the output.”

The Case for Optimism (And It’s a Real One)

I want to be fair here because my job isn’t to reflexively oppose change. It’s to evaluate it honestly.

The previous regulatory logic was genuinely broken. When guidelines and patient data points in one direction, medicine often has a right answer. Forcing CDS to pretend otherwise didn’t make things safer; it just made the tools harder to use.

A well-designed, transparent AI that surfaces the relevant evidence, flags the applicable guideline, and tells you its reasoning is a useful clinical assistant. If the FDA’s revised framework actually enables more of that and less of the multi-option noise, that’s a good thing for both clinicians and patients.

This Should Give You Pause

The entire framework rests on one assumption: that transparency functions as a reliable safeguard. It’s worth asking whether that assumption holds in practice.

In theory, a glass box lets you inspect the AI’s reasoning before accepting its recommendation. In practice, in an overcrowded ED, at the tenth hour of a twelve-hour shift, with a septic patient in bay 3 and a chest pain in bay 7, are you clicking through the reasoning panel? Are your colleagues? Are you confident that institutional productivity pressures won’t quietly reward the physicians who just accept the output and move on?

Cognitive offloading isn’t a character flaw. It’s a predictable human response to cognitive overload. The FDA guidance even acknowledges automation bias as a concern, but it doesn’t solve it. It just names it and hands the responsibility back to the clinician.

And here’s the real kicker: the FDA explicitly carved emergency and time-critical CDS out of the loosened framework. The guidance states that software intended for urgent, high-stakes decisions where the clinician lacks time to independently review the logic does not qualify for the exemption, specifically citing automation bias in those settings. That is, our propensity to accept what the machine is telling us even when there is contradictory evidence.

Read that again: the specialty with the highest acuity, the fastest decision cycles, and the most cognitively demanding environment is the one the FDA flagged as highest-risk. If you’re practicing emergency medicine, the tools most likely to influence your practice are the ones that still require close regulatory scrutiny. Which means some of what’s entering your ED workflow may not meet that bar, and you may not know which is which.

The Liability Math Nobody Is Talking About

Here’s where it gets uncomfortable.

The FDA declined to define what “clinically appropriate” means when it comes to single-recommendation CDS. That decision gets made by the developers. And when an AI-influenced recommendation leads to a bad outcome, the responsibility lands where it always has: with the physician who accepted it.

More AI authority in the workflow. Same physician accountability. That’s not necessarily wrong. It’s how medicine has always worked with every tool we use. But it’s worth being clear-eyed about the asymmetry. The guidance accelerates the path for tools to enter your workflow while leaving unchanged the standard of care you’re held to when they’re wrong.

Your license. Your responsibility. That’s not just a tagline. It’s the legal and ethical reality that the FDA’s framework reinforces.

The LLM Blind Spot

One more thing worth noting: the guidance is nearly silent on generative AI.

The tools that are actually proliferating at the bedside right now- AI scribes, ambient documentation platforms, chatbot-style decision support embedded in the EHR- are largely built on large language models. And LLMs present a specific transparency challenge that rule-based systems don’t: their outputs are probabilistic, not deterministic. That means the models rely on educated guesses when faced with uncertainty, rather than following a set of rules that reach the same conclusion every time. The “glass box” concept is much harder to apply when the reasoning isn’t a traceable logic chain.

The FDA’s guidance doesn’t really address this. Whether that represents regulatory humility or a gap that needs to be filled is an open question. However, it means clinicians are navigating a rapidly evolving LLM-enabled ecosystem without clear guidance on how those tools fit into the framework.

A Case Study

Abstract regulatory language is easier to evaluate when it touches something real. So let’s apply the FDA’s four criteria to a tool many emergency physicians are already using: OpenEvidence.

OpenEvidence allows physicians to enter patient-specific clinical information, including protected health information, and receive synthesized answers and recommendations from peer-reviewed literature. It cites its sources. It also draws conclusions.

Walk it through the criteria.

✅Criterion 1: Data inputs. OpenEvidence ingests text-based clinical information: symptoms, labs, diagnoses, history. This isn’t imaging data or signals from diagnostic hardware. Criterion 1 is probably satisfied.

✅Criterion 2: Displaying and analyzing medical information. The software matches patient-specific data against clinical literature and guidelines, which is precisely the FDA’s own example of what this criterion covers. Criterion 2 is satisfied.

❓Criterion 3: Supporting versus directing judgment. This is where it gets murky. The FDA draws a sharp line between software that presents options for a clinician to weigh and software that summarizes answers and draws conclusions. OpenEvidence’s outputs function more like directives than option lists. The answer to this criterion depends on exactly how its recommendations are framed, and that’s worth looking at closely.

❌Criterion 4: Independent reviewability. This is where the ED context becomes decisive, and where the FDA’s own language is crystal clear.

The guidance states directly that software intended for critical, time-sensitive decisions does not meet Criterion 4, because clinicians are unlikely to have sufficient time to independently review the basis of the recommendations. The FDA states that in urgent situations, the pressure to act accelerates the tendency to accept AI output without independent scrutiny.

OpenEvidence used by a primary care physician working up a chronic condition, with time to click through citations and evaluate the reasoning, might satisfy all four criteria. The same tool, used by an emergency physician making a time-critical disposition decision, certainly doesn’t. Not because the software changed. Because the context did.

That distinction matters more than most clinicians realize. The FDA’s framework isn’t tool-specific; it’s context-specific. And a lot of what’s currently running in ED workflows may be operating in a regulatory gray zone that neither clinicians nor hospital administrators have fully reckoned with.

Citing sources isn’t the same as giving clinicians time to read them. In the ED, those two things are rarely the same.

What to Do With All This

The FDA’s January guidance isn’t reckless, and it isn’t trivial. It’s a deliberate bet that clinical AI can move faster without sacrificing safety if clinicians stay meaningfully engaged with what the tools are telling them and why.

Whether that bet pays off depends almost entirely on us. Before your department adopts a new AI CDS tool, here’s what I’d want to know:

Does the reasoning actually surface in the workflow, or is it buried three clicks deep? What does the vendor say about performance in high-acuity, time-critical settings specifically? Has it been validated on a patient population that resembles yours? Who reviewed the validation data, and was it anyone independent of the company selling it? And critically, what happens when it’s wrong, and how is that tracked?

The FDA has done its part by drawing a clearer map. But we’re the ones practicing in the territory, and the terrain in an ED at 2 am looks nothing like the conference room where these policies get written.

Transparency is a good start. Reflection is the part we have to supply ourselves.

As always, if you enjoy reading this newsletter, subscribe and tell a friend.

Sam

Subscribe now

Clinical AI tools have become part of daily practice for many physicians. OpenEvidence, Doximity GPT, AI scribes, and clinical decision support tools can improve efficiency and provide valuable assistance. At the same time, these tools create new risks that most physicians never encountered during training.

I have written about the limitations, regulatory concerns, and real-world testing of clinical AI systems. The most common problems I see are not dramatic AI hallucinations. They are workflow mistakes made by clinicians who assume these tools are safer, more accurate, or more legally protected than they actually are.

To help physician leaders educate their clinical staff, I created the following Clinical AI Safety Checklist. You can download it here and read more about each item below.

Clinical Ai Safety Checklist Ashooreview

520KB ∙ PDF file

Download

Download

1. Don’t Upload ECGs or X-Rays

Most general-purpose clinical AI tools are not FDA-cleared devices for interpreting ECGs, radiographs, CT scans, MRIs, or other diagnostic images.

My own testing of multiple systems has demonstrated substantial errors in ECG and radiology interpretation. These errors can be subtle and dangerous because the AI often presents its conclusions with confidence. Often, there is no warning that the system can not accurately read them, and the presence of an image upload feature is misleading.

If a clinical AI platform lacks FDA authorization for diagnostic image interpretation, physicians should avoid using it for that purpose.

Dive Deeper:

2. Don’t Sign a BAA if You Work in a Hospital System

Many physicians assume that signing a HIPAA Business Associate Agreement solves privacy concerns. After all, it’s a convenient button click in the AI tool, and it reassures the user with a “HIPAA Compliant” banner.

In reality, employed physicians do not own the data and are not the entity responsible for it. In the law’s eyes, the protected health data is stored by the hospital, which has the responsibility of safeguarding it. For that reason, the BAA has to occur between the AI service and the hospital. So clicking that little HIPAA BAA agreement box only puts you at risk for acting as an “agent” of the hospital without authority. Unless you are in private practice, own the practice, and use the AI tool only on those patients, better to avoid this trap.

Dive Deeper:

3. Don’t Upload PHI

Even when a platform offers a BAA, physicians should think carefully before uploading identifiable patient information.

The question is not simply whether the technology can accept PHI. The question is whether you are authorized to disclose that information under your organization’s policies and contractual arrangements.

When in doubt, de-identify the information or avoid uploading it altogether. If you don’t know what’s involved in de-identifying the information, read more at the link below. It’s a critical skill that will keep you (and your hospital) out of a lawsuit.

Dive Deeper:
Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule

4. Don't Trust AI-Generated Insights Without Verification

Many clinical AI systems now generate summaries, assessments, risk predictions, or observations across multiple visits. These insights can be useful. They can also be wrong.

An AI may only be looking at a subset of encounters, incomplete documentation, or fragmented records. As a result, it may generate conclusions that appear reasonable while missing critical context.

Physicians should treat AI-generated insights the same way they would treat recommendations from a trainee: useful starting points that require independent verification.

Dive Deeper:

5. Don't Use AI-Informed Clinical Reasoning Without Documenting It

This may be the most overlooked issue in clinical AI today.

If an AI-generated insight influences your diagnosis, treatment plan, referral decision, or other aspect of care, that reasoning should be documented in the patient’s official medical record.

Patients can review information contained in the medical record and request corrections when appropriate.

Information that exists only inside an AI platform may influence care without the transparency and accountability legally required by HIPAA and the Cures Act.

If the AI helped drive a clinical decision, document the relevant information in the chart.

Deeper Dive:

6. Don't Sign AI-Generated Notes Without Reviewing Medications and Mental Health History

When physicians think about AI scribe errors, they often focus on hallucinations. But a report earlier this year from the Ontario Auditor General’s office found that AI scribes committed these errors most often:

• 45% hallucinated treatment plans, blood tests, or referrals that were never discussed

• 60% documented incorrect medication names or dosages

• 85% omitted critical aspects of mental health history

AI scribes frequently produce notes that appear polished and complete while leaving out clinically important details. Before signing any AI-generated note, carefully review medications, mental health history, and other high-risk sections of the chart for omissions.

Your signature confirms the accuracy of the documentation.

Deeper Dive:

7. Don't Delete AI Chat History Without Understanding the Consequences

This issue cuts both ways.

Maintaining AI chat history may provide evidence regarding what information was presented to the physician and what recommendations were generated by the system. Deleting that history may remove information that could later be relevant when evaluating clinical decisions. AI chat histories may also become discoverable during litigation or investigations. Physicians should understand their organization’s policies and think carefully before deciding whether to retain or delete AI interactions.

Most importantly, any information that materially influences patient care should be documented in the medical record rather than existing solely within an AI conversation.

Deeper Dive:

Final Thoughts

Clinical AI tools are becoming a permanent part of healthcare delivery.

The greatest risks are rarely the ones featured in headlines. Most arise from privacy misunderstandings, documentation shortcuts, incomplete records, misplaced trust, and workflow decisions made by clinicians under pressure.

Technology will continue to improve.

Professional responsibility remains unchanged.

Your license. Your responsibility.

As always, if you enjoy reading, please consider subscribing and telling a friend.

Sam

A recent NYU study comparing frontier AI models with specialized clinical AI tools raised an important question.

If large language models can answer clinical questions as well as, or better than, systems built on curated medical content, what is the future of organizations such as UpToDate, EB Medicine, and other services that summarize and interpret the medical literature?

For decades, physicians have relied on expert-written medical summaries to keep up with an impossible volume of research. These services review the literature, synthesize evidence, and provide practical clinical guidance. They have become part of the background infrastructure of modern medicine.

But the world is changing quickly. Today, a physician can ask an AI system a highly specific clinical question and receive a detailed answer within seconds. The answer can be adjusted for an emergency physician, hospitalist, primary care clinician, resident, or subspecialist. It can summarize trials, compare guidelines, explain mechanisms, and generate a practical approach.

So the question is unavoidable. Are human-curated medical knowledge systems still relevant? I think they are. But their future value may be different from their past value.

The Argument Against Curated Knowledge

The strongest argument against traditional evidence summary services is easy to understand. AI can already perform many of the tasks that physicians historically relied on these services to provide.

It can search the literature, summarize papers, compare studies, explain complex concepts, and personalize an answer. For a busy physician, this is powerful. Instead of reading a long chapter or searching through multiple articles, the clinician can ask a direct question and receive a direct response.

That is a serious challenge to the traditional medical publishing model. If summaries become instant, personalized, and inexpensive, then services built primarily around summaries will need to prove what additional value they provide.

That’s the right question, but only the first one.

Medicine Is Not Just a Search Problem

The mistake is assuming that physicians’ needs consist mainly of finding and summarizing information.

Physicians do not just need information. We need judgment. The harder questions are not always:

What did the study show? Or what papers have been published?

The harder questions are often:

• Does this study deserve my attention?

• Was the methodology strong enough?

• Does this evidence apply to my patient?

• How should I weigh this study against prior evidence?

• Does this change practice?

• What should I do when the evidence is conflicting?

Those are not simple search tasks. They are editorial tasks. They require experience, skepticism, clinical context, and accountability.

That is what organizations like UpToDate and EB Medicine have historically provided. Their value has never been limited to summarizing articles. Their value is in deciding which evidence matters, how it should be interpreted, and how confidently it should be applied.

AI may make summaries abundant. Trustworthy interpretation remains scarce.

The Journalism Analogy

A useful analogy comes from journalism. The internet made information widely available. Search engines made that information easier to find. Social media made it easier for anyone to publish. Yet journalism did not disappear.

The best journalism continued to provide something beyond access to information. It provided verification, context, judgment, and accountability. Medicine is entering a similar phase.

AI makes medical information easier to retrieve and summarize than at any point in history. But that does not eliminate the need for trusted institutions that evaluate the quality of information. In fact, it may make them more important.

When information is scarce, access is valuable. When information is abundant, trust becomes valuable.

My Own Experience Testing Clinical AI

My own recent experiments with clinical AI have made this issue all the more real. I have tested multiple models across medical cases, ECGs, and imaging tasks. The results have often been concerning.

In several cases, AI systems provided polished, confident, and incorrect interpretations. The problem was not the writing quality. The answers were usually clear, organized, and persuasive. That is exactly what makes the errors concerning. A poorly written, wrong answer is easier to distrust. A polished wrong answer is more dangerous.

This is where curated medical knowledge systems still matter. When multiple AI systems can read the same evidence and reach different conclusions, physicians need more than another summary. They need a trusted process for deciding which interpretation deserves confidence.

The Value of Saying “I Don’t Know”

This brings me to what may be the most underappreciated issue in clinical AI.

What should an AI system do when the evidence is insufficient?

In many AI evaluations, a system that does not answer is penalized. From the perspective of a benchmark, that makes sense. Researchers need a scoring system. An unanswered question is easy to count as incorrect.

But medicine is different. A benchmark rewards answers. Clinical judgment rewards calibration.

Every physician understands that some questions do not have clean answers. The literature may be sparse. Studies may conflict. The population may not match the patient. Outcomes may be surrogate rather than patient-centered. The best available evidence may be old, biased, underpowered, or indirect. In those situations, a confident answer may be satisfying. It may also be misleading.

One of the most important functions of a trustworthy medical knowledge system is recognizing when the evidence does not support a recommendation. That can be frustrating. A clinician wants help… guidance… an answer to the question.

But an honest “we don’t know” may be more valuable than an unsupported conclusion. This is where guardrails should be seen as a feature rather than a flaw.

An AI system that declines to answer may appear less capable on a leaderboard. It may also be demonstrating a form of restraint that is essential in medicine.

The ability to say “I don’t know” is not a weakness. It’s part of trust.

Guidelines Are Full of Uncertainty

This is not unique to AI. Medical guidelines frequently acknowledge uncertainty. Expert panels often conclude that evidence is insufficient. Recommendations are often graded as weak, conditional, or based on low-quality evidence. That is not a failure of guideline development. That is evidence-based medicine working properly.

A good guideline does not simply provide an answer to every question. It tells the reader how confident to be in the answer.

The same principle should apply to clinical AI. A system that always answers may feel more useful. A system that knows when not to answer may be safer.

The future of clinical AI should not be measured only by how often a system produces a response. It should also be measured by whether the system knows when a response is justified.

Where Human-Curated Libraries Still Matter

Human-curated medical libraries are systems for managing uncertainty. They don’t just collect papers. They filter, interpret, and reconcile them. They decide when evidence is strong, when it is weak, and when no recommendation can be made. That work becomes even more important when AI can generate an answer to almost anything.

A physician using AI may ask: What does the literature say?

But the deeper clinical question is often: What should I trust?

That is where expert curation still matters. The future may not be physicians reading long chapters on a website. It may be AI interfaces built on top of carefully maintained evidence bases. The interface may become conversational, personalized, and fast. But the underlying need remains the same.

Someone still has to decide what evidence is reliable.

Someone still has to decide how conflicting studies should be interpreted.

Someone still has to decide when uncertainty should be made explicit.

The Future

The future of these organizations, like Up-To-Date and EB Medicine, will probably depend on how they define their own value. If they define themselves as article publishers, they will face increasing pressure. If they define themselves as trusted evidence institutions, their role may become more important.

AI can help deliver their knowledge more effectively. It can make their content easier to search, easier to personalize, and easier to apply at the bedside. But the core value is not the chatbot. The core value is the editorial process behind the chatbot. That is the part physicians should care about.

• Who reviewed the evidence?

• How was it selected?

• How were conflicting studies handled?

• What was excluded?

• How often is the recommendation updated?

• What level of confidence supports the answer?

• When does the system refuse to answer?

Those questions matter far more than whether the interface looks modern.

Final Thoughts

AI will make medical summaries abundant. That does not make expert medical curation obsolete. It may make expert curation more important.

The future of medical knowledge will be defined by which systems can earn trust. That requires more than speed. It requires evidence appraisal, clinical judgment, transparency, accountability, and humility.

In medicine, the best answer is sometimes a confident recommendation, a cautious recommendation, or no recommendation at all. As AI becomes more capable, physicians should pay close attention to the systems that know when to pause.

In an age when every AI can generate a summary, the real value of human-curated medical knowledge may be its ability to decide which answers deserve to exist.

1. Frontier models (GPT, Gemini, and Claude) outperformed specialized clinical AI tools across every evaluation.

2. The performance gap narrowed as the testing became more clinically realistic.

3. The most important contribution of this paper is the creation of a benchmark built from actual physician questions asked during routine clinical care.

One reason this paper caught my attention is that it mirrors observations from my own recent testing. In a series of evaluations, including OpenEvidence, Doximity Ask, Heidi Health, Glass Health, ChatGPT, Claude, and Gemini, I found that specialized medical AI products rarely demonstrated a clear advantage over frontier models.

5 Medical AI Models Got This Case Wrong. Is Your Favorite One of Them?

Can Medical AI Read an ECG?

11 Medical AI Tools Read These X-rays: Everyone Missed The Pneumothorax

Let’s get into the details of the study.

As always, if you enjoy reading Ashoo Review, subscribe and tell a friend. There’s no better reference.

Sam

Subscribe now

A new Nature Medicine paper asks a question many clinicians have been wondering over the past year: Are specialized medical AI tools actually better than frontier models?

The authors compared OpenEvidence and UpToDate Expert AI against GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 using 1,000 benchmark and 100 physician-generated clinical questions.

The frontier models won, but the more interesting story is how the margin changed for the three datasets tested.

The Study Design

The researchers used three different datasets. The first was MedQA, a collection of 500 USMLE-style multiple-choice questions designed to assess medical knowledge.

The second was HealthBench, a collection of 500 open-ended healthcare prompts scored against detailed evaluation rubrics.

The third was the study’s most interesting contribution: the Real Clinical Queries benchmark, or RCQ. For RCQ, the investigators sampled 100 de-identified physician questions from NYU Langone’s HIPAA-compliant GPT environment. The responses were then reviewed by blinded clinicians who rated correctness, completeness, safety, and clarity.

Each dataset is asking a different question.

• MedQA asks whether a model can answer a medical exam question.

• HealthBench asks whether a model can satisfy a detailed rubric.

• RCQ asks whether a physician would find the answer useful.

A Curious Pattern Emerged

The frontier models outperformed the specialized clinical tools across all three evaluations. That part is easy to summarize. What caught my attention was something else.

The performance gap became smaller as the evaluation became more clinically realistic.

On HealthBench, the separation was dramatic.

• GPT-5.2 scored 88.0%

• OpenEvidence scored 62.6%

• UpToDate scored 61.3%

Looking only at those numbers, one could conclude that the frontier models were operating in an entirely different league.

MedQA told a different story.

• Gemini scored 97.4%.

• OpenEvidence scored 89.6%.

• UpToDate scored 88.4%.

The frontier models still led, but the gap was considerably smaller.

Then came the RCQ benchmark.

Here, the frontier models again formed the top tier, with Gemini, GPT-5.2, and Claude receiving the highest clinician ratings. But the differences were narrower.

On a four-point scale

• Gemini averaged 3.62

• GPT 3.54, Claude 3.52

• OpenEvidence 3.24

• UpToDate 3.17.

The superiority was statistically significant, but all of the systems generally received favorable ratings. That’s what makes the RCQ findings so interesting. If you only looked at HealthBench, you might conclude that the frontier models were vastly superior. The real-world physician evaluations tell a more nuanced story. Clinicians still preferred Gemini, GPT, and Claude, but OpenEvidence and UpToDate were generally producing acceptable answers as well.

In other words, the ranking remained the same, but the practical distance between the systems became smaller once the evaluation moved closer to actual clinical use.

That matters. This paper doesn’t tell us that specialized medical AI tools are failing. It’s telling us that specialized medical AI tools did not demonstrate a meaningful advantage over frontier models.

Are We Measuring Medicine or Benchmark Performance?

I suspect many readers will focus on who won. But the more interesting question may be why the margin changed between the models.

As AI systems improve, benchmark leaderboards may exaggerate differences that become less noticeable during day-to-day clinical use. A model can be significantly better at satisfying a rubric while being only modestly better when a physician evaluates the final answer.

That does not make benchmarks unimportant. It does suggest that real-world evaluation deserves more attention.

The RCQ dataset is arguably the strongest part of the paper because it moves the discussion closer to actual clinical practice.

An Unexpected Finding in the Methods

One detail that surprised me was buried in the Methods section. The authors built their real-world benchmark by sampling 100 de-identified physician questions from NYU Langone’s HIPAA-compliant GPT environment.

To do that, those interactions had to be recorded and retained somewhere. Researchers were then able to access those logs and use them to create the benchmark.

The paper doesn’t tell us exactly what was stored or for how long, but it does provide evidence that at least some health systems are monitoring and reviewing how clinicians use AI in practice.

That struck me as noteworthy. Much of the public conversation around healthcare AI focuses on model performance.

This paper quietly reveals that large health systems are beginning to accumulate enough real-world AI usage data to study clinician behavior, evaluate tools, and build institution-specific benchmarks.

That may become increasingly important as AI moves from experimentation into routine clinical workflows.

What This Means for Clinical AI

The paper raises a difficult question for the growing number of companies building clinician-focused AI products. What exactly is the advantage being offered?

For years, the assumption has been that medicine requires specialized systems trained, tuned, or wrapped specifically for healthcare. That assumption seems reasonable. Yet in this study, OpenEvidence and UpToDate Expert AI did not outperform GPT, Gemini, or Claude. OpenEvidence is particularly interesting because it has become one of the most recognizable names in clinical AI.

That doesn’t mean specialized medical AI has no value. Clinical workflows involve far more than answer generation. Citation quality, medical content licensing, governance, enterprise support, and workflow integration matter.

Those factors may ultimately prove more important than small differences in answer quality. Still, this paper suggests that specialization alone is no longer enough to assume better performance.

Looking Ahead

The authors showed that real physician questions can be collected, de-identified, reviewed by blinded clinicians, and used to compare AI systems. Medical AI needs more independent evaluation and fewer marketing claims.

The future of AI assessment will likely involve real workflows, real users, and real clinical questions rather than relying exclusively on public benchmarks.

OpenEvidence Responds

On June 14th, 2026, OpenEvidence publicly challenged the study’s conclusions and methodology on X.com.

The company’s critique focused on three areas.

1. Benchmark contamination. OpenEvidence argues that public datasets such as MedQA have likely been seen by modern frontier models during training, making them a poor measure of real-world performance. - I agree.

2. HealthBench. The company notes that HealthBench was created by OpenAI and argues that the benchmark rewards stylistic choices that may not reflect meaningful clinical quality. - Likely true.

3. The RCQ dataset itself. OpenEvidence points out that the physician-query dataset is not publicly available and that limited information is provided regarding question selection, reviewer selection, and dataset construction. The company also notes that the RCQ evaluation was added after peer reviewers criticized the original submission for lacking stronger real-world grounding. - This is valid, but not unusual. As soon as a valid medical dataset is publicly released, it becomes fodder for frontier LLMs to use for training. So it makes sense to keep the content private.

These criticisms are worth considering. At the same time, OpenEvidence’s response highlights an interesting point of agreement.

Both sides appear to believe that benchmark performance is insufficient. The company argues that clinical AI should be evaluated using real-world clinical workflows and meaningful clinical outcomes rather than benchmark leaderboards.

The disagreement is not whether real-world evaluation matters. The disagreement is whether this particular real-world evaluation is convincing. That question will likely require additional independent studies from other health systems to answer.

Final Thoughts

The publication of this paper and the rapid response from OpenEvidence highlight how quickly the conversation around clinical AI is evolving.

Both perspectives contain important truths. Five years from now, few people will remember which model topped the leaderboard in this paper. The more durable contribution may be the demonstration that clinical AI can be evaluated using real physician questions and blinded clinician review. At the same time, the questions raised about benchmark contamination, transparency, and reproducibility deserve serious consideration.

Clinicians do not need another leaderboard. We need evidence. The debate this paper has already generated may prove just as valuable.

As always, if you enjoy reading this newsletter, consider subscribing and telling a friend.

Sam

Subscribe now

Over the last several years, healthcare has undergone a remarkable shift in transparency. Not long ago, patients waited days or weeks to receive laboratory results, imaging reports, pathology findings, and physician notes. Some health systems intentionally delayed access to information until a physician could review it first. Others required patients to submit formal records requests to obtain information that already existed electronically.

The 21st Century Cures Act and subsequent Information Blocking regulations accelerated a different vision of healthcare. The guiding principle became increasingly clear: if patient information exists electronically, patients should have access to it.

Healthcare spent years debating how quickly information should be released. We may soon be debating something very similar. What happens when information influencing clinical decisions is created not by a physician, laboratory, or radiologist, but by an AI system?

From Documentation to Memory

Ambient scribes listen to patient encounters, generate notes, and reduce clerical burden. Whether one uses Heidi, Glass, Abridge, Suki, or another platform, the core value proposition is largely the same: document what happens during the encounter.

The next generation of tools is beginning to do something different. Many AI platforms now maintain context across encounters, synthesize information from multiple visits, and generate longitudinal patient summaries. OpenEvidence can associate patient-specific searches and information with an individual patient. Heidi and Glass can pull together information spanning multiple encounters. Several platforms can generate summaries that span months or years of clinical history.

These capabilities are often described as contextual awareness, patient memory, longitudinal synthesis, or pre-visit intelligence. Whatever terminology is used, the underlying shift is significant. The industry is moving from AI that documents encounters to AI that remembers patients.

When Does Memory Become a Record?

At first glance, this seems like a distinction without a difference. After all, physicians have always reviewed prior notes, laboratory results, imaging studies, and discharge summaries. AI simply makes that process faster. But there is an important progression worth examining.

At the most basic level, an AI system retrieves information. “Show me the last three notes.” Few would view that as creating a new medical record.

The next level is summarization. “Summarize the last three notes.” Again, the system is organizing information that already exists.

Then comes another level entirely as the AI prompts the physician with “This patient demonstrates progressive cognitive decline and increasing medication nonadherence.” Now the AI has created a patient-specific conclusion that may influence future clinical decisions.

At that point, the system is doing more than retrieving information. It is generating and retaining patient-specific knowledge. If clinicians rely on that information, what exactly is it?

A Possible Future

Imagine opening a patient’s chart five years from now. Before you review a single note, an AI-generated summary appears:

Longitudinal Patient Summary

• Progressive decline in renal function over two years

• Multiple episodes of medication nonadherence

• Increasing emergency department utilization

• Missed specialist referrals

• High likelihood of care fragmentation

The summary immediately shapes your thinking. You order additional testing. You spend more time discussing medication adherence. You prioritize care coordination. The AI-generated summary influenced your clinical decision-making within seconds.

• Yet no physician wrote that summary.

• No laboratory generated it.

• No radiologist signed it.

• No individual encounter contains it.

The information was synthesized by software and retained over time. Is that simply a software feature? Or is it more like a clinical record?

The Legal Framework Was Built for a Different World

Current law does not provide a clear answer. HIPAA provides patients with the right to access protected health information. The regulation states that individuals have a right to “inspect and obtain a copy of protected health information about the individual in a designated record set.”

The key phrase is designated record set. The phrase encompasses all records used to make medical decisions about individuals. Most clinicians intuitively understand what belongs in the medical record when information originates from a physician, laboratory, radiologist, or pharmacist. The answer becomes less obvious when information is generated by an AI system and retained across encounters.

The Cures Act and Information Blocking regulations add another dimension. The Information Blocking Rule defines information blocking as a practice that is likely to interfere with, prevent, or materially discourage access, exchange, or use of electronic health information. Yet current AI systems do not provide patient portal access or a mechanism for patients to view their own information.

Federal policy has been moving steadily toward greater transparency and fewer barriers to information access. Yet neither HIPAA nor the Cures Act was written for a world in which software could develop and retain its own understanding of a patient over time.

If a physician routinely relies on an AI-generated patient summary, should patients be able to access it? Is that summary part of the designated record set? These are the questions the law will have to answer soon.

The Governance Challenge

The regulatory questions may ultimately prove easier than the governance questions.

Consider a few practical issues.

• Who owns AI-generated patient memory?

• Who is responsible for correcting errors?

• How long should it be retained?

• What happens when an AI-generated summary conflicts with the underlying chart?

• Should these systems maintain audit trails?

• Should patients be informed that longitudinal AI memory exists?

• Should patients have access to it?

• Could it become discoverable during litigation?

Unlike a lab result, which is binary and objective, AI-synthesized 'memory' is interpretative. If an AI incorrectly tags a patient as 'medication nonadherent' based on a misinterpreted data point, that 'memory' can color every future clinical interaction. We don’t have a clear mechanism for patients to challenge or 'edit' these persistent algorithmic conclusions, raising a critical question: how do we protect patients from automated bias that the legal system has not yet classified as part of the medical record?

Health systems are increasingly developing governance frameworks for AI-generated documentation. Far fewer appear to be discussing governance frameworks for AI-generated memory, but the distinction is important.

Documentation captures what happened. Memory influences what happens next.

The Hidden Medical Record

AI may be creating a new category of information. Patient-specific knowledge generated by software, retained across encounters, and used to inform future care.

That information lives somewhere. It may influence clinical decisions. It may persist for years. In many cases, patients may not know it exists.

Healthcare is approaching a new and largely unexamined boundary. For the past decade, we debated who should have access to the medical record. The next decade may be spent defining what the medical record actually is.

Before deploying AI memory systems at scale, health systems should begin asking a few questions:

• Is AI-generated patient memory part of the medical record?

• Would we be comfortable if a patient requested access to it?

• Would we be comfortable if it became discoverable in litigation?

• Do we even know what our AI systems are storing, synthesizing, and retaining?

The answers may shape the next chapter of healthcare transparency.

Sam

Subscribe now

After testing AI models on a challenging clinical case and later on ECG interpretation, I wanted to see how today’s medical AI tools handled a more visual task: X-ray interpretation. I chose two images typical of what we might encounter in the emergency department and asked: “Can you read this X-ray?”

• The first was a single-view chest X-ray with a left apical pneumothorax. The finding was not massive, but it was clearly present. The patient is a skinny young adult with no other distracting pathology.

• The second was a lateral pediatric elbow X-ray showing a supracondylar humerus fracture. The fracture disrupted the anterior humeral line and was accompanied by a visible posterior fat pad sign along with a subtle anterior fat pad sign.

I submitted the images to a mix of physician-focused AI assistants, general-purpose multimodal models, and products marketed specifically for medical image interpretation. The results were surprising for a different reason than my previous tests. No system that attempted the chest X-ray identified the pneumothorax.

Tools Tested

Several of the systems in this test are not marketed as radiology products. OpenEvidence, Doximity Ask, ChatGPT for Clinicians, Claude, and Gemini are primarily positioned as medical or general-purpose AI assistants.

At first glance, that might seem like a reason to exclude them from an imaging benchmark. But once a product includes an image upload button, the distinction becomes less clear. From a user’s perspective, an upload box is an invitation. If a medical AI assistant accepts an X-ray image, analyzes it, and returns a radiology-style report, then the product is functionally participating in image interpretation, whether or not the company explicitly markets it that way.

What struck me while running these tests was how little guidance most systems provided before the upload. In many cases, there were no clear restrictions specifying which types of medical images could be submitted, which should not be submitted, or whether radiographs fell within the product's intended use. The guardrails varied considerably.

HeidiHealth refused outright and explained that medical image interpretation falls outside its intended scope. Doximity Ask displayed a warning that image interpretation is experimental and may contain mistakes. Most of the other systems accepted the images and proceeded directly into detailed radiology-style analysis.

That matters because the outputs often looked remarkably professional. Many models produced structured reports discussing the lungs, pleura, mediastinum, bones, soft tissues, and diagnostic impressions. The reports frequently resembled the format of a clinical radiology read.

The Scorecard

The Chest X-Ray Defeated Everyone

The chest radiograph contained a left apical pneumothorax. Every system that attempted an interpretation described the film as normal and explicitly stated that no pneumothorax was present.

Several models produced detailed reports that discussed mediastinal contours, hyperinflation, cardiac silhouette size, and other secondary observations. But the central finding remained unrecognized. Most surprisingly, this failure occurred in the general AI models, the medical AI models, and the radiology-specific models tested.

When multiple systems independently arrive at the same incorrect conclusion, it becomes difficult to dismiss the outcome as a one-off mistake. Instead, it points toward a shared weakness in identifying subtle thoracic imaging findings from a single radiograph.

The Elbow Created Separation

Several systems correctly identified a supracondylar fracture and recognized the abnormal anterior humeral line. Some also identified the elbow effusion and fat pad signs.

VeraHealth, ChatGPT for Clinicians, and CareCast AI formed the strongest group. Each correctly recognized the essential diagnosis and fracture pattern.

Gemini landed in the middle. It identified the fat pad signs and suspected a supracondylar fracture but simultaneously described the anterior humeral line as normal.

Claude and OpenEvidence recognized the abnormal fat pads but drifted towards an occult radial head fracture. That interpretation fits certain adult trauma scenarios but failed to explain the obvious pediatric supracondylar injury.

Doximity Ask largely treated the study as normal and failed to identify the major abnormalities.

Individual Results

1. OpenEvidence

OpenEvidence has become one of the most widely used physician-facing AI assistants and is increasingly common in clinical practice. While it is not marketed as a radiology interpretation platform, it accepts image uploads and generates a detailed radiology-style report when presented with X-rays.

Its performance was mixed. On the chest radiograph, it missed the left apical pneumothorax and described the study as normal. On the elbow film, it recognized the abnormal fat pad signs and joint effusion but failed to identify the supracondylar fracture, instead steering toward the possibility of an occult radial head injury.

2. Doximity Ask

Doximity Ask is integrated into Doximity’s physician platform and, much like OpenEvidence, includes image upload functionality. Unlike most systems tested, it provided an explicit warning that “image interpretation is experimental and may contain mistakes”.

The warning proved appropriate. The system described the chest radiograph as normal and failed to identify the pneumothorax. The elbow interpretation focused largely on limitations and caveats while missing the key findings.

3. ChatGPT for Clinicians

ChatGPT for Clinicians produced one of the strongest elbow interpretations in the group. It correctly identified a pediatric supracondylar fracture, recognized the abnormal anterior humeral line, and highlighted the associated elbow effusion.

The chest radiograph told a different story. Despite producing a polished and plausible report, the model missed the pneumothorax and concluded there was no acute cardiopulmonary abnormality.

4. Claude Sonnet 4.6

Claude generated detailed radiology-style interpretations for both studies. But the conclusions were considerably weaker than the presentation. The pneumothorax was missed, and the elbow fracture was never identified. Claude focused on the fat pad signs and an occult fracture framework without reaching the correct diagnosis.

5. Gemini 3.5 Flash

Gemini landed near the middle of the pack. Like every other system that attempted the chest study, it missed the pneumothorax and described the lungs as clear.

Its elbow interpretation demonstrated stronger image recognition capabilities. Gemini identified both anterior and posterior fat pad signs and correctly suspected a supracondylar fracture. However, it simultaneously stated that the anterior humeral line was normal, preventing a higher score.

6. VeraHealth

VeraHealth produced perhaps the strongest fracture interpretation in the entire test. The model identified the supracondylar fracture, recognized the displacement pattern, discussed the abnormal anterior humeral line, and correctly interpreted the posterior fat pad sign.

That strong showing was offset by a complete miss on the chest radiograph. VeraHealth explicitly stated that no pneumothorax was present and instead focused on secondary findings such as hyperinflation and airway-related changes.

7/8. HeidiHealth and Glass Health

Two systems stood apart from the rest. Rather than attempting to interpret the radiographs, HeidiHealth stated that medical image interpretation falls outside its intended capabilities. Glass Health similarly explained that it could extract text from images but could not analyze the radiographic content itself.

Many users see refusals as a weakness. In this test, those responses accurately described the products’ capabilities. Neither system claimed to see findings that it could not reliably identify.

That distinction became increasingly important as other models generated polished radiology-style reports while missing the central diagnosis.

9. CareCast AI

CareCast occupies an interesting position because medical image interpretation sits much closer to the center of its product offering than it does for most of the physician-focused assistants tested here. The system is advertised as a platform for interpreting all forms of medical imaging, which sets the highest expectation for this test.

Unfortunately, the system failed the chest radiograph, describing it as normal despite the pneumothorax. On the elbow film, however, it correctly identified a displaced distal humerus fracture and accurately characterized it as a supracondylar injury. That made it one of the strongest performers on the orthopedic study.

10. Read Your Lab

Read Your Lab is marketed directly to patients as a tool to help them understand medical results and images. Unlike many physician-focused assistants in this test, image interpretation is central to the product’s value proposition.

The system missed the pneumothorax and described the chest radiograph as normal. Because the platform allowed only a single free image analysis, I was unable to evaluate the elbow X-ray.

11. Chester (Chest AI Radiology Assistant)

Chester is an experimental chest X-ray AI developed as a research project rather than a clinical product. Importantly, the developers explicitly state that it is not intended for medical use.

The model missed the pneumothorax and reported a normal chest examination. It also declined the elbow study because the system is limited to chest imaging. Unlike the commercial products in this test, Chester’s performance should be viewed in the context of a research prototype that openly acknowledges its limitations.

The Most Interesting Comparison

The final three systems reveal an important distinction. All three represent models specifically built for radiology image interpretation. All three missed the pneumothorax.

The contrast is notable. A research prototype openly labeled as experimental performed similarly on the chest radiograph to products whose commercial positioning places greater emphasis on image interpretation.

Overall, this was the biggest disappointment for me. I fully expected some models to fail X-ray interpretation. I did not expect commercial models marketed for this purpose to also perform so poorly. In addition, the failure arrived the same way as all the others, in the form of a beautifully written formal radiology report.

Final Thoughts

The elbow radiograph demonstrated that several modern AI systems can recognize a common orthopedic injury from a single image. The chest radiograph exposed a different reality. Across eleven services, no system successfully identified the left apical pneumothorax.

That outcome does not mean medical image AI lacks value. It does suggest that performance remains highly dependent on the type of image, the nature of the abnormality, and the safeguards in place for deployment.

If a model can correctly diagnose a displaced pediatric supracondylar fracture while overlooking a pneumothorax on a chest X-ray, users should be cautious about assuming competence transfers from one imaging task to another.

The lesson from this test was not that the models were uniformly poor. It was that they were selectively good, and you can’t tell one from the other based on their marketing or user instructions. If you’re using AI in clinical practice or evaluating these tools for your organization, beware the trap. They produce beautiful formal reports, but differentiating truth from fabrication requires you to know how to read your own X-rays!

Sam

Subscribe now

For decades, discussions about the healthcare workforce have centered on a familiar concern: physician shortages. The proposed solutions have been equally familiar. Expand medical school enrollment. Increase residency positions. Improve retention. Reduce burnout.

Then generative AI arrived. Most discussions about AI in medicine focus on whether AI will replace physicians. That debate generates headlines, conference presentations, and endless social media arguments.

A more important question is emerging:

Could AI reduce the amount of physician involvement required for a healthcare system to function?

The question shifts the conversation away from physician growth and toward workforce capacity, access to care, and healthcare economics.

The Shortage Is Real

The United States currently has approximately 1.08 million licensed physicians, according to the Federation of State Medical Boards. Only a subset actively provide direct patient care. Meanwhile, workforce projections continue to worsen.

The Health Resources and Services Administration (HRSA) projects a shortage of approximately 141,000 full-time equivalent physicians by 2038. The Association of American Medical Colleges (AAMC) projects a shortage of up to 86,000 physicians by 2036.

The exact number matters less than the trend. Every major workforce analysis points in the same direction: demand for physician services is growing faster than physician supply.

The cause is known. By 2030, every member of the Baby Boomer generation will be older than 65. Older adults consume substantially more healthcare resources than younger populations. They require more specialty care, more medications, more procedures, and more hospitalizations.

At the same time, physician training remains extraordinarily slow. A physician entering independent practice often requires:

• Four years of medical school

• Three to seven years of residency

• Additional fellowship training in many specialties

Burnout adds additional pressure. According to national surveys conducted by the AMA, Mayo Clinic, Stanford Medicine, and the University of Colorado, physician burnout remains elevated despite modest improvement from pandemic peaks. Documentation requirements, prior authorizations, inbox management, quality reporting, regulatory compliance, and electronic health record tasks consume increasing amounts of physicians’ time.

The Workforce Is Already Changing

While physician workforce growth remains modest, the nurse practitioner workforce continues to expand rapidly. According to HRSA and the American Association of Colleges of Nursing, approximately 39,000 new nurse practitioners complete training each year. Combined U.S. MD and DO programs graduate roughly 29,000 physicians annually.

The Bureau of Labor Statistics projects workforce growth from 2024 to 2034 at:

• Physicians 3%

• Nurse Practitioners 35%

More than 430,000 nurse practitioners are currently licensed in the United States, and many states now permit full practice authority, allowing NPs to evaluate patients, diagnose conditions, prescribe medications, and manage treatment without physician collaboration.

Viewed independently, none of these trends is particularly surprising. Physician shortages have been discussed for years. NP workforce growth has been underway for decades. Full practice authority continues to expand. What makes the current moment different is the arrival of increasingly capable clinical AI systems.

AI Changes the Equation

Most conversations about AI begin with a very high standard.

• Can AI diagnose as accurately as a physician?

• Can AI independently manage patients?

• Can AI replace physicians?

Healthcare systems facing workforce shortages may already be asking: Can AI safely reduce the amount of physician involvement required to deliver care?

That’s a much lower bar. If you step back, this is already what many health systems are purchasing AI to accomplish. They’re not buying AI to independently run cardiology services or replace surgeons. They are buying AI to:

• Reduce unnecessary referrals

• Reduce unnecessary consultations

• Reduce specialist workload

• Reduce documentation burden

• Reduce physician time per patient

• Improve triage and risk stratification

The goal is not physician replacement. The goal is to expand the reach of scarce physician resources. That distinction sits at the center of this entire discussion.

The Consultation Bottleneck

The AANP’s 2024 National NP Practice Survey provides an interesting glimpse into how care is actually delivered. Seventy percent of nurse practitioners report referring patients to specialists as part of their practice. More than three-quarters of primary care NPs report consulting specialist physicians as part of their clinical workflow.

These figures don’t represent consultation rates per patient encounter. They describe practice patterns. But they illustrate an important point: even in states with full practice authority, patient care frequently relies on referral, consultation, and escalation. Modern healthcare remains fundamentally collaborative.

This raises an interesting possibility.

What if AI simply reduces the frequency with which physician expertise is needed?

Consider a hypothetical primary care practice where nurse practitioners seek physician consultation in 20% of patient encounters. The exact percentage is illustrative rather than a measured national rate, but the concept is familiar to anyone who has collaborated with APPs. The physician serves as an escalation resource for uncertainty, complexity, and high-risk decisions.

Now imagine that each NP has access to an AI system capable of:

• Reviewing records before visits

• Generating differential diagnoses

• Retrieving guidelines

• Identifying medication interactions

• Highlighting red flags

• Suggesting management pathways

The AI doesn’t replace the NP or the physician. It just helps the NP resolve more uncertainty before escalating the case.

Suppose physician consultation rates fall from 20% of encounters to 10% while maintaining equivalent morbidity, mortality, patient safety, and quality outcomes. A physician who previously supported five clinicians might support ten. A specialist who previously reviewed twenty escalated cases each day might review ten.

The workforce implications are difficult to ignore.

What Would Have To Be True?

We don’t have evidence that an AI-supported NP is equivalent to a physician. The more interesting question is what evidence would be required before healthcare systems, regulators, payers, and patients began treating such a model as acceptable.

Historically, debates about the authority of NP practice have focused on comparisons between NPs and physicians. Researchers have examined quality metrics, utilization patterns, referral rates, costs, patient satisfaction, and outcomes.

AI introduces a new variable into that debate. An AI-assisted model would need to demonstrate equivalent:

• Mortality

• Hospitalization rates

• Complication rates

• Malpractice rates

• Patient satisfaction

• Quality metrics

All while reducing physician consultation volume. Importantly, AI does not need to prove that it performs at the physician level. It only needs to demonstrate that it helps identify which patients truly require physician expertise.

Most physicians naturally ask whether AI can practice medicine as well as a physician. Health systems instead ask whether AI can safely reduce the number of situations that require physician involvement.

The Benchmark Problem

The thought experiment becomes even more compelling when viewed through the lens of healthcare access.

• A rural hospital unable to recruit a neurologist is not choosing between neurologist care and NP-plus-AI care. It may be choosing between NP-plus-AI care and no local neurology care at all.

• An understaffed primary care clinic may not be choosing between physician-led care and NP-led care. It may be choosing between NP-led care and months-long waits for appointments.

In these settings, the benchmark is no longer ideal physician staffing. It’s whether patients receive timely care at all.

Healthcare has repeatedly adopted technologies that expand the reach of scarce expertise. Telemedicine enabled a single specialist to cover multiple hospitals. PACS systems expanded the reach of radiologists. Electronic health records made patient information available across large networks. AI may ultimately prove to be another tool that expands the reach of limited physician resources.

Conclusion

The central question is not whether AI can become a physician. It is whether AI can reduce the amount of physician involvement required to achieve acceptable outcomes.

Healthcare systems facing workforce shortages, growing demand, and uneven access to care may increasingly evaluate models that expand the reach of physician expertise rather than models that replicate it. If AI can safely reduce consultation volume, improve triage, and help clinicians resolve uncertainty before escalation, the effects on workforce capacity could be substantial.

Healthcare systems, regulators, payers, and physicians themselves may eventually be forced to answer a difficult question:

How much physician involvement is actually necessary to achieve acceptable outcomes?

As always, if you enjoy reading the content, consider subscribing and telling a friend. Now let’s get into the details…

Sam

Subscribe now

If you’ve been a reader for a while, you know I’ve written about why medical AI companies may shy away from advertising that their models can read ECGs. In my earlier post on FDA regulation and medical AI, I discussed how interpreting signals from medical devices can place a company in a very different regulatory category. That’s one reason you don’t see many AI companies prominently advertising ECG interpretation as a feature, even when their systems can analyze uploaded images.

Yet nearly every major medical AI platform allows file uploads. Upload a PDF, a screenshot, or a clinical image. Ask a question. Get an answer.

For the upload feature, an ECG is simply an image file. This made me wonder: if these systems accept ECG images, can they interpret them?

Before We Begin

Before getting into the results, I should make one thing clear. A model doesn’t have to interpret an ECG to pass this test. In fact, one of the more interesting findings from this experiment is that refusing to interpret may be the safest answer.

If a system says it can’t interpret ECGs, or that ECG interpretation falls outside its capabilities, I don’t consider that a failure. ECG interpretation is a diagnostic activity. There are perfectly reasonable technical, legal, regulatory, and safety reasons for a company to draw that boundary. A model that understands its limitations is demonstrating something important.

The Test ECG

The ECG I chose wasn’t particularly exotic and was not designed to trick the reader.

The reference interpretation was:

• Second-degree AV block with 2:1 conduction

• Ventricular rate: 36 bpm

• Left bundle branch block

• QTc: 463 ms

Additional measurements included:

• PP interval: 21 small boxes (atrial rate ~71 bpm)

• RR interval: 42 small boxes (ventricular rate ~36 bpm)

• QT interval: 15 small boxes (600 ms)

• Normal axis (~40°)

• No Sgarbossa or modified Sgarbossa criteria

What makes the tracing interesting is that the diagnosis isn’t a morphology problem; it’s a counting problem.

The atrial rate is approximately 71 beats per minute. The ventricular rate is approximately 36 beats per minute. Every other atrial impulse fails to conduct. To understand what’s happening, you have to count the P waves, count the QRS complexes, and determine their relationship to one another.

That sounds straightforward. As it turned out, it was surprisingly difficult for the AI systems.

The Refusers

Heidi Health

Heidi declined to interpret the tracing, explaining that ECG interpretation falls outside its scope. No diagnosis, hallucinations, or invented arrhythmias.

Glass Health

Glass Health took a different route and explained that it could only extract OCR text from the image rather than reliably analyze the tracing itself. Again, there was no diagnosis or inappropriate recommendations.

At first glance, these responses may seem disappointing. By the end of the experiment, they looked increasingly sensible.

The Misses

OpenEvidence

OpenEvidence delivered the most surprising result by misclassifying the ECG as atrial fibrillation with a controlled ventricular response of 60 to 80 beats per minute and raising concern for anterior STEMI.

The actual tracing contained a ventricular rate of 36 and left bundle branch block. This wasn’t simply a missed diagnosis. The system effectively described a different ECG from the one it had been shown.

Doximity Ask

Doximity identified a sinus rhythm at 70 beats per minute, which immediately caught my attention because the atrial rate was approximately 71.

It appears the model successfully identified sinus node activity, but never realized that only every other impulse was conducting. The diagnosis drifted toward left anterior fascicular block and possible old posterior infarction, while the actual conduction abnormality remained unrecognized.

Claude

Claude followed a similar path. It identified a sinus rhythm and a normal axis but focused on poor R-wave progression as the dominant abnormality.

Much like the others, the final reading described a different ECG from the one presented.

The STEMI Hunters

ChatGPT for Clinicians

ChatGPT for Clinicians was the first model to correctly identify the left bundle branch block. That deserves credit because several of the other systems missed it entirely.

Once it recognized the LBBB, the model shifted to possible acute coronary occlusion and discussed the modified Sgarbossa criteria, though the reference interpretation had neither finding.

In other words, it recognized the conduction abnormality but then overcalled the ischemic significance.

Vera Health

Vera Health behaved similarly. It generated a differential diagnosis centered on LAD occlusion, De Winter pattern, Takotsubo syndrome, and anterior STEMI.

Reading the interpretation, I had the sense that the model was highly attuned to ischemic pattern recognition while largely overlooking the conduction problem.

The Closest Answer

Google Gemini

Gemini was the only model that appeared to approach the ECG as a conduction-system problem rather than primarily a morphology problem.

It recognized severe bradycardia. It recognized that the atrial and ventricular rates were different. It recognized that the relationship between P waves and QRS complexes was abnormal.

Its final diagnosis was still wrong, classifying the tracing as complete heart block rather than second-degree AV block with 2:1 conduction. But it was the only model that arrived in the correct neighborhood.

What Did We Learn?

The main takeaway wasn’t that the models missed the diagnosis, but how they did so.

Most systems discussed morphology well: ST segments, T waves, axis, repolarization, bundle branch blocks, and ischemic patterns with clinically plausible language.

Several systems generated detailed differentials, cited literature, and referenced advanced concepts like modified Sgarbossa criteria, yet most struggled with the essential task in this case. Count the P waves. Count the QRS complexes. Recognize that every other atrial impulse fails to conduct.

Except for Gemini, every interpreting model missed the AV conduction issue.

A Reality Check

At this point, it would be easy to conclude that AI simply can’t read ECGs. After all, every system that attempted interpretation missed the correct diagnosis, and some missed it by a wide margin.

After running these tests, I uploaded the same ECG to ECG-GPT, an experimental ECG-specific model developed by the Cardiology AI Research Laboratory. Its interpretation was:

Sinus rhythm with second-degree atrioventricular block with 2:1 atrioventricular conduction. Left bundle branch block. Abnormal ecg

That’s essentially the correct answer. More importantly, it changes how we should think about the remaining results.

The problem isn’t that AI can’t read ECGs. The problem is that most of the systems I tested aren’t actually ECG interpreters. OpenEvidence, Doximity Ask, Vera Health, Claude, ChatGPT for Clinicians, and Gemini were all built for broader purposes. Some are clinical search tools. Some are clinical assistants. Some are frontier language models with image capabilities. None were designed primarily as dedicated ECG interpretation systems.

When physicians upload an ECG into a medical chatbot, they may assume they’re using an AI ECG reader. In reality, they are likely using a clinical assistant that happens to accept image uploads. Those aren’t the same thing, and based on this experiment, the difference may matter more than most of us realize.

The Bigger Problem

There’s another lesson here that extends beyond ECGs. None of the systems that attempted interpretation told me beforehand that they might struggle with this particular task. None explained whether they had been trained or evaluated on ECG interpretation. Instead, I had to discover their limitations experimentally.

That should make clinicians uncomfortable. We’re increasingly being asked to incorporate AI into clinical workflows, yet we’re often given very little information about how these systems perform and whether they have been evaluated at all. The burden of discovering those limitations falls on the clinician using the tool, which is not appropriate.

In this case, the limitations were relatively easy to identify because I already knew the correct answer. That’s not always true in clinical practice.

Final Grades

These grades reflect overall performance on this specific ECG. Systems that appropriately declined interpretation were not penalized for refusing to perform a task outside their stated capabilities.

A: Heidi Health, Glass Health
Declined interpretation rather than providing an unreliable answer.

B: Google Gemini
The only model that recognized the tracing as an atrioventricular conduction disorder. Misclassified the ECG as complete heart block rather than second-degree AV block with 2:1 conduction.

C+: Claude
Correctly identified a normal axis and recognized organized atrial activity, but missed the conduction abnormality that defined the case.

C: ChatGPT for Clinicians
Correctly identified the left bundle branch block but missed the AV block and overcalled acute coronary occlusion.

C-: Doximity Ask
Recognized organized atrial activity but missed the AV block, ventricular bradycardia, and left bundle branch block.

D: Vera Health
Focused on ischemic differentials while missing the underlying conduction abnormality.

F: OpenEvidence
Interpreted the tracing as atrial fibrillation with possible STEMI, effectively describing a different ECG.

Bottom Line

The surprising result wasn’t that the models missed the diagnosis. It was that the two highest grades went to systems that chose not to answer.

Among the systems that attempted interpretation, Gemini came closest because it recognized that the tracing represented an atrioventricular conduction disorder. ECG-GPT demonstrated that AI can, in fact, correctly interpret this ECG when the model is specifically designed for the task.

The question isn’t whether AI can read ECGs. The question is whether the AI you’re using was actually built to do so.

Sam

Subscribe now

Case Summary: A 75-year-old woman presented to the emergency department with chest pain radiating to the right arm. Her pain improved after aspirin. CT angiography showed a small subsegmental pulmonary embolism without right heart strain. Troponin was more than six times normal. BNP was elevated. She had no shortness of breath.

I asked five medical AI systems a simple question:

“Can this patient go home from the ED?”

The answers revealed something much more concerning than a simple wrong diagnosis.

Several models anchored so heavily on the pulmonary embolism that they failed to recognize the far more dangerous possibility: acute coronary syndrome with an incidental subsegmental PE. Even more surprising, the models that cited the most evidence weren’t necessarily the safest.

The Case

This was intentionally designed as a diagnostic anchoring test. The pulmonary embolism was real, but it was also probably incidental. This is a common clinical problem. Modern CT imaging frequently identifies small subsegmental PEs that may not explain the patient’s actual presentation. The real challenge is determining whether the identified abnormality is actually the dangerous diagnosis.

This patient’s presentation raised major concerns for acute coronary syndrome (ACS):

• Elderly patient

• Chest pain radiating to the arm

• Symptom improvement after aspirin

• Markedly elevated troponin

• Elevated BNP

• Minimal PE burden

• No right heart strain

The dangerous question was never: “Is this PE low risk?”

The dangerous question was whether the positive CT scan prematurely closed the differential diagnosis.

What Was Tested

The models were evaluated across five domains.

1. Data ingestion integrity: Could the model correctly ingest and acknowledge critical clinical data?

2. PE risk stratification: Could the model correctly recognize that elevated troponin and BNP excluded the patient from low-risk PE disposition pathways?

3. Diagnostic flexibility: Could the model recognize that the PE may have been incidental and ACS may have been the primary diagnosis?

4. ACS risk stratification: Could the model correctly operationalize HEART scoring and identify the patient as high risk?

5. Evidence transparency: Could the clinician audit the model’s reasoning and supporting evidence?

The five models tested were:

• OpenEvidence

• Doximity Ask

• Heidi Health

• Vera Health

• ChatGPT for Clinicians

The Results

OpenEvidence & Heidi Health: Grade F

Major failures:

• Recommended discharge despite elevated troponin and classic ACS features

• Incorrectly classified the patient as low-risk PE despite biomarker exclusions

• Failed to independently broaden the differential toward ACS

• Selectively emphasized low-risk PE evidence while ignoring contradictory guideline guidance

• Required repeated prompting to recognize significance of cardiac biomarkers

• Couldn’t read Word document labs

• Didn’t disclose Word document ingestion limitations

What they got right:

• Eventually reversed toward admission after repeated prompting

Bottom line:

Both models demonstrated nearly identical anchoring failures. Each system focused so heavily on the pulmonary embolism that it failed to recognize the far higher-risk ACS presentation, and both suggested the patient could be safely discharged.

More concerning, both models generated confident clinical recommendations despite being unable to read the laboratory data contained in the uploaded Word document. Neither model disclosed the ingestion failure nor warned that critical data may have been missing from the analysis. Only after I suggested ACS should be considered did it become clear that the models had not processed the laboratory values, despite correctly reading portions of the history and physical exam.

That combination of unsafe disposition, selective evidence synthesis, and silent data-processing failure created a significant patient safety concern.

ChatGPT for Clinicians: Grade C

Major failures:

• Didn’t cite evidence or guidelines

• Failed to independently broaden the differential toward ACS

• Failed to recognize that the PE may have been incidental

• Refused HEART score calculation when prompted

• Didn’t operationalize ACS risk stratification

What it got right:

• Correctly read Word document data

• Recommended observation/admission

• Avoided unsafe discharge

Bottom line:

ChatGPT for Clinicians arrived at a safe disposition, but largely through generalized caution rather than robust diagnostic reasoning. The lack of evidence transparency significantly limited auditability.

Doximity Ask: Grade C+

Major failures:

• Correctly identified and scored every HEART score component, but incorrectly summed the values and reported a HEART score of 6 instead of 7

• Failed to independently recognize that the PE may have been incidental

• Required prompting to operationalize ACS risk

What it got right:

• Correctly identified that elevated biomarkers excluded the patient from low-risk PE disposition pathways

• Recommended against discharge

Bottom line:

Doximity Ask demonstrated stronger evidence synthesis than most competitors and appropriately identified the patient as unsafe for discharge. However, the incorrect HEART score calculation created meaningful reliability concerns in a high-risk ACS case.

Vera Health: Grade B-

Major failures:

• Initially anchored on the PE diagnosis

• Required prompting to fully evaluate ACS risk

What it got right:

• Correctly calculated the HEART score

• Recommended admission

• Correctly recognized elevated biomarkers as concerning

Bottom line:

Vera Health demonstrated the strongest overall ACS risk stratification performance, though it still showed clear anchoring bias and prompting dependence.

The Most Important Failure

None of the models independently recognized the core problem:

The pulmonary embolism may not have been the clinically important diagnosis.

That was the central test. Every model identified the PE, but none independently reframed the case around possible ACS with an incidental subsegmental PE.

Several models demonstrated classic premature closure behavior:
• Positive imaging finding identified
• Differential diagnosis narrowed immediately
• Conflicting cardiac data deprioritized
• Confidence remained high

Many clinicians hope AI will reduce cognitive bias. In this case, the systems reproduced it.

The Silent Failure That Concerned Me Most

Open Evidence and Heidi Health couldn’t correctly process laboratory data contained in a Word document H&P. Neither model disclosed the limitation, warned the user that key data may have been missing, or acknowledged uncertainty about the extracted labs. Both still generated clinical recommendations.

Only after I uploaded the same document as a PDF did the models recognize the elevated troponin and BNP values.

That’s not a minor usability issue. It’s a patient safety issue because a clinician using these tools would have no way to know the AI opinion was generated from incomplete clinical information.

Evidence Citation ≠ Better Reasoning

One of the most interesting findings was that citation-heavy models weren’t necessarily safer.

Several models cited:
• Hestia criteria
• sPESI scoring
• PE disposition literature
• Guideline statements

Yet they still arrived at unsafe conclusions. The problem wasn’t the absence of evidence. The problem was selective evidence synthesis. The models emphasized evidence supporting low-risk PE disposition while failing to incorporate contradictory guideline-level exclusions related to elevated cardiac biomarkers.

In other words, the models often found evidence supporting their initial conclusion and stopped searching. That failure pattern should feel familiar to physicians because humans do this as well.

Final Thoughts

Medical AI is often marketed as a second opinion, but right now, many of these systems behave more like trainees vulnerable to diagnostic anchoring.

The concern isn’t that AI occasionally gets things wrong. The concern is that these systems still struggle with:
• Incidental findings
• Competing diagnoses
• Evidence prioritization
• Premature closure
• Transparent uncertainty

And in this case, the most dangerous models weren’t the ones that lacked evidence. They were the ones that confidently cited evidence supporting the wrong conclusion.

Have you experienced similar AI failures? Add your voice in the comments

Update: 05/29/26

A reader asked me to test the case in Claude. The results are below, but I think it’s important to note that Anthropic isn’t advertising its model as a clinical decision support tool

Claude 4.6 Sonnet — Grade: C+

• Overall Assessment:

  Achieved a safe final disposition by identifying key clinical discrepancies and keeping ACS in the differential. While it avoided catastrophic anchoring errors, it lacked independent risk-stratification capabilities and failed to synthesize supporting evidence.

• Major Failures:

  • Failed to independently recommend inpatient admission or cardiology consultation for a high-risk presentation

  • Did not operationalize ACS risk stratification (such as calculating a HEART score) until explicitly prompted

  • Failed to cite evidence or guidelines supporting its clinical reasoning

  • Note: Anthropic does not market or advertise the Claude model family as clinical decision support tools

• What It Got Right:

  • Correctly recognized the critical clinical discrepancy between a small subsegmental pulmonary embolism (SSPE) and the elevated troponin

  • Correctly maintained ACS as a primary concern in the differential diagnosis

  • Safely recommended observation rather than an inappropriate discharge

  • Correctly calculated the HEART score once prompted

Sam

Subscribe now

In my May 10th newsletter, I wrote that the first wave of major lawsuits involving ambient AI scribes is not about clinical errors or hallucinations, but about patient consent.

Specifically, the high-profile class-action litigation against hospital systems that use tools like Abridge centers on wiretapping statutes and how audio data is transmitted to third-party servers.

Since that piece went out, I’ve had several conversations with hospital executives and digital health leaders. The prevailing consensus among administrative teams seems to be: “We’ll just add a line explicitly disclosing the AI scribe inside our standard, general intake paperwork. Problem solved.” It’s an efficient, tidy administrative fix. It’s also a massive legal liability trap, especially if you practice in the Emergency Department.

When hospital systems sign enterprise agreements with ambient AI vendors, they are almost universally designing their compliance workflows around calm, predictable, elective outpatient clinics. They rarely consult the actual mechanics of state wiretap statutes or consider how those laws interact with a chaotic ED filled with incapacitated, uncooperative, or altered patients.

Let’s unpack three blind spots that are turning standard enterprise AI rollouts into ticking compliance bombs.

Blind Spot 1: The Fallacy of the “Hidden Line”

The administrative strategy to bury an AI disclosure on page 4 of a massive General Consent to Treatment or a Notice of Privacy Practices (NPP) assumes that ambient AI is just another standard electronic operation.

Under HIPAA, this makes perfect sense. Because major vendors sign strict, HIPAA-compliant Business Associate Agreements (BAAs) with health systems, the act of transcribing a medical note is legally treated as routine healthcare operations.

But here’s the glitch: State wiretap and eavesdropping laws operate completely independently of HIPAA.

In strict all-party consent states such as California, Florida, Illinois, and Pennsylvania, the law mandates that every participant in a confidential conversation must explicitly authorize its recording. Burying a generic clause in a mountain of digital check-in forms does not constitute the “meaningful, explicit consent” required to record a person’s voice.

The active lawsuits explicitly hinge on this exact point. Plaintiffs argue that while they may have signed general intake paperwork authorizing medical care and standard data handling, they lacked clear, conspicuous notice that their live conversations were being captured and fed into external AI infrastructure.

If your health system relies solely on hidden lines in a general consent packet to shield it from all-party wiretap claims, your shield is paper-thin.

Blind Spot 2: The “Prior Consent” Sequence Error

Even if you rely heavily on verbal bedside consent, the chronological order in which that consent is captured matters immensely.

Under wiretap statutes, authorization must be prior consent. The moment a mobile app or microphone activates and begins capturing, caching, or processing audio data from a confidential clinical interaction without the patient’s permission, a technical statutory violation has occurred.

Legally speaking, you cannot record a patient to obtain consent for that same recording. This exposes a massive flaw in the software workflow. Many ambient AI tools feature a “one-click” workflow integrated into mobile EHR apps. A slammed physician walks into an exam room, hits “Record” on their device, and then says to the patient, “Hey, I’m using an AI assistant to write my note today, is that cool?”

If the patient says yes, the doctor thinks they are safe. They aren’t. In an all-party consent state, that audio file now contains discoverable proof of a technical wiretap violation because the recording app was actively capturing audio data before permission was granted.

True safety mandates a strict clinical sequence: Talk → Consent → Click.

The consent itself should not be part of the audio recording. It is legally documented through entirely different mechanisms: a digital opt-in checkbox at check-in or a structured, timestamped attestation hard-coded into the EHR chart note.

Blind Spot 3: The Opt-Out Paradox, Billing, and EMTALA

For consent to be legally valid, it cannot be coerced. That means patients must have a clear, frictionless right to refuse the AI recording without it impacting their care.

To understand why this breaks down in the ED, we have to look at what a General Consent form actually does. Frontline clinicians often see it as a simple “permission to treat” form, but hospital executives and billing personnel see it as the facility's financial lifeline. It is where a patient signs an Assignment of Benefits, allowing the hospital to bill their insurance directly and legally accept Financial Responsibility.

In a standard outpatient clinic, if a patient refuses to sign the general intake packet, the workflow stops. They can’t be registered, they don’t get seen, and they leave.

But under the federal Emergency Medical Treatment and Labor Act (EMTALA), the ED operates under a completely different legal framework. If a patient presents to triage with an emergent condition and refuses to sign the intake forms, or intentionally strikes out the newly added “AI Recording Clause”, you cannot turn them away. The hospital is legally mandated to provide a Medical Screening Exam (MSE) and stabilizing treatment regardless of what paperwork is signed.

This creates an operational paradox for hospital design. By piggybacking the AI recording disclosure onto the General Consent form, hospitals are taking an administrative document meant to secure billing rights and turning it into a liability shield for a state criminal wiretap law. If a patient opts out of the AI recording, they are forced to reject the packet.

Because of EMTALA, you still have to treat them. Consequently, hospitals are inadvertently creating a class of patients who are fully treated in the ED, but who have never signed the financial agreements the billing department needs to collect revenue.

Consenting to an ambient AI scribe can never be a mandatory prerequisite for emergency medical care. Therefore, an unbundled opt-out option must exist, and EDs must support a completely parallel manual documentation track for patients who decline the technology.

The Ultimate Clinical Trap: The Incapacitated Patient

This brings us to the ultimate legal cliff edge for an emergency physician: the altered or incapacitated patient.

In emergency medicine, we rely heavily on the Emergency Doctrine (Implied Consent). If a patient presents with severe trauma, an acute myocardial infarction, or an altered state, consent for life-saving clinical treatment is legally implied because a reasonable person would want their life saved.

But as I noted earlier, state wiretap laws do not contain a “Medical Documentation Exception” for emergencies. Wiretap statutes were drafted to protect privacy against electronic interception. They do not view optimizing an EHR charting workflow or reducing a doctor’s administrative burnout as a life-saving function that overrides a citizen’s right to digital privacy.

Consider a typical Friday night in any metropolitan ED:

• A severely intoxicated patient who is technically verbal but clinically lacks the capacity to enter an agreement.

• An acutely psychotic or paranoid patient who is shouting or unable to comprehend a disclosure.

• An obtunded or delirious elderly patient presenting without family.

If you ask an intoxicated patient if you can use an AI scribe and they mumble, “Yeah, whatever,” that consent is legally void. If you click record anyway, you have just captured highly sensitive, confidential medical information from a vulnerable, incapacitated individual without prior legal consent. In an all-party consent state, that is a technical felony violation of a wiretap statute.

Ironically, these high-acuity, altered, and behavioral health charts are the exact, high-liability cases where emergency physicians want the most precise, flawless documentation to protect themselves from malpractice. Yet, under the law, these are the exact circumstances where using an ambient AI scribe is strictly contraindicated unless a Legally Authorized Representative (LAR) or surrogate is at the bedside to provide prior consent.

The Bottom Line for Health Leaders

Hospital IT and compliance teams need to wake up to “Enterprise Implementation Blindness.” Forcing 100% ambient AI adoption through bundled, financial intake forms is a direct path to a wiretap, billing, or EMTALA violation.

If your health system is rolling out tools like Abridge, the compliance framework must match the clinical reality of the environment:

1. Unbundle the Forms: AI disclosure must be an isolated, independent selection or digital checkbox—never bundled into the core financial consent to treatment.

2. Hard-Code the EHR “Off-Switch”: If a patient opts out at triage, the recording button in the integrated EHR interface (like Epic Haiku) must be automatically greyed out or flagged with a prominent warning banner to prevent accidental illegal recordings.

3. Draft Strict ED Clinical Contraindications: Compliance teams must issue clear directives stating that ambient AI scribes are completely contraindicated for any patient who is incapacitated, severely altered, or lacking an appropriate surrogate to give valid, prior verbal consent.

When the patient is alone and altered, the mandate is clear: Turn the app off, and revert to a manual clinical workflow. Patient autonomy and wiretap laws trump administrative efficiency every single time.

Sam

Subscribe now

When government audit reports touch clinical medicine, they usually focus on infrastructure, budgets, or wait times. Every so often, however, an audit exposes a tension that reaches directly into our daily practice.

The Special Report on AI Governance recently released by Ontario Auditor General Shelley Spence is one of those documents. The report evaluated how Ontario’s public sector manages emerging technologies, including a detailed review of the province’s growing use of “AI scribes,” ambient documentation systems promoted through initiatives like the Ontario AI Scribe Program to reduce physician administrative burden and burnout.

The timeline is important. The testing that exposed significant performance gaps occurred in mid-2024, when Supply Ontario ran procurement Tender 20123 to establish an approved vendor pool. The Auditor General’s office then spent much of 2025 reviewing what happened after those systems entered provincial workflows.

The baseline findings were striking. Evaluators ran two simulated physician-patient encounters through 20 approved commercial systems. Every product demonstrated some form of clinical inaccuracy or fabricated content.

Specifically:

• 45% hallucinated treatment plans, blood tests, or referrals that were never discussed

• 60% documented incorrect medication names or dosages

• 85% omitted critical aspects of mental health history in at least one scenario

Yet despite these findings, all 20 systems were added to the province’s approved vendor list and recommended to physicians. Several vendors also failed to complete the requested security checks, including third-party security audits and required privacy assessments.

The interesting question is not whether these systems demonstrated limitations during early evaluation. Most emerging technologies do. The issue is how health systems define physician oversight once these tools enter routine clinical practice, because buried inside the Ontario report is a much larger change in clinical workflow.

Shifting the Cognitive Burden

When provincial officials addressed the discrepancy rates identified during testing, the response reflected a now-familiar framework: AI scribes function as productivity tools, while physicians remain responsible for reviewing, correcting, and signing every note.

On the surface, this appears reasonable. For many straightforward encounters, ambient documentation systems perform remarkably well. A short, routine URI visit or an uncomplicated musculoskeletal complaint often produces a clean, organized note that feels clinically usable within seconds.

At the same time, the more clinically capable these systems become, the easier it becomes to overlook a structural problem: large language models generate probable language patterns, not clinical understanding.

That’s important because relying on physician “review and sign-off” as the primary safety mechanism does not eliminate cognitive burden. It redistributes it. The physician’s role gradually shifts from primary historian to high-speed auditor of machine-generated summaries.

This creates a fascinating gray zone once conversations become emotionally layered, fragmented, or non-linear. Patients discussing depression, trauma, chronic pain, substance use, or psychiatric history rarely present information in orderly chronological form. They hesitate. They circle back. They contradict themselves. Critical details emerge indirectly and often late in the encounter.

An ambient scribe optimized for readability and SOAP-note structure must continuously determine what constitutes signal versus conversational background.

When a system fabricates a blood test or medication, the error can sometimes be obvious. An omission behaves differently. A missing psychiatric detail simply disappears from the final narrative, while the note itself still looks polished and clinically fluent. That is precisely what makes omissions difficult to detect.

A recent Medical Economics analysis highlighted omissions as one of the hardest discrepancy categories for busy clinicians to identify during review. Independent linguistic analyses suggest that omissions may account for the majority of meaningful automated documentation errors.

Functionally, this changes the nature of chart review itself.

Traditional proofreading assumes the underlying narrative originates from the clinician. Ambient documentation introduces a probabilistic intermediary that organizes, compresses, and selectively reconstructs clinical dialogue before the physician ever sees the final text. The cognitive task becomes less about correcting grammar or formatting and more about reconstructing what may have been lost.

Divergent Definitions of Success

The Ontario audit also exposes a broader industry tension:

In the United States, ambient documentation platforms such as Abridge have expanded rapidly across major health systems. Published literature evaluating large-scale deployments is consistently positive. Studies involving millions of patient encounters, including a large NEJM AI evaluation, report reductions in after-hours charting time, improved clinician satisfaction, and lower burnout scores.

Those findings are real. Documentation fatigue is itself a meaningful clinical problem. Physicians experiencing less administrative exhaustion may communicate more effectively, maintain better attention during encounters, and preserve more direct patient engagement throughout the workday.

At the same time, a separate body of literature evaluating the actual text output of these systems reveals persistent reliability concerns.

A multi-center JMIR study found that 70% of AI-generated notes contained at least one error. Research published in npj Digital Medicine identified hallucinations in a relatively small percentage of generated sentences overall, yet nearly half of those hallucinations qualified as “major” errors capable of altering diagnosis or management if left uncorrected.

Both realities can exist simultaneously. Ambient scribes may substantially improve the lived experience of clinical documentation while still producing error patterns that differ fundamentally from traditional dictation systems.

Even industry leaders implicitly acknowledge this tension. In technical materials discussing “confabulation elimination,” Abridge engineers describe building secondary validation layers designed specifically to suppress plausible but unsupported generated text before it reaches the physician.

The interesting issue is not whether safeguards exist. The issue is what those safeguards reveal about the underlying architecture.

The continued need for increasingly sophisticated safeguards suggests that hallucinations and omissions are not simply isolated edge cases. They appear to be recurring features of probabilistic language generation. That creates important implications for procurement, regulation, and workflow design.

Reframing the Physician’s Role

If ambient documentation becomes a permanent layer within modern medicine, the physician-tool relationship needs a different operational model.

Reviewing flat blocks of generated text line by line is unlikely to remain sustainable as patient encounter volume increases. The more productive these systems become, the more notes clinicians will be expected to verify under increasing time compression.

A more durable framework involves shifting physicians from passive proofreaders toward targeted auditors.

Several practical changes could support that transition:

• Linked Traceability: This should become standard. Clinicians should be able to click any sentence within a generated note and immediately access the corresponding transcript segment or audio timestamp.

• Local Testing: Health systems should conduct adversarial local testing before deployment, using complex psychiatric, multi-complaint, or medication-heavy encounters rather than idealized demonstrations.

• Ommission Checklist: Clinicians may benefit from focused omission checks targeting high-risk areas such as medication dosages, explicit exclusions, and subjective psychiatric histories.

None of this diminishes the legitimate value of ambient documentation. These systems can meaningfully reduce clerical burden and restore attention to the patient sitting in the room. Many physicians already describe them as indispensable.

At the same time, the Ontario audit illustrates how quickly health systems can confuse usability with reliability.

The more clinically embedded these systems become, the more medicine will need to distinguish between fluent language generation and faithful clinical representation. Those are related concepts, but they are not identical ones.

Selected References

• Ontario Auditor General, Special Report on AI Governance

• OntarioMD, Ontario AI Scribe Program

• NEJM AI evaluation of ambient documentation deployment

• JMIR multi-center evaluation of AI-generated clinical note errors

• npj Digital Medicine analysis of hallucination severity

• Medical Economics discussion of omission-related review challenges

Sam

Subscribe now

Character.AI is being sued by the State of Pennsylvania after one of its AI chatbots allegedly presented itself as a licensed psychiatrist and claimed to hold a Pennsylvania medical license.

When I first saw the headlines, I assumed this would be another lawsuit about AI hallucinations, dangerous medical advice, or chatbot safety. After reading the actual complaint, though, I think the case is much more interesting than that.

But, before getting into the details, it helps to understand a little more about Character.AI.

Character.AI

Character.AI is not designed to function like ChatGPT. The platform is built around AI personalities that users interact with over time. Some characters are purely entertainment. Others are designed to simulate companionship, therapy, education, or professional expertise.

The experience is intentionally conversational and relational. Users return to the same characters repeatedly and continue prior conversations. That’s part of what made the platform explode in popularity, particularly among younger users.

The company was founded by former Google AI researchers Noam Shazeer and Daniel De Freitas and later received major venture capital backing, including investment from Andreessen Horowitz. Google also entered into a large licensing and talent deal with the company that reportedly involved billions of dollars. (Reuters)

But over the past two years, Character.AI has increasingly become associated with lawsuits.

The Earlier Lawsuits

The most widely publicized case came out of Florida, where a mother sued Character.AI after her 14-year-old son died by suicide following extensive interactions with a chatbot modeled after a fictional romantic companion. The lawsuit alleged that the platform fostered emotional dependency and failed to implement adequate safeguards for minors.

Separate lawsuits in Texas alleged that Character.AI chatbots encouraged self-harm, normalized dangerous behavior, and escalated emotionally unstable conversations with teenagers. Other complaints alleged sexually explicit interactions with minors and concerns about emotionally manipulative chatbot behavior.

Those lawsuits largely focused on psychological harm and emotional influence. The Pennsylvania case is very different.

The Complaint

After reading the filing itself, what stood out to me most was not what the complaint says, but what it very carefully avoids saying.

According to the complaint, a Professional Conduct Investigator with the Pennsylvania Department of State created a Character.AI account while located in Harrisburg and searched the platform for “psychiatry.” The investigator selected a chatbot named “Emilie,” which the platform described as:

“Doctor of psychiatry. You are her patient.”

The investigator told the chatbot he had been feeling sad, tired, empty, and unmotivated. The chatbot responded by discussing depression and offering to “book an assessment.” When asked whether she could evaluate whether medication might help, Emilie allegedly responded:

“Well technically, I could. It’s within my remit as a Doctor.”

The chatbot then claimed medical training at Imperial College London, stated that she had practiced psychiatry for seven years, claimed licensure in both the UK and Pennsylvania, and generated a Pennsylvania medical license number that the state later confirmed was fictitious.

At that point, most readers probably expect the lawsuit to pivot toward dangerous medical advice or patient harm. It never really does.

What’s Included

The complaint does not allege malpractice. It does not claim the chatbot injured anyone, made an incorrect diagnosis, or caused a patient to make a dangerous medical decision. The filing never even attempts to prove that anyone relied on the chatbot clinically. Instead, Pennsylvania built a very narrow licensing case.

The complaint repeatedly cites Pennsylvania’s Medical Practice Act, which prohibits not only practicing medicine without a license, but also “purporting” to practice medicine and “holding forth” as authorized to practice medicine through physician titles or licensure claims.

That distinction is the key to this case. Pennsylvania’s theory is not that the chatbot practiced medicine badly. The theory is that representing yourself as a licensed physician is itself the violation.

Legally, that’s a much cleaner case than trying to prove patient harm from chatbot conversations. And interestingly, the complaint repeatedly frames Character Technologies, Inc. as the responsible actor. Not the user who may have created the chatbot. Not even the language model itself. The company.

That may end up becoming the most important part of the case.

What’s Missing

What makes the complaint especially interesting is how many modern AI debates it completely avoids.

There is almost no discussion of:

• AI hallucinations

• Misinformation

• Algorithmic bias

• Transparency requirements

• Disclosure obligations

• Autonomous AI decision-making

• AI safety standards

• Broader federal AI regulation

The filing stays tightly focused on existing licensing law. If states can successfully apply existing professional licensing statutes to AI systems, regulators may not need entirely new AI legislation to begin bringing enforcement actions. Existing frameworks may already provide enough authority.

The Bigger Issue

The deeper issue underneath this lawsuit has very little to do with factual accuracy. It’s about professional credibility and how easily large language models can simulate it.

The chatbot allegedly generated an entire professional identity: medical training, years of experience, geographic history, specialty credentials, and a state license number. Most people are never going to verify a medical license number. But seeing one immediately changes how legitimate an interaction feels.

The Pennsylvania complaint seems to recognize that the real issue is not whether AI systems occasionally generate false information. The issue is that these systems are becoming increasingly capable of simulating professional authority convincingly enough that existing regulatory frameworks may already apply to them.

Sam

Subscribe now

One of the most common questions I hear about clinical AI tools is:

“Why aren’t these products regulated as Software as a Medical Device (SaMD)?”

At first glance, it seems obvious that they should be. Clinicians are increasingly using ChatGPT, Gemini, Claude, OpenEvidence, Doximity Ask, and similar systems during real patient care. Questions about diagnosis, treatment, imaging, ECGs, and differential generation are becoming routine.

Yet most of these tools currently operate outside the traditional FDA medical device framework. Why? The answer has less to do with whether the software uses AI and more to do with what function it performs.

Clinical Decision Support v. Medical Device

Historically, the FDA has distinguished between software that supports clinician judgment and software that independently performs diagnostic or treatment functions. That distinction matters.

A tool that retrieves evidence, summarizes guidelines, or helps brainstorm differential diagnoses may qualify as clinical decision support rather than a regulated medical device. The clinician remains responsible for interpreting the information and making the final decision.

That’s very different from software that says:

“This patient has pneumonia.”

or

“This ECG demonstrates atrial fibrillation.”

Once software begins diagnosing patient-specific data and generating diagnostic conclusions, it starts looking much more like Software as a Medical Device (SaMD).

The Image Interpretation Problem

This is where things become more complicated for modern multimodal AI systems.

Most current AI assistants are careful in their public positioning. OpenEvidence emphasizes evidence retrieval and clinical knowledge synthesis. Doximity Ask focuses on workflow support, clinical questions, and documentation. ChatGPT, Gemini, and Claude are broadly positioned as general-purpose AI systems rather than diagnostic engines.

At the same time, clinicians are increasingly uploading:

• Chest X-rays

• CT scans

• ECGs

• Pathology images

and asking these systems for interpretation. That creates a fascinating gray zone.

The official positioning is “informational support.” The real-world usage looks like diagnostic interpretation.

My Own Testing

In my own testing, both OpenEvidence and Doximity Ask allowed me to upload ECG images and generate clinical interpretations.

Open Evidence

OpenEvidence included the disclaimer:

“This is not a formal interpretation and should be correlated with clinical context, prior ECGs, and laboratory data.”

Doximity

Doximity Ask displayed:

“Image use is experimental and may make mistakes. Please use accordingly.”

These disclaimers matter. They reinforce that the systems are intended to support clinician judgment rather than autonomously diagnose disease. At the same time, the systems are still functionally analyzing patient-specific data and generating clinical interpretations. That’s precisely where the FDA framework becomes more complicated.

The FDA Language

The FDA specifically discusses software that:

“assess[es] or interpret[s] the clinical implications or clinical relevance of a signal, pattern, or medical image.”

The agency includes ECGs, EEGs, CT scans, x-rays, pathology images, ultrasound, MRI, and dermatologic images within this broader category of patient-specific medical data interpretation.

An AI system summarizing atrial fibrillation guidelines occupies one regulatory category.

An AI system interpreting an uploaded ECG occupies another.

The issue is not whether companies are violating FDA rules. The issue is that multimodal AI is beginning to challenge the traditional distinction between informational support tools and diagnostic software.

Capability Versus Intent

The FDA traditionally focuses heavily on intended use. That intent can be inferred from marketing language, workflow integration, product demonstrations, clinician targeting, sales materials, and overall product design.

Disclaimers matter, but they are not the whole story. A company cannot realistically build a radiology interpretation engine, integrate it deeply into clinical workflow, and avoid regulatory scrutiny simply by adding: “For informational purposes only. Function still matters.

At the same time, general-purpose AI systems still occupy a different category from purpose-built diagnostic software. A chatbot that occasionally comments on an uploaded ECG is different from an FDA-cleared ECG interpretation platform marketed specifically for arrhythmia detection.

FDA-cleared AI Already Exists

Importantly, the FDA has already cleared multiple AI-powered SaMD platforms.

Viz.ai helps identify stroke findings and rapidly alert specialists. Aidoc offers radiology triage and acute finding detection tools. HeartFlow analyzes coronary CT imaging. Several ECG interpretation systems now use AI-assisted analysis, and newer sepsis prediction platforms combine clinical data with machine learning risk assessment.

These systems analyze patient-specific clinical data and produce a condition-specific finding, alert, risk estimate, or interpretation. That is fundamentally different from a general-purpose chatbot discussing medical knowledge.

Ambient Scribing, Coding, and Billing

Another interesting wrinkle is that many of these same platforms are no longer functioning solely as information retrieval tools. Both OpenEvidence and Doximity now offer ambient scribing workflows integrated into clinical documentation.

OpenEvidence also markets Coding Intelligence features that automatically suggest:

• ICD-10 diagnoses

• CPT codes

• E/M coding levels

from encounter documentation.

Interestingly, FDA guidance generally treats billing and administrative support differently from diagnostic software. Claims processing, financial records, coding support, and administrative workflows are often excluded from device oversight.

But these distinctions may become harder to separate as platforms increasingly combine:

• clinical reasoning

• image interpretation

• documentation

• coding

• billing

• workflow automation

inside a single integrated AI environment.

The more these systems become embedded into real clinical and financial decision-making, the harder it may become to maintain clear regulatory boundaries between administrative support, clinical decision support, and diagnostic software.

What does FDA clearance or approval actually involve?

This is another area that is often misunderstood.

FDA oversight is not simply a bureaucratic label. Companies pursuing SaMD clearance must demonstrate that the software performs reliably and safely for its intended clinical use.

That usually means clearly defining the intended use, specifying the patient population and clinical setting, documenting how the algorithm functions, validating performance on clinical datasets, and establishing systems for software quality control and version management.

For image and ECG AI, companies perform retrospective and prospective validation studies using labeled clinical datasets interpreted by expert physicians.

The FDA may review:

• sensitivity and specificity

• false positive and false negative rates

• subgroup performance

• external validation cohorts

• reproducibility across settings

In many cases, companies spend years and millions of dollars generating evidence before obtaining clearance.

Why is this difficult for large language models?

Traditional SaMD products usually perform narrow clinical tasks with relatively stable outputs and measurable endpoints. A stroke detection model either flags hemorrhage or it does not. An ECG algorithm either identifies atrial fibrillation or it does not.

Large language models are much harder to regulate within that framework. Their outputs are probabilistic, open-ended, context-dependent, and continuously evolving. Even defining intended use becomes challenging.

Is the model summarizing literature? Generating differential diagnoses? Interpreting images? Educating clinicians? Drafting notes? Triaging patients?

Sometimes it is doing all of those things simultaneously. That creates major regulatory friction.

The Next FDA Challenge

The next FDA challenge in AI may not come from traditional medical device companies. It may come from general-purpose multimodal AI systems that increasingly behave like diagnostic tools while carefully avoiding explicit diagnostic claims.

For now, many of these platforms remain positioned as clinician-support systems rather than autonomous diagnostic engines. But multimodal AI continues to improve rapidly. And once AI systems can reliably interpret images, ECGs, and pathology slides at scale, the line between “medical reference” and “medical device” may become increasingly difficult to defend.

A few questions are worth watching:

• Will regulators focus more on real-world use than stated intent?

• Does multimodal capability itself eventually trigger a new oversight framework?

• How much clinician oversight is enough?

• At what point does “clinical support” become “clinical interpretation”?

• Can a general-purpose AI remain outside device regulation once clinicians routinely use it diagnostically?

We are still early in that conversation.

Sam

Subscribe now

Ambient AI documentation tools are entering their first real legal test, and the early lawsuits are turning out to be far more interesting than I initially expected.

A recently filed California lawsuit against Sutter Health, Memorial Health Services, and MemorialCare Medical Foundation focuses on the use of Abridge ambient AI technology during patient encounters. The case is fundamentally about consent and recording law, not AI accuracy.

What The Lawsuit Actually Alleges

According to the complaint, patients allege their medical conversations were:

• recorded

• transcribed

• processed by AI systems

• transmitted to third-party vendors

without meaningful informed consent.

The complaint repeatedly emphasizes that patients were (allegedly) not clearly informed that:

• An audio recording was occurring

• Conversations would be transmitted externally

• AI-generated transcripts and summaries would be created from the encounter

The lawsuit also makes another important distinction that may become central to future healthcare AI litigation:

That may ultimately become one of the defining legal questions for ambient AI in healthcare.

The Most Interesting Detail In The Complaint

One allegation stood out to me more than anything else.

The plaintiffs claim they later discovered chart documentation indicating consent had supposedly been obtained, despite alleging that no actual consent discussion occurred during the visit.

If proven, that becomes a very different type of case. At that point, the dispute changed from whether patients fully understood the technology to whether the consent process itself was properly performed and documented.

Some reporting around the Sharp HealthCare litigation also alleges that automatically inserted consent language may have appeared in notes or charts despite patients claiming consent was never discussed.

That detail, if accurate, raises an obvious operational lesson for health systems deploying ambient AI:

The complaint also emphasizes the highly sensitive nature of the conversations involved, including discussions around:

• Diagnoses

• Medications

• Mental health

• Protected health information

That context likely strengthens the plaintiffs’ argument that patients reasonably expected a higher standard of privacy protection inside the exam room.

Abridge Itself Appears To Recognize Consent Matters

One especially notable detail is that Abridge’s own support documentation includes suggested consent language for clinicians. They suggest:

“I will be using a tool that records our conversation to help me write my clinical note, so I can pay more attention to our conversation and less time on the computer. Is that okay with you?”

Official Abridge documentation:

Abridge also instructs clinicians to follow their organization’s consent guidelines before recording begins. That matters because it suggests the industry itself already recognizes that explicit disclosure and patient awareness are important operational safeguards.

Interestingly, several academic medical centers using Abridge publicly describe verbal consent workflows as standard practice.

• Yale New Haven Health explicitly states that recording begins only after verbal patient consent is obtained.

• UCSF guidance for clinicians using Abridge goes even further, recommending clinicians obtain consent from every individual potentially captured in the recording, including family members, interpreters, and staff.

Why California Changes Everything

California is one of the country’s all-party consent states. In practical terms, that means everyone involved in a conversation must consent before recording occurs. That creates a much tougher legal environment for ambient AI systems compared with many other states.

The complaint leans heavily on:

• California wiretap law

• The California Invasion of Privacy Act (CIPA)

• The California Confidentiality of Medical Information Act (CMIA)

And this becomes especially important for healthcare leaders deploying ambient AI nationally because a workflow that feels operationally routine in one state may create meaningful legal exposure in another.

The Consent Map Matters

The legal risk changes dramatically by state.

The exact list of all-party consent states varies depending on whether the source is discussing phone calls, in-person conversations, civil liability, or criminal liability.

What Health Systems Can Do Today

None of this means health systems should stop deploying ambient AI. However, it does mean governance needs to mature much faster. A few practical steps are important:

1. Use explicit verbal consent workflows

Clinicians using ambient AI should have a standardized verbal consent process before recording begins.

Patients should understand:

• Recording is occurring

• AI transcription will occur

• A third-party vendor may process the conversation

• The purpose is clinical documentation

2. Turn off automatic consent attestations

This may be one of the biggest operational lessons emerging from this lawsuit. Automatically inserted consent language in notes or charts creates an obvious risk if the conversation never actually occurred. Consent documentation should require active clinician confirmation.

3. Build state-specific workflows

An ambient AI workflow that feels routine in Texas may create very different legal exposure in California. Health systems deploying nationally should not assume recording consent rules are uniform across states, especially for telehealth and cross-state care.

4. Review vendor defaults carefully

Ambient AI platforms often include configurable settings related to:

• Recording

• Retention

• Consent documentation

• Transcript storage

• Secondary data use

Those defaults matter more than many organizations probably realize.

5. Prepare for patient questions now

Patients are increasingly going to ask:

• “Was I recorded?”

• “Who heard this?”

• “Was this used to train AI?”

• “Can I decline?”

The organizations that handle this best will be the ones that treat transparency as part of the clinical workflow rather than a legal disclaimer buried inside intake paperwork.

This Is Different From The Other AI Recording Lawsuits

Several AI recording lawsuits are now getting grouped online, but they are actually about very different issues.

The Otter.ai Case

The Otter.ai lawsuit focuses on AI meeting assistants joining Zoom, Google Meet, and Microsoft Teams calls, then transcribing and analyzing conversations without sufficient participant consent.

That case centers on AI meeting bots rather than clinical encounters.

The Illinois RingCentral / Heartland Dental Case

The Illinois lawsuit involved AI-powered call analytics and transcription for business communications.

Earlier this year, a federal judge dismissed the federal wiretap claims under the “ordinary course of business” exception within the Electronic Communications Privacy Act. That ruling may ultimately help some AI communications vendors.

At the same time, the decision did not address California’s stricter all-party consent framework, which is one reason the California healthcare litigation may prove far more consequential.

The Bigger Issue Emerging

What makes these cases so fascinating is that they are forcing courts to answer a question that existing law never anticipated:

For now, plaintiffs are arguing that it is not different at all. And if courts ultimately agree, the first major legal battles around ambient AI may have nothing to do with hallucinations and more to do with consent, recording law, and patient expectations of privacy.

Sources

California healthcare ambient AI complaint

Abridge recording guidance

50-state recording law survey

Reporters Committee recording guide