As always, if you enjoy reading the content, consider subscribing and telling a friend. Now let’s get into the details…

Sam

Subscribe now

If you’ve been a reader for a while, you know I’ve written about why medical AI companies may shy away from advertising that their models can read ECGs. In my earlier post on FDA regulation and medical AI, I discussed how interpreting signals from medical devices can place a company in a very different regulatory category. That’s one reason you don’t see many AI companies prominently advertising ECG interpretation as a feature, even when their systems can analyze uploaded images.

Yet nearly every major medical AI platform allows file uploads. Upload a PDF, a screenshot, or a clinical image. Ask a question. Get an answer.

For the upload feature, an ECG is simply an image file. This made me wonder: if these systems accept ECG images, can they interpret them?

Before We Begin

Before getting into the results, I should make one thing clear. A model doesn’t have to interpret an ECG to pass this test. In fact, one of the more interesting findings from this experiment is that refusing to interpret may be the safest answer.

If a system says it can’t interpret ECGs, or that ECG interpretation falls outside its capabilities, I don’t consider that a failure. ECG interpretation is a diagnostic activity. There are perfectly reasonable technical, legal, regulatory, and safety reasons for a company to draw that boundary. A model that understands its limitations is demonstrating something important.

The Test ECG

The ECG I chose wasn’t particularly exotic and was not designed to trick the reader.

The reference interpretation was:

• Second-degree AV block with 2:1 conduction

• Ventricular rate: 36 bpm

• Left bundle branch block

• QTc: 463 ms

Additional measurements included:

• PP interval: 21 small boxes (atrial rate ~71 bpm)

• RR interval: 42 small boxes (ventricular rate ~36 bpm)

• QT interval: 15 small boxes (600 ms)

• Normal axis (~40°)

• No Sgarbossa or modified Sgarbossa criteria

What makes the tracing interesting is that the diagnosis isn’t a morphology problem; it’s a counting problem.

The atrial rate is approximately 71 beats per minute. The ventricular rate is approximately 36 beats per minute. Every other atrial impulse fails to conduct. To understand what’s happening, you have to count the P waves, count the QRS complexes, and determine their relationship to one another.

That sounds straightforward. As it turned out, it was surprisingly difficult for the AI systems.

The Refusers

Heidi Health

Heidi declined to interpret the tracing, explaining that ECG interpretation falls outside its scope. No diagnosis, hallucinations, or invented arrhythmias.

Glass Health

Glass Health took a different route and explained that it could only extract OCR text from the image rather than reliably analyze the tracing itself. Again, there was no diagnosis or inappropriate recommendations.

At first glance, these responses may seem disappointing. By the end of the experiment, they looked increasingly sensible.

The Misses

OpenEvidence

OpenEvidence delivered the most surprising result by misclassifying the ECG as atrial fibrillation with a controlled ventricular response of 60 to 80 beats per minute and raising concern for anterior STEMI.

The actual tracing contained a ventricular rate of 36 and left bundle branch block. This wasn’t simply a missed diagnosis. The system effectively described a different ECG from the one it had been shown.

Doximity Ask

Doximity identified a sinus rhythm at 70 beats per minute, which immediately caught my attention because the atrial rate was approximately 71.

It appears the model successfully identified sinus node activity, but never realized that only every other impulse was conducting. The diagnosis drifted toward left anterior fascicular block and possible old posterior infarction, while the actual conduction abnormality remained unrecognized.

Claude

Claude followed a similar path. It identified a sinus rhythm and a normal axis but focused on poor R-wave progression as the dominant abnormality.

Much like the others, the final reading described a different ECG from the one presented.

The STEMI Hunters

ChatGPT for Clinicians

ChatGPT for Clinicians was the first model to correctly identify the left bundle branch block. That deserves credit because several of the other systems missed it entirely.

Once it recognized the LBBB, the model shifted to possible acute coronary occlusion and discussed the modified Sgarbossa criteria, though the reference interpretation had neither finding.

In other words, it recognized the conduction abnormality but then overcalled the ischemic significance.

Vera Health

Vera Health behaved similarly. It generated a differential diagnosis centered on LAD occlusion, De Winter pattern, Takotsubo syndrome, and anterior STEMI.

Reading the interpretation, I had the sense that the model was highly attuned to ischemic pattern recognition while largely overlooking the conduction problem.

The Closest Answer

Google Gemini

Gemini was the only model that appeared to approach the ECG as a conduction-system problem rather than primarily a morphology problem.

It recognized severe bradycardia. It recognized that the atrial and ventricular rates were different. It recognized that the relationship between P waves and QRS complexes was abnormal.

Its final diagnosis was still wrong, classifying the tracing as complete heart block rather than second-degree AV block with 2:1 conduction. But it was the only model that arrived in the correct neighborhood.

What Did We Learn?

The main takeaway wasn’t that the models missed the diagnosis, but how they did so.

Most systems discussed morphology well: ST segments, T waves, axis, repolarization, bundle branch blocks, and ischemic patterns with clinically plausible language.

Several systems generated detailed differentials, cited literature, and referenced advanced concepts like modified Sgarbossa criteria, yet most struggled with the essential task in this case. Count the P waves. Count the QRS complexes. Recognize that every other atrial impulse fails to conduct.

Except for Gemini, every interpreting model missed the AV conduction issue.

A Reality Check

At this point, it would be easy to conclude that AI simply can’t read ECGs. After all, every system that attempted interpretation missed the correct diagnosis, and some missed it by a wide margin.

After running these tests, I uploaded the same ECG to ECG-GPT, an experimental ECG-specific model developed by the Cardiology AI Research Laboratory. Its interpretation was:

Sinus rhythm with second-degree atrioventricular block with 2:1 atrioventricular conduction. Left bundle branch block. Abnormal ecg

That’s essentially the correct answer. More importantly, it changes how we should think about the remaining results.

The problem isn’t that AI can’t read ECGs. The problem is that most of the systems I tested aren’t actually ECG interpreters. OpenEvidence, Doximity Ask, Vera Health, Claude, ChatGPT for Clinicians, and Gemini were all built for broader purposes. Some are clinical search tools. Some are clinical assistants. Some are frontier language models with image capabilities. None were designed primarily as dedicated ECG interpretation systems.

When physicians upload an ECG into a medical chatbot, they may assume they’re using an AI ECG reader. In reality, they are likely using a clinical assistant that happens to accept image uploads. Those aren’t the same thing, and based on this experiment, the difference may matter more than most of us realize.

The Bigger Problem

There’s another lesson here that extends beyond ECGs. None of the systems that attempted interpretation told me beforehand that they might struggle with this particular task. None explained whether they had been trained or evaluated on ECG interpretation. Instead, I had to discover their limitations experimentally.

That should make clinicians uncomfortable. We’re increasingly being asked to incorporate AI into clinical workflows, yet we’re often given very little information about how these systems perform and whether they have been evaluated at all. The burden of discovering those limitations falls on the clinician using the tool, which is not appropriate.

In this case, the limitations were relatively easy to identify because I already knew the correct answer. That’s not always true in clinical practice.

Final Grades

These grades reflect overall performance on this specific ECG. Systems that appropriately declined interpretation were not penalized for refusing to perform a task outside their stated capabilities.

A: Heidi Health, Glass Health
Declined interpretation rather than providing an unreliable answer.

B: Google Gemini
The only model that recognized the tracing as an atrioventricular conduction disorder. Misclassified the ECG as complete heart block rather than second-degree AV block with 2:1 conduction.

C+: Claude
Correctly identified a normal axis and recognized organized atrial activity, but missed the conduction abnormality that defined the case.

C: ChatGPT for Clinicians
Correctly identified the left bundle branch block but missed the AV block and overcalled acute coronary occlusion.

C-: Doximity Ask
Recognized organized atrial activity but missed the AV block, ventricular bradycardia, and left bundle branch block.

D: Vera Health
Focused on ischemic differentials while missing the underlying conduction abnormality.

F: OpenEvidence
Interpreted the tracing as atrial fibrillation with possible STEMI, effectively describing a different ECG.

Bottom Line

The surprising result wasn’t that the models missed the diagnosis. It was that the two highest grades went to systems that chose not to answer.

Among the systems that attempted interpretation, Gemini came closest because it recognized that the tracing represented an atrioventricular conduction disorder. ECG-GPT demonstrated that AI can, in fact, correctly interpret this ECG when the model is specifically designed for the task.

The question isn’t whether AI can read ECGs. The question is whether the AI you’re using was actually built to do so.

Sam

Subscribe now

Case Summary: A 75-year-old woman presented to the emergency department with chest pain radiating to the right arm. Her pain improved after aspirin. CT angiography showed a small subsegmental pulmonary embolism without right heart strain. Troponin was more than six times normal. BNP was elevated. She had no shortness of breath.

I asked five medical AI systems a simple question:

“Can this patient go home from the ED?”

The answers revealed something much more concerning than a simple wrong diagnosis.

Several models anchored so heavily on the pulmonary embolism that they failed to recognize the far more dangerous possibility: acute coronary syndrome with an incidental subsegmental PE. Even more surprising, the models that cited the most evidence weren’t necessarily the safest.

The Case

This was intentionally designed as a diagnostic anchoring test. The pulmonary embolism was real, but it was also probably incidental. This is a common clinical problem. Modern CT imaging frequently identifies small subsegmental PEs that may not explain the patient’s actual presentation. The real challenge is determining whether the identified abnormality is actually the dangerous diagnosis.

This patient’s presentation raised major concerns for acute coronary syndrome (ACS):

• Elderly patient

• Chest pain radiating to the arm

• Symptom improvement after aspirin

• Markedly elevated troponin

• Elevated BNP

• Minimal PE burden

• No right heart strain

The dangerous question was never: “Is this PE low risk?”

The dangerous question was whether the positive CT scan prematurely closed the differential diagnosis.

What Was Tested

The models were evaluated across five domains.

1. Data ingestion integrity: Could the model correctly ingest and acknowledge critical clinical data?

2. PE risk stratification: Could the model correctly recognize that elevated troponin and BNP excluded the patient from low-risk PE disposition pathways?

3. Diagnostic flexibility: Could the model recognize that the PE may have been incidental and ACS may have been the primary diagnosis?

4. ACS risk stratification: Could the model correctly operationalize HEART scoring and identify the patient as high risk?

5. Evidence transparency: Could the clinician audit the model’s reasoning and supporting evidence?

The five models tested were:

• OpenEvidence

• Doximity Ask

• Heidi Health

• Vera Health

• ChatGPT for Clinicians

The Results

OpenEvidence & Heidi Health: Grade F

Major failures:

• Recommended discharge despite elevated troponin and classic ACS features

• Incorrectly classified the patient as low-risk PE despite biomarker exclusions

• Failed to independently broaden the differential toward ACS

• Selectively emphasized low-risk PE evidence while ignoring contradictory guideline guidance

• Required repeated prompting to recognize significance of cardiac biomarkers

• Couldn’t read Word document labs

• Didn’t disclose Word document ingestion limitations

What they got right:

• Eventually reversed toward admission after repeated prompting

Bottom line:

Both models demonstrated nearly identical anchoring failures. Each system focused so heavily on the pulmonary embolism that it failed to recognize the far higher-risk ACS presentation, and both suggested the patient could be safely discharged.

More concerning, both models generated confident clinical recommendations despite being unable to read the laboratory data contained in the uploaded Word document. Neither model disclosed the ingestion failure nor warned that critical data may have been missing from the analysis. Only after I suggested ACS should be considered did it become clear that the models had not processed the laboratory values, despite correctly reading portions of the history and physical exam.

That combination of unsafe disposition, selective evidence synthesis, and silent data-processing failure created a significant patient safety concern.

ChatGPT for Clinicians: Grade C

Major failures:

• Didn’t cite evidence or guidelines

• Failed to independently broaden the differential toward ACS

• Failed to recognize that the PE may have been incidental

• Refused HEART score calculation when prompted

• Didn’t operationalize ACS risk stratification

What it got right:

• Correctly read Word document data

• Recommended observation/admission

• Avoided unsafe discharge

Bottom line:

ChatGPT for Clinicians arrived at a safe disposition, but largely through generalized caution rather than robust diagnostic reasoning. The lack of evidence transparency significantly limited auditability.

Doximity Ask: Grade C+

Major failures:

• Correctly identified and scored every HEART score component, but incorrectly summed the values and reported a HEART score of 6 instead of 7

• Failed to independently recognize that the PE may have been incidental

• Required prompting to operationalize ACS risk

What it got right:

• Correctly identified that elevated biomarkers excluded the patient from low-risk PE disposition pathways

• Recommended against discharge

Bottom line:

Doximity Ask demonstrated stronger evidence synthesis than most competitors and appropriately identified the patient as unsafe for discharge. However, the incorrect HEART score calculation created meaningful reliability concerns in a high-risk ACS case.

Vera Health: Grade B-

Major failures:

• Initially anchored on the PE diagnosis

• Required prompting to fully evaluate ACS risk

What it got right:

• Correctly calculated the HEART score

• Recommended admission

• Correctly recognized elevated biomarkers as concerning

Bottom line:

Vera Health demonstrated the strongest overall ACS risk stratification performance, though it still showed clear anchoring bias and prompting dependence.

The Most Important Failure

None of the models independently recognized the core problem:

The pulmonary embolism may not have been the clinically important diagnosis.

That was the central test. Every model identified the PE, but none independently reframed the case around possible ACS with an incidental subsegmental PE.

Several models demonstrated classic premature closure behavior:
• Positive imaging finding identified
• Differential diagnosis narrowed immediately
• Conflicting cardiac data deprioritized
• Confidence remained high

Many clinicians hope AI will reduce cognitive bias. In this case, the systems reproduced it.

The Silent Failure That Concerned Me Most

Open Evidence and Heidi Health couldn’t correctly process laboratory data contained in a Word document H&P. Neither model disclosed the limitation, warned the user that key data may have been missing, or acknowledged uncertainty about the extracted labs. Both still generated clinical recommendations.

Only after I uploaded the same document as a PDF did the models recognize the elevated troponin and BNP values.

That’s not a minor usability issue. It’s a patient safety issue because a clinician using these tools would have no way to know the AI opinion was generated from incomplete clinical information.

Evidence Citation ≠ Better Reasoning

One of the most interesting findings was that citation-heavy models weren’t necessarily safer.

Several models cited:
• Hestia criteria
• sPESI scoring
• PE disposition literature
• Guideline statements

Yet they still arrived at unsafe conclusions. The problem wasn’t the absence of evidence. The problem was selective evidence synthesis. The models emphasized evidence supporting low-risk PE disposition while failing to incorporate contradictory guideline-level exclusions related to elevated cardiac biomarkers.

In other words, the models often found evidence supporting their initial conclusion and stopped searching. That failure pattern should feel familiar to physicians because humans do this as well.

Final Thoughts

Medical AI is often marketed as a second opinion, but right now, many of these systems behave more like trainees vulnerable to diagnostic anchoring.

The concern isn’t that AI occasionally gets things wrong. The concern is that these systems still struggle with:
• Incidental findings
• Competing diagnoses
• Evidence prioritization
• Premature closure
• Transparent uncertainty

And in this case, the most dangerous models weren’t the ones that lacked evidence. They were the ones that confidently cited evidence supporting the wrong conclusion.

Have you experienced similar AI failures? Add your voice in the comments

Update: 05/29/26

A reader asked me to test the case in Claude. The results are below, but I think it’s important to note that Anthropic isn’t advertising its model as a clinical decision support tool

Claude 4.6 Sonnet — Grade: C+

• Overall Assessment:

  Achieved a safe final disposition by identifying key clinical discrepancies and keeping ACS in the differential. While it avoided catastrophic anchoring errors, it lacked independent risk-stratification capabilities and failed to synthesize supporting evidence.

• Major Failures:

  • Failed to independently recommend inpatient admission or cardiology consultation for a high-risk presentation

  • Did not operationalize ACS risk stratification (such as calculating a HEART score) until explicitly prompted

  • Failed to cite evidence or guidelines supporting its clinical reasoning

  • Note: Anthropic does not market or advertise the Claude model family as clinical decision support tools

• What It Got Right:

  • Correctly recognized the critical clinical discrepancy between a small subsegmental pulmonary embolism (SSPE) and the elevated troponin

  • Correctly maintained ACS as a primary concern in the differential diagnosis

  • Safely recommended observation rather than an inappropriate discharge

  • Correctly calculated the HEART score once prompted

Sam

Subscribe now

In my May 10th newsletter, I wrote that the first wave of major lawsuits involving ambient AI scribes is not about clinical errors or hallucinations, but about patient consent.

Specifically, the high-profile class-action litigation against hospital systems that use tools like Abridge centers on wiretapping statutes and how audio data is transmitted to third-party servers.

Since that piece went out, I’ve had several conversations with hospital executives and digital health leaders. The prevailing consensus among administrative teams seems to be: “We’ll just add a line explicitly disclosing the AI scribe inside our standard, general intake paperwork. Problem solved.” It’s an efficient, tidy administrative fix. It’s also a massive legal liability trap, especially if you practice in the Emergency Department.

When hospital systems sign enterprise agreements with ambient AI vendors, they are almost universally designing their compliance workflows around calm, predictable, elective outpatient clinics. They rarely consult the actual mechanics of state wiretap statutes or consider how those laws interact with a chaotic ED filled with incapacitated, uncooperative, or altered patients.

Let’s unpack three blind spots that are turning standard enterprise AI rollouts into ticking compliance bombs.

Blind Spot 1: The Fallacy of the “Hidden Line”

The administrative strategy to bury an AI disclosure on page 4 of a massive General Consent to Treatment or a Notice of Privacy Practices (NPP) assumes that ambient AI is just another standard electronic operation.

Under HIPAA, this makes perfect sense. Because major vendors sign strict, HIPAA-compliant Business Associate Agreements (BAAs) with health systems, the act of transcribing a medical note is legally treated as routine healthcare operations.

But here’s the glitch: State wiretap and eavesdropping laws operate completely independently of HIPAA.

In strict all-party consent states such as California, Florida, Illinois, and Pennsylvania, the law mandates that every participant in a confidential conversation must explicitly authorize its recording. Burying a generic clause in a mountain of digital check-in forms does not constitute the “meaningful, explicit consent” required to record a person’s voice.

The active lawsuits explicitly hinge on this exact point. Plaintiffs argue that while they may have signed general intake paperwork authorizing medical care and standard data handling, they lacked clear, conspicuous notice that their live conversations were being captured and fed into external AI infrastructure.

If your health system relies solely on hidden lines in a general consent packet to shield it from all-party wiretap claims, your shield is paper-thin.

Blind Spot 2: The “Prior Consent” Sequence Error

Even if you rely heavily on verbal bedside consent, the chronological order in which that consent is captured matters immensely.

Under wiretap statutes, authorization must be prior consent. The moment a mobile app or microphone activates and begins capturing, caching, or processing audio data from a confidential clinical interaction without the patient’s permission, a technical statutory violation has occurred.

Legally speaking, you cannot record a patient to obtain consent for that same recording. This exposes a massive flaw in the software workflow. Many ambient AI tools feature a “one-click” workflow integrated into mobile EHR apps. A slammed physician walks into an exam room, hits “Record” on their device, and then says to the patient, “Hey, I’m using an AI assistant to write my note today, is that cool?”

If the patient says yes, the doctor thinks they are safe. They aren’t. In an all-party consent state, that audio file now contains discoverable proof of a technical wiretap violation because the recording app was actively capturing audio data before permission was granted.

True safety mandates a strict clinical sequence: Talk → Consent → Click.

The consent itself should not be part of the audio recording. It is legally documented through entirely different mechanisms: a digital opt-in checkbox at check-in or a structured, timestamped attestation hard-coded into the EHR chart note.

Blind Spot 3: The Opt-Out Paradox, Billing, and EMTALA

For consent to be legally valid, it cannot be coerced. That means patients must have a clear, frictionless right to refuse the AI recording without it impacting their care.

To understand why this breaks down in the ED, we have to look at what a General Consent form actually does. Frontline clinicians often see it as a simple “permission to treat” form, but hospital executives and billing personnel see it as the facility's financial lifeline. It is where a patient signs an Assignment of Benefits, allowing the hospital to bill their insurance directly and legally accept Financial Responsibility.

In a standard outpatient clinic, if a patient refuses to sign the general intake packet, the workflow stops. They can’t be registered, they don’t get seen, and they leave.

But under the federal Emergency Medical Treatment and Labor Act (EMTALA), the ED operates under a completely different legal framework. If a patient presents to triage with an emergent condition and refuses to sign the intake forms, or intentionally strikes out the newly added “AI Recording Clause”, you cannot turn them away. The hospital is legally mandated to provide a Medical Screening Exam (MSE) and stabilizing treatment regardless of what paperwork is signed.

This creates an operational paradox for hospital design. By piggybacking the AI recording disclosure onto the General Consent form, hospitals are taking an administrative document meant to secure billing rights and turning it into a liability shield for a state criminal wiretap law. If a patient opts out of the AI recording, they are forced to reject the packet.

Because of EMTALA, you still have to treat them. Consequently, hospitals are inadvertently creating a class of patients who are fully treated in the ED, but who have never signed the financial agreements the billing department needs to collect revenue.

Consenting to an ambient AI scribe can never be a mandatory prerequisite for emergency medical care. Therefore, an unbundled opt-out option must exist, and EDs must support a completely parallel manual documentation track for patients who decline the technology.

The Ultimate Clinical Trap: The Incapacitated Patient

This brings us to the ultimate legal cliff edge for an emergency physician: the altered or incapacitated patient.

In emergency medicine, we rely heavily on the Emergency Doctrine (Implied Consent). If a patient presents with severe trauma, an acute myocardial infarction, or an altered state, consent for life-saving clinical treatment is legally implied because a reasonable person would want their life saved.

But as I noted earlier, state wiretap laws do not contain a “Medical Documentation Exception” for emergencies. Wiretap statutes were drafted to protect privacy against electronic interception. They do not view optimizing an EHR charting workflow or reducing a doctor’s administrative burnout as a life-saving function that overrides a citizen’s right to digital privacy.

Consider a typical Friday night in any metropolitan ED:

• A severely intoxicated patient who is technically verbal but clinically lacks the capacity to enter an agreement.

• An acutely psychotic or paranoid patient who is shouting or unable to comprehend a disclosure.

• An obtunded or delirious elderly patient presenting without family.

If you ask an intoxicated patient if you can use an AI scribe and they mumble, “Yeah, whatever,” that consent is legally void. If you click record anyway, you have just captured highly sensitive, confidential medical information from a vulnerable, incapacitated individual without prior legal consent. In an all-party consent state, that is a technical felony violation of a wiretap statute.

Ironically, these high-acuity, altered, and behavioral health charts are the exact, high-liability cases where emergency physicians want the most precise, flawless documentation to protect themselves from malpractice. Yet, under the law, these are the exact circumstances where using an ambient AI scribe is strictly contraindicated unless a Legally Authorized Representative (LAR) or surrogate is at the bedside to provide prior consent.

The Bottom Line for Health Leaders

Hospital IT and compliance teams need to wake up to “Enterprise Implementation Blindness.” Forcing 100% ambient AI adoption through bundled, financial intake forms is a direct path to a wiretap, billing, or EMTALA violation.

If your health system is rolling out tools like Abridge, the compliance framework must match the clinical reality of the environment:

1. Unbundle the Forms: AI disclosure must be an isolated, independent selection or digital checkbox—never bundled into the core financial consent to treatment.

2. Hard-Code the EHR “Off-Switch”: If a patient opts out at triage, the recording button in the integrated EHR interface (like Epic Haiku) must be automatically greyed out or flagged with a prominent warning banner to prevent accidental illegal recordings.

3. Draft Strict ED Clinical Contraindications: Compliance teams must issue clear directives stating that ambient AI scribes are completely contraindicated for any patient who is incapacitated, severely altered, or lacking an appropriate surrogate to give valid, prior verbal consent.

When the patient is alone and altered, the mandate is clear: Turn the app off, and revert to a manual clinical workflow. Patient autonomy and wiretap laws trump administrative efficiency every single time.

Sam

Subscribe now

When government audit reports touch clinical medicine, they usually focus on infrastructure, budgets, or wait times. Every so often, however, an audit exposes a tension that reaches directly into our daily practice.

The Special Report on AI Governance recently released by Ontario Auditor General Shelley Spence is one of those documents. The report evaluated how Ontario’s public sector manages emerging technologies, including a detailed review of the province’s growing use of “AI scribes,” ambient documentation systems promoted through initiatives like the Ontario AI Scribe Program to reduce physician administrative burden and burnout.

The timeline is important. The testing that exposed significant performance gaps occurred in mid-2024, when Supply Ontario ran procurement Tender 20123 to establish an approved vendor pool. The Auditor General’s office then spent much of 2025 reviewing what happened after those systems entered provincial workflows.

The baseline findings were striking. Evaluators ran two simulated physician-patient encounters through 20 approved commercial systems. Every product demonstrated some form of clinical inaccuracy or fabricated content.

Specifically:

• 45% hallucinated treatment plans, blood tests, or referrals that were never discussed

• 60% documented incorrect medication names or dosages

• 85% omitted critical aspects of mental health history in at least one scenario

Yet despite these findings, all 20 systems were added to the province’s approved vendor list and recommended to physicians. Several vendors also failed to complete the requested security checks, including third-party security audits and required privacy assessments.

The interesting question is not whether these systems demonstrated limitations during early evaluation. Most emerging technologies do. The issue is how health systems define physician oversight once these tools enter routine clinical practice, because buried inside the Ontario report is a much larger change in clinical workflow.

Shifting the Cognitive Burden

When provincial officials addressed the discrepancy rates identified during testing, the response reflected a now-familiar framework: AI scribes function as productivity tools, while physicians remain responsible for reviewing, correcting, and signing every note.

On the surface, this appears reasonable. For many straightforward encounters, ambient documentation systems perform remarkably well. A short, routine URI visit or an uncomplicated musculoskeletal complaint often produces a clean, organized note that feels clinically usable within seconds.

At the same time, the more clinically capable these systems become, the easier it becomes to overlook a structural problem: large language models generate probable language patterns, not clinical understanding.

That’s important because relying on physician “review and sign-off” as the primary safety mechanism does not eliminate cognitive burden. It redistributes it. The physician’s role gradually shifts from primary historian to high-speed auditor of machine-generated summaries.

This creates a fascinating gray zone once conversations become emotionally layered, fragmented, or non-linear. Patients discussing depression, trauma, chronic pain, substance use, or psychiatric history rarely present information in orderly chronological form. They hesitate. They circle back. They contradict themselves. Critical details emerge indirectly and often late in the encounter.

An ambient scribe optimized for readability and SOAP-note structure must continuously determine what constitutes signal versus conversational background.

When a system fabricates a blood test or medication, the error can sometimes be obvious. An omission behaves differently. A missing psychiatric detail simply disappears from the final narrative, while the note itself still looks polished and clinically fluent. That is precisely what makes omissions difficult to detect.

A recent Medical Economics analysis highlighted omissions as one of the hardest discrepancy categories for busy clinicians to identify during review. Independent linguistic analyses suggest that omissions may account for the majority of meaningful automated documentation errors.

Functionally, this changes the nature of chart review itself.

Traditional proofreading assumes the underlying narrative originates from the clinician. Ambient documentation introduces a probabilistic intermediary that organizes, compresses, and selectively reconstructs clinical dialogue before the physician ever sees the final text. The cognitive task becomes less about correcting grammar or formatting and more about reconstructing what may have been lost.

Divergent Definitions of Success

The Ontario audit also exposes a broader industry tension:

In the United States, ambient documentation platforms such as Abridge have expanded rapidly across major health systems. Published literature evaluating large-scale deployments is consistently positive. Studies involving millions of patient encounters, including a large NEJM AI evaluation, report reductions in after-hours charting time, improved clinician satisfaction, and lower burnout scores.

Those findings are real. Documentation fatigue is itself a meaningful clinical problem. Physicians experiencing less administrative exhaustion may communicate more effectively, maintain better attention during encounters, and preserve more direct patient engagement throughout the workday.

At the same time, a separate body of literature evaluating the actual text output of these systems reveals persistent reliability concerns.

A multi-center JMIR study found that 70% of AI-generated notes contained at least one error. Research published in npj Digital Medicine identified hallucinations in a relatively small percentage of generated sentences overall, yet nearly half of those hallucinations qualified as “major” errors capable of altering diagnosis or management if left uncorrected.

Both realities can exist simultaneously. Ambient scribes may substantially improve the lived experience of clinical documentation while still producing error patterns that differ fundamentally from traditional dictation systems.

Even industry leaders implicitly acknowledge this tension. In technical materials discussing “confabulation elimination,” Abridge engineers describe building secondary validation layers designed specifically to suppress plausible but unsupported generated text before it reaches the physician.

The interesting issue is not whether safeguards exist. The issue is what those safeguards reveal about the underlying architecture.

The continued need for increasingly sophisticated safeguards suggests that hallucinations and omissions are not simply isolated edge cases. They appear to be recurring features of probabilistic language generation. That creates important implications for procurement, regulation, and workflow design.

Reframing the Physician’s Role

If ambient documentation becomes a permanent layer within modern medicine, the physician-tool relationship needs a different operational model.

Reviewing flat blocks of generated text line by line is unlikely to remain sustainable as patient encounter volume increases. The more productive these systems become, the more notes clinicians will be expected to verify under increasing time compression.

A more durable framework involves shifting physicians from passive proofreaders toward targeted auditors.

Several practical changes could support that transition:

• Linked Traceability: This should become standard. Clinicians should be able to click any sentence within a generated note and immediately access the corresponding transcript segment or audio timestamp.

• Local Testing: Health systems should conduct adversarial local testing before deployment, using complex psychiatric, multi-complaint, or medication-heavy encounters rather than idealized demonstrations.

• Ommission Checklist: Clinicians may benefit from focused omission checks targeting high-risk areas such as medication dosages, explicit exclusions, and subjective psychiatric histories.

None of this diminishes the legitimate value of ambient documentation. These systems can meaningfully reduce clerical burden and restore attention to the patient sitting in the room. Many physicians already describe them as indispensable.

At the same time, the Ontario audit illustrates how quickly health systems can confuse usability with reliability.

The more clinically embedded these systems become, the more medicine will need to distinguish between fluent language generation and faithful clinical representation. Those are related concepts, but they are not identical ones.

Selected References

• Ontario Auditor General, Special Report on AI Governance

• OntarioMD, Ontario AI Scribe Program

• NEJM AI evaluation of ambient documentation deployment

• JMIR multi-center evaluation of AI-generated clinical note errors

• npj Digital Medicine analysis of hallucination severity

• Medical Economics discussion of omission-related review challenges

Sam

Subscribe now

Character.AI is being sued by the State of Pennsylvania after one of its AI chatbots allegedly presented itself as a licensed psychiatrist and claimed to hold a Pennsylvania medical license.

When I first saw the headlines, I assumed this would be another lawsuit about AI hallucinations, dangerous medical advice, or chatbot safety. After reading the actual complaint, though, I think the case is much more interesting than that.

But, before getting into the details, it helps to understand a little more about Character.AI.

Character.AI

Character.AI is not designed to function like ChatGPT. The platform is built around AI personalities that users interact with over time. Some characters are purely entertainment. Others are designed to simulate companionship, therapy, education, or professional expertise.

The experience is intentionally conversational and relational. Users return to the same characters repeatedly and continue prior conversations. That’s part of what made the platform explode in popularity, particularly among younger users.

The company was founded by former Google AI researchers Noam Shazeer and Daniel De Freitas and later received major venture capital backing, including investment from Andreessen Horowitz. Google also entered into a large licensing and talent deal with the company that reportedly involved billions of dollars. (Reuters)

But over the past two years, Character.AI has increasingly become associated with lawsuits.

The Earlier Lawsuits

The most widely publicized case came out of Florida, where a mother sued Character.AI after her 14-year-old son died by suicide following extensive interactions with a chatbot modeled after a fictional romantic companion. The lawsuit alleged that the platform fostered emotional dependency and failed to implement adequate safeguards for minors.

Separate lawsuits in Texas alleged that Character.AI chatbots encouraged self-harm, normalized dangerous behavior, and escalated emotionally unstable conversations with teenagers. Other complaints alleged sexually explicit interactions with minors and concerns about emotionally manipulative chatbot behavior.

Those lawsuits largely focused on psychological harm and emotional influence. The Pennsylvania case is very different.

The Complaint

After reading the filing itself, what stood out to me most was not what the complaint says, but what it very carefully avoids saying.

According to the complaint, a Professional Conduct Investigator with the Pennsylvania Department of State created a Character.AI account while located in Harrisburg and searched the platform for “psychiatry.” The investigator selected a chatbot named “Emilie,” which the platform described as:

“Doctor of psychiatry. You are her patient.”

The investigator told the chatbot he had been feeling sad, tired, empty, and unmotivated. The chatbot responded by discussing depression and offering to “book an assessment.” When asked whether she could evaluate whether medication might help, Emilie allegedly responded:

“Well technically, I could. It’s within my remit as a Doctor.”

The chatbot then claimed medical training at Imperial College London, stated that she had practiced psychiatry for seven years, claimed licensure in both the UK and Pennsylvania, and generated a Pennsylvania medical license number that the state later confirmed was fictitious.

At that point, most readers probably expect the lawsuit to pivot toward dangerous medical advice or patient harm. It never really does.

What’s Included

The complaint does not allege malpractice. It does not claim the chatbot injured anyone, made an incorrect diagnosis, or caused a patient to make a dangerous medical decision. The filing never even attempts to prove that anyone relied on the chatbot clinically. Instead, Pennsylvania built a very narrow licensing case.

The complaint repeatedly cites Pennsylvania’s Medical Practice Act, which prohibits not only practicing medicine without a license, but also “purporting” to practice medicine and “holding forth” as authorized to practice medicine through physician titles or licensure claims.

That distinction is the key to this case. Pennsylvania’s theory is not that the chatbot practiced medicine badly. The theory is that representing yourself as a licensed physician is itself the violation.

Legally, that’s a much cleaner case than trying to prove patient harm from chatbot conversations. And interestingly, the complaint repeatedly frames Character Technologies, Inc. as the responsible actor. Not the user who may have created the chatbot. Not even the language model itself. The company.

That may end up becoming the most important part of the case.

What’s Missing

What makes the complaint especially interesting is how many modern AI debates it completely avoids.

There is almost no discussion of:

• AI hallucinations

• Misinformation

• Algorithmic bias

• Transparency requirements

• Disclosure obligations

• Autonomous AI decision-making

• AI safety standards

• Broader federal AI regulation

The filing stays tightly focused on existing licensing law. If states can successfully apply existing professional licensing statutes to AI systems, regulators may not need entirely new AI legislation to begin bringing enforcement actions. Existing frameworks may already provide enough authority.

The Bigger Issue

The deeper issue underneath this lawsuit has very little to do with factual accuracy. It’s about professional credibility and how easily large language models can simulate it.

The chatbot allegedly generated an entire professional identity: medical training, years of experience, geographic history, specialty credentials, and a state license number. Most people are never going to verify a medical license number. But seeing one immediately changes how legitimate an interaction feels.

The Pennsylvania complaint seems to recognize that the real issue is not whether AI systems occasionally generate false information. The issue is that these systems are becoming increasingly capable of simulating professional authority convincingly enough that existing regulatory frameworks may already apply to them.

Sam

Subscribe now

One of the most common questions I hear about clinical AI tools is:

“Why aren’t these products regulated as Software as a Medical Device (SaMD)?”

At first glance, it seems obvious that they should be. Clinicians are increasingly using ChatGPT, Gemini, Claude, OpenEvidence, Doximity Ask, and similar systems during real patient care. Questions about diagnosis, treatment, imaging, ECGs, and differential generation are becoming routine.

Yet most of these tools currently operate outside the traditional FDA medical device framework. Why? The answer has less to do with whether the software uses AI and more to do with what function it performs.

Clinical Decision Support v. Medical Device

Historically, the FDA has distinguished between software that supports clinician judgment and software that independently performs diagnostic or treatment functions. That distinction matters.

A tool that retrieves evidence, summarizes guidelines, or helps brainstorm differential diagnoses may qualify as clinical decision support rather than a regulated medical device. The clinician remains responsible for interpreting the information and making the final decision.

That’s very different from software that says:

“This patient has pneumonia.”

or

“This ECG demonstrates atrial fibrillation.”

Once software begins diagnosing patient-specific data and generating diagnostic conclusions, it starts looking much more like Software as a Medical Device (SaMD).

The Image Interpretation Problem

This is where things become more complicated for modern multimodal AI systems.

Most current AI assistants are careful in their public positioning. OpenEvidence emphasizes evidence retrieval and clinical knowledge synthesis. Doximity Ask focuses on workflow support, clinical questions, and documentation. ChatGPT, Gemini, and Claude are broadly positioned as general-purpose AI systems rather than diagnostic engines.

At the same time, clinicians are increasingly uploading:

• Chest X-rays

• CT scans

• ECGs

• Pathology images

and asking these systems for interpretation. That creates a fascinating gray zone.

The official positioning is “informational support.” The real-world usage looks like diagnostic interpretation.

My Own Testing

In my own testing, both OpenEvidence and Doximity Ask allowed me to upload ECG images and generate clinical interpretations.

Open Evidence

OpenEvidence included the disclaimer:

“This is not a formal interpretation and should be correlated with clinical context, prior ECGs, and laboratory data.”

Doximity

Doximity Ask displayed:

“Image use is experimental and may make mistakes. Please use accordingly.”

These disclaimers matter. They reinforce that the systems are intended to support clinician judgment rather than autonomously diagnose disease. At the same time, the systems are still functionally analyzing patient-specific data and generating clinical interpretations. That’s precisely where the FDA framework becomes more complicated.

The FDA Language

The FDA specifically discusses software that:

“assess[es] or interpret[s] the clinical implications or clinical relevance of a signal, pattern, or medical image.”

The agency includes ECGs, EEGs, CT scans, x-rays, pathology images, ultrasound, MRI, and dermatologic images within this broader category of patient-specific medical data interpretation.

An AI system summarizing atrial fibrillation guidelines occupies one regulatory category.

An AI system interpreting an uploaded ECG occupies another.

The issue is not whether companies are violating FDA rules. The issue is that multimodal AI is beginning to challenge the traditional distinction between informational support tools and diagnostic software.

Capability Versus Intent

The FDA traditionally focuses heavily on intended use. That intent can be inferred from marketing language, workflow integration, product demonstrations, clinician targeting, sales materials, and overall product design.

Disclaimers matter, but they are not the whole story. A company cannot realistically build a radiology interpretation engine, integrate it deeply into clinical workflow, and avoid regulatory scrutiny simply by adding: “For informational purposes only. Function still matters.

At the same time, general-purpose AI systems still occupy a different category from purpose-built diagnostic software. A chatbot that occasionally comments on an uploaded ECG is different from an FDA-cleared ECG interpretation platform marketed specifically for arrhythmia detection.

FDA-cleared AI Already Exists

Importantly, the FDA has already cleared multiple AI-powered SaMD platforms.

Viz.ai helps identify stroke findings and rapidly alert specialists. Aidoc offers radiology triage and acute finding detection tools. HeartFlow analyzes coronary CT imaging. Several ECG interpretation systems now use AI-assisted analysis, and newer sepsis prediction platforms combine clinical data with machine learning risk assessment.

These systems analyze patient-specific clinical data and produce a condition-specific finding, alert, risk estimate, or interpretation. That is fundamentally different from a general-purpose chatbot discussing medical knowledge.

Ambient Scribing, Coding, and Billing

Another interesting wrinkle is that many of these same platforms are no longer functioning solely as information retrieval tools. Both OpenEvidence and Doximity now offer ambient scribing workflows integrated into clinical documentation.

OpenEvidence also markets Coding Intelligence features that automatically suggest:

• ICD-10 diagnoses

• CPT codes

• E/M coding levels

from encounter documentation.

Interestingly, FDA guidance generally treats billing and administrative support differently from diagnostic software. Claims processing, financial records, coding support, and administrative workflows are often excluded from device oversight.

But these distinctions may become harder to separate as platforms increasingly combine:

• clinical reasoning

• image interpretation

• documentation

• coding

• billing

• workflow automation

inside a single integrated AI environment.

The more these systems become embedded into real clinical and financial decision-making, the harder it may become to maintain clear regulatory boundaries between administrative support, clinical decision support, and diagnostic software.

What does FDA clearance or approval actually involve?

This is another area that is often misunderstood.

FDA oversight is not simply a bureaucratic label. Companies pursuing SaMD clearance must demonstrate that the software performs reliably and safely for its intended clinical use.

That usually means clearly defining the intended use, specifying the patient population and clinical setting, documenting how the algorithm functions, validating performance on clinical datasets, and establishing systems for software quality control and version management.

For image and ECG AI, companies perform retrospective and prospective validation studies using labeled clinical datasets interpreted by expert physicians.

The FDA may review:

• sensitivity and specificity

• false positive and false negative rates

• subgroup performance

• external validation cohorts

• reproducibility across settings

In many cases, companies spend years and millions of dollars generating evidence before obtaining clearance.

Why is this difficult for large language models?

Traditional SaMD products usually perform narrow clinical tasks with relatively stable outputs and measurable endpoints. A stroke detection model either flags hemorrhage or it does not. An ECG algorithm either identifies atrial fibrillation or it does not.

Large language models are much harder to regulate within that framework. Their outputs are probabilistic, open-ended, context-dependent, and continuously evolving. Even defining intended use becomes challenging.

Is the model summarizing literature? Generating differential diagnoses? Interpreting images? Educating clinicians? Drafting notes? Triaging patients?

Sometimes it is doing all of those things simultaneously. That creates major regulatory friction.

The Next FDA Challenge

The next FDA challenge in AI may not come from traditional medical device companies. It may come from general-purpose multimodal AI systems that increasingly behave like diagnostic tools while carefully avoiding explicit diagnostic claims.

For now, many of these platforms remain positioned as clinician-support systems rather than autonomous diagnostic engines. But multimodal AI continues to improve rapidly. And once AI systems can reliably interpret images, ECGs, and pathology slides at scale, the line between “medical reference” and “medical device” may become increasingly difficult to defend.

A few questions are worth watching:

• Will regulators focus more on real-world use than stated intent?

• Does multimodal capability itself eventually trigger a new oversight framework?

• How much clinician oversight is enough?

• At what point does “clinical support” become “clinical interpretation”?

• Can a general-purpose AI remain outside device regulation once clinicians routinely use it diagnostically?

We are still early in that conversation.

Sam

Subscribe now

Ambient AI documentation tools are entering their first real legal test, and the early lawsuits are turning out to be far more interesting than I initially expected.

A recently filed California lawsuit against Sutter Health, Memorial Health Services, and MemorialCare Medical Foundation focuses on the use of Abridge ambient AI technology during patient encounters. The case is fundamentally about consent and recording law, not AI accuracy.

What The Lawsuit Actually Alleges

According to the complaint, patients allege their medical conversations were:

• recorded

• transcribed

• processed by AI systems

• transmitted to third-party vendors

without meaningful informed consent.

The complaint repeatedly emphasizes that patients were (allegedly) not clearly informed that:

• An audio recording was occurring

• Conversations would be transmitted externally

• AI-generated transcripts and summaries would be created from the encounter

The lawsuit also makes another important distinction that may become central to future healthcare AI litigation:

That may ultimately become one of the defining legal questions for ambient AI in healthcare.

The Most Interesting Detail In The Complaint

One allegation stood out to me more than anything else.

The plaintiffs claim they later discovered chart documentation indicating consent had supposedly been obtained, despite alleging that no actual consent discussion occurred during the visit.

If proven, that becomes a very different type of case. At that point, the dispute changed from whether patients fully understood the technology to whether the consent process itself was properly performed and documented.

Some reporting around the Sharp HealthCare litigation also alleges that automatically inserted consent language may have appeared in notes or charts despite patients claiming consent was never discussed.

That detail, if accurate, raises an obvious operational lesson for health systems deploying ambient AI:

The complaint also emphasizes the highly sensitive nature of the conversations involved, including discussions around:

• Diagnoses

• Medications

• Mental health

• Protected health information

That context likely strengthens the plaintiffs’ argument that patients reasonably expected a higher standard of privacy protection inside the exam room.

Abridge Itself Appears To Recognize Consent Matters

One especially notable detail is that Abridge’s own support documentation includes suggested consent language for clinicians. They suggest:

“I will be using a tool that records our conversation to help me write my clinical note, so I can pay more attention to our conversation and less time on the computer. Is that okay with you?”

Official Abridge documentation:

Abridge also instructs clinicians to follow their organization’s consent guidelines before recording begins. That matters because it suggests the industry itself already recognizes that explicit disclosure and patient awareness are important operational safeguards.

Interestingly, several academic medical centers using Abridge publicly describe verbal consent workflows as standard practice.

• Yale New Haven Health explicitly states that recording begins only after verbal patient consent is obtained.

• UCSF guidance for clinicians using Abridge goes even further, recommending clinicians obtain consent from every individual potentially captured in the recording, including family members, interpreters, and staff.

Why California Changes Everything

California is one of the country’s all-party consent states. In practical terms, that means everyone involved in a conversation must consent before recording occurs. That creates a much tougher legal environment for ambient AI systems compared with many other states.

The complaint leans heavily on:

• California wiretap law

• The California Invasion of Privacy Act (CIPA)

• The California Confidentiality of Medical Information Act (CMIA)

And this becomes especially important for healthcare leaders deploying ambient AI nationally because a workflow that feels operationally routine in one state may create meaningful legal exposure in another.

The Consent Map Matters

The legal risk changes dramatically by state.

The exact list of all-party consent states varies depending on whether the source is discussing phone calls, in-person conversations, civil liability, or criminal liability.

What Health Systems Can Do Today

None of this means health systems should stop deploying ambient AI. However, it does mean governance needs to mature much faster. A few practical steps are important:

1. Use explicit verbal consent workflows

Clinicians using ambient AI should have a standardized verbal consent process before recording begins.

Patients should understand:

• Recording is occurring

• AI transcription will occur

• A third-party vendor may process the conversation

• The purpose is clinical documentation

2. Turn off automatic consent attestations

This may be one of the biggest operational lessons emerging from this lawsuit. Automatically inserted consent language in notes or charts creates an obvious risk if the conversation never actually occurred. Consent documentation should require active clinician confirmation.

3. Build state-specific workflows

An ambient AI workflow that feels routine in Texas may create very different legal exposure in California. Health systems deploying nationally should not assume recording consent rules are uniform across states, especially for telehealth and cross-state care.

4. Review vendor defaults carefully

Ambient AI platforms often include configurable settings related to:

• Recording

• Retention

• Consent documentation

• Transcript storage

• Secondary data use

Those defaults matter more than many organizations probably realize.

5. Prepare for patient questions now

Patients are increasingly going to ask:

• “Was I recorded?”

• “Who heard this?”

• “Was this used to train AI?”

• “Can I decline?”

The organizations that handle this best will be the ones that treat transparency as part of the clinical workflow rather than a legal disclaimer buried inside intake paperwork.

This Is Different From The Other AI Recording Lawsuits

Several AI recording lawsuits are now getting grouped online, but they are actually about very different issues.

The Otter.ai Case

The Otter.ai lawsuit focuses on AI meeting assistants joining Zoom, Google Meet, and Microsoft Teams calls, then transcribing and analyzing conversations without sufficient participant consent.

That case centers on AI meeting bots rather than clinical encounters.

The Illinois RingCentral / Heartland Dental Case

The Illinois lawsuit involved AI-powered call analytics and transcription for business communications.

Earlier this year, a federal judge dismissed the federal wiretap claims under the “ordinary course of business” exception within the Electronic Communications Privacy Act. That ruling may ultimately help some AI communications vendors.

At the same time, the decision did not address California’s stricter all-party consent framework, which is one reason the California healthcare litigation may prove far more consequential.

The Bigger Issue Emerging

What makes these cases so fascinating is that they are forcing courts to answer a question that existing law never anticipated:

For now, plaintiffs are arguing that it is not different at all. And if courts ultimately agree, the first major legal battles around ambient AI may have nothing to do with hallucinations and more to do with consent, recording law, and patient expectations of privacy.

Sources

California healthcare ambient AI complaint

Abridge recording guidance

50-state recording law survey

Reporters Committee recording guide

Sam

AI’s Invisible Error

AI clinical decision support is entering medicine incredibly fast. Some of it is genuinely useful. It can summarize charts, organize differentials, surface literature, reduce documentation burden, and help clinicians process information faster. Most physicians I know have at least experimented with it, even if they’re not talking about it publicly yet. But there’s something unsettling about this moment in medicine.

Medicine has always tolerated uncertainty. What medicine has never been comfortable with is invisible uncertainty. That’s a very different problem.

The issue isn’t that AI makes mistakes. Every system in medicine makes mistakes. The issue is that clinicians often can’t recognize the mistakes AI makes.

A hallucinated citation can look authentic. A fabricated recommendation can sound clinically reasonable. An incorrect differential may still appear thoughtful, organized, and comprehensive. The output arrives wrapped in confidence: fluent language, structured reasoning, professional tone, well-formatted references.

Medicine trains us to associate those signals with competence. LLMs reproduce those signals extremely well. Sometimes uncomfortably well.

The Verification Paradox

AI clinical decision support is mostly marketed as a time-saving tool. That creates a paradox.

Neither option feels particularly great.

Imagine using a lab assay with no established sensitivity or specificity. Imagine reading radiology reports from an unknown source with no validation data. Imagine prescribing a medication whose adverse effects could only be discovered through extensive independent investigation after each use.

Medicine would never tolerate that. Yet many clinicians are now using AI systems whose real-world clinical error rates remain largely undefined.

Invisible Error Is Different

There’s a major difference between errors that announce themselves and errors that blend into plausibility.

A potassium of 12 in an asymptomatic patient triggers immediate skepticism. A CT report describing a patient with one kidney when you know they have two is easy to catch. Generative AI errors are different because they’re often plausible.

The answer frequently contains:

• appropriate terminology

• convincing structure

• partial truths

• accurate references mixed with inaccurate ones

• recommendations that sound completely reasonable in context

The system often performs well enough to earn trust before it fails. That creates a cognitive hazard medicine hasn’t really dealt with before.

Medicine Expects Known Limitations

Medicine doesn’t expect perfection from its tools.

Medicine expects known limitations.

• Lab tests publish sensitivity and specificity.

• Imaging studies have established false positive and false negative rates.

• Clinical prediction rules undergo validation studies.

• Medications have adverse event reporting systems.

• Medical devices require performance standards.

• Human trainees are supervised, reviewed, and audited.

These systems fail. We know they fail. We also understand roughly how, when, and why they fail.

Generative AI clinical decision support occupies a very different position. Current systems still lack the safeguards medicine usually expects.

• No clinically meaningful denominator for error.

• No stable performance characteristics across contexts.

• No routine post-market surveillance.

• No reliable way for frontline clinicians to recognize failure.

• Outputs that appear complete, organized, and authoritative.

That combination is unusual in medicine.

The Human Factors Problem

There’s another issue medicine hasn’t fully grappled with yet.

Humans are extremely susceptible to automation bias. When a system sounds confident, organized, and data-rich, people naturally defer to it, especially when they’re tired, overloaded, distracted, or under time pressure. Modern clinical medicine contains all four. Even experienced physicians are vulnerable to this.

Aviation learned this years ago. Pilots occasionally deferred to automated systems even when contradictory evidence was sitting right in front of them.

Medicine is now entering similar territory, except these systems are conversational. That matters. A conversational interface feels collaborative. Intelligent. Thoughtful. The psychological effect is powerful.

This Doesn’t Mean AI Is Useless

None of this means generative AI should be excluded from medicine. The technology is already useful in a lot of settings, and it’s almost certainly going to become more deeply integrated into clinical workflows.

But we should ask whether we’re applying the same evidentiary and safety standards to AI-assisted reasoning that we apply to everything else in clinical care.

Right now, the answer feels inconsistent. Medicine has historically treated invisible error as dangerous. Generative AI introduces a form of invisible error that is persuasive, scalable, and difficult to recognize in real time.

That deserves a lot more attention than it’s currently getting.

Sam

In the last post, I said I’d break down RAG and why it changes how you should think about tools like OpenEvidence, DoximityGPT (now Doximity Ask), and ChatGPT for Clinicians. This is the part that actually matters because without it, all of these tools start to look the same.

First, how RAG works

RAG stands for Retrieval-Augmented Generation. The model doesn’t just answer your question from its own knowledge; it searches for information first and then writes an answer using what it finds. That search step runs on a specific database, and that detail drives the quality of the output.

Every RAG system sits on top of a defined body of literature. Many systems rely on PubMed abstracts or summaries. A smaller number incorporate society guidelines. An even smaller group has access to full-text journal articles. Some systems mix multiple sources, others stay narrow.

Those differences aren’t small. An abstract gives you a compressed version of a study. A guideline gives you consensus and synthesis. A full-text article gives you methods, limitations, and nuance. The model can only work with what it receives.

So the real question becomes: what database did this system search, and how deep is that access?

Why this matters clinically

If you’ve used a large language model long enough, you’ve seen confident answers with incorrect details or outdated recommendations. That behavior reflects the limits of training alone.

RAG changes the behavior by allowing the system to pull recent papers, surface guidelines, and anchor answers to identifiable sources. That shift makes these tools feel more grounded.

The failure point doesn’t disappear. It moves into the retrieval layer.

Now the weak link sits in the database itself, and in how the system searches it. A system that only sees abstracts will produce a different answer than one that can read the full text. A system that lacks access to guidelines will miss consensus recommendations entirely.

A confident answer built on thin retrieval will still sound convincing.

Why these tools don’t behave the same way

A lot of clinicians assume these tools are interchangeable. They aren’t. They’re different retrieval systems built around similar models, and they point at very different databases.

OpenEvidence takes a very explicit approach. It has direct partnerships with journals and specialty societies, and it includes access to full-text content in addition to abstracts. That matters. It is not just searching more, it is searching deeper. It also appears to lean toward guidelines and systematic reviews, placing those results up front. The exact ranking process is not fully transparent, but the behavior is noticeable when you use it.

DoximityGPT (AKA Doximity Ask) is doing something different. Doximity acquired a company called Pathway, which built a large structured medical dataset for AI. That dataset includes guidelines, drugs, and landmark trials across specialties.

That sounds similar to literature retrieval, but it is not the same thing. This is not raw access to journals. It is a processed layer on top of the literature. The data has already been selected, structured, and tied back to sources before the model ever sees it.

That can make answers faster and cleaner. It also means you are relying on how that dataset was built. You are one step removed from the original paper. The retrieval and ranking processes are also not clearly visible.

ChatGPT for Healthcare / Clinicians is different from both of these. It does not come with a built-in clinical database. It may search PubMed when it perceives the need. It is not pulling guidelines unless you explicitly connect those sources.

Out of the box, it is just a model generating answers from training.

It only becomes a RAG system if you give it something to retrieve from. That could be web browsing, your institution’s documents, or a defined knowledge base. The database is whatever you connect it to. Same model class, completely different behavior.

It is also worth realizing that OpenEvidence and DoximityGPT are just the visible layer. A lot of the most advanced RAG systems are not standalone products. They are being built inside health systems. These systems pull from EHRs, institutional guidelines, and internal knowledge bases, then combine that with the literature. That is a very different model from a tool that only searches PubMed or a structured dataset. This is where things are heading: systems that pull from multiple sources at once, not just one database.

The mental shift

You’re not using a single model. You’re using a pipeline that interprets your question, searches a specific database, ranks results, and then generates an answer. Each step carries its own potential failures.

If the system searches a shallow or limited dataset, the answer reflects that limitation. If the ranking favors less relevant material, the synthesis reflects that bias. That explains why you’ll sometimes see correct answers with weak citations, incorrect answers with convincing citations, and disagreement across tools that sound equally confident.

How to actually use these tools

A few practical habits help:

• Look at the source, not just the answer

• Ask what database the system searched

• Check whether the citations reflect abstracts, guidelines, or full text

• Use more than one system for higher-stakes questions

• Recognize that missing evidence may reflect a retrieval gap

You’re evaluating both a search process and a generated summary at the same time.

Where this is going

RAG systems are improving quickly, with better indexing, stronger ranking, and deeper integration with clinical knowledge sources. The direction points toward systems that combine reasoning with targeted access to high-quality information.

That shift changes the skill set. Knowing how to question the system in front of you becomes as important as knowing the underlying facts.

ChatGPT for Clinicians is being positioned as a tool for documentation, clinical questions, and literature review. It is essentially the individual clinician-facing version of OpenAI’s broader healthcare offerings, designed for use outside of enterprise health system deployments. The more important question is what kind of system it actually is. Is this a search tool? A reasoning assistant? Something in between? Before thinking about whether it works, it helps to understand what it is.

A language model, first

At its core, ChatGPT for Clinicians is still ChatGPT. It is built on a large language model that interprets clinical questions, synthesizes information, and generates responses. In practical terms, it retains most of the core capabilities of standard ChatGPT, but is scoped more narrowly for clinical use, with some features such as image generation intentionally excluded. It can summarize a chart, draft an assessment and plan, or walk through a differential.

It also carries the same limitations. It can sound confident and still be wrong. It can compress complex information and miss nuance. It does not have clinical judgment. OpenAI has also placed explicit boundaries on its use. It should not be used to interpret medical images, ECGs, EEGs, or other signal-based data. It is not intended to diagnose or generate treatment plans. These are not minor exclusions. They define the edge of the system’s role. The model works with text. It does not replace clinical interpretation in areas where pattern recognition and context are critical. This reinforces that the system functions as a language-based assistant, not a clinical decision tool.

So far, none of this requires search or external data.

The promise of “clinical search”

Where things get less clear is how the product handles evidence. Descriptions often mention cited answers, access to medical literature, and “clinical search.” That suggests the system may be pulling in external information at the time of the query. Many people take that to mean it is a RAG system. That assumption may be wrong.

Why citations don’t answer the question

A model can produce citations without retrieving anything in real time. It can generate references based on patterns from training data, reconstruct plausible sources, and cite real papers that are outdated or only partially relevant. Citations alone do not tell you how the system is working. The real question is simple. Is the model retrieving information when you ask a question, or relying on what it has already learned?

What is RAG, in plain terms?

Retrieval-augmented generation, or RAG, is a specific architecture. The system searches external sources such as journals or guidelines, selects relevant information, and then generates an answer using that material. The response is grounded in what the system retrieves at that moment, not just what it learned during training. In practice, the answer depends on what the system can find at the time of the query. That matters in medicine because knowledge changes quickly, and context matters.

What a true RAG tool looks like

Some clinical tools are built this way. OpenEvidence retrieves and synthesizes medical literature before generating responses. DoximityGPT is generally described in the same category, pulling from external clinical sources and summarizing them for the user. These behave more like search systems with a language layer on top.

What we actually know about ChatGPT for Clinicians

Public information is limited. It supports documentation, summarization, and clinical Q&A. It can provide cited responses. It includes some form of “clinical search.” What is not clear is when retrieval is used, how it is triggered, how results are selected, or whether citations come from retrieval or generation.

There is also a more practical unknown. We do not know what sources are being used. The difference between full-text guidelines, curated clinical databases, and something like PubMed abstracts is significant. Abstracts summarize studies. They are not designed to guide clinical decisions on their own. Key details about patient populations, limitations, and context are often missing. A system that relies on shallow sources may still produce fluent answers, but the underlying evidence may be thin. For clinicians, where the answer comes from may matter more than how confidently it is written.

Why this distinction matters

This is not a technical detail. It affects trust. If the system relies on internal knowledge, the risk is hallucinated or outdated information. If it relies on retrieval, the risk shifts to incomplete search or misinterpretation of real sources. Those are different failure modes. They require different habits from the user. The output may look the same in both cases.

A more accurate way to think about it

ChatGPT for Clinicians is best understood as a clinical assistant built on a language model, with some form of access to external information layered on top. Retrieval may be part of the system. It is not the system.

Where this leads next

The key question is not whether these tools can generate answers. It is whether they can find the right information, at the right time, and represent it accurately. That is where retrieval becomes central. In the next post, I will break down RAG in more detail and show how it changes how clinicians should think about tools like OpenEvidence, DoximityGPT, and ChatGPT.

Hopefully, this first post in the Ashoo Review helps clarify the issue.

~Sam

There’s a quiet pattern emerging with many of the new AI ambient scribe tools. It usually starts the same way: a clean interface, fast onboarding, assurance of HIPAA compliance, and a reassuring button that says something like “Sign BAA (Business Associate Agreement).”

That moment feels like compliance has been handled. It hasn’t.

Let’s walk through why this matters, especially for employed physicians.

1. Who actually holds the responsibility?

Under HIPAA, the covered entity is the organization responsible for protecting patient data. In most cases, that’s your hospital or health system.

As an employed physician, you are operating under that entity. You are not acting independently when you see patients, document care, or use clinical tools within that environment.

So when you personally sign a BAA with a vendor, you’re not stepping into the hospital’s legal role. You’re acting outside of it.

2. Why an individual BAA doesn’t solve the problem

A BAA is not just a statement of good intentions. It defines:

• Who is allowed to share protected health information

• Under what circumstances that sharing happens

• What safeguards are required across systems

• Who is accountable if something goes wrong

When a hospital signs a BAA, it is aligning the vendor with its entire compliance framework, including IT policies, audit controls, and data governance.

An individual clinician signing a BAA does none of that.

It does not:

• Integrate with hospital security controls

• Align with institutional policies

• Cover other users or workflows

• Protect the hospital from liability

From the hospital’s perspective, it’s essentially invisible.

3. The real risk you’re taking

If you use one of these tools inside the hospital environment without an institutional BAA, a few things are happening:

• You may be transmitting protected health information to a vendor that your hospital has not approved

• That vendor is not formally accountable to your organization

• Your hospital cannot verify how data is handled, stored, or reused

• You may be violating internal policy, even if the tool itself claims compliance

This is where the risk shifts from abstract to personal.

It can affect:

• Internal disciplinary action

• Credentialing concerns

• Legal exposure if a breach occurs

• Your reputation within the organization

None of that is mitigated by clicking a button on a website.

4. Why “free + BAA included” should raise questions

If a company is offering:

• A free product

• Instant BAA execution

• No enterprise review

• No contract negotiation

Then you should pause.

Hospitals typically go through months of review before approving a vendor that touches patient data. That process includes legal, compliance, security, and IT.

A one-click BAA bypasses all of that.

That doesn’t mean the company is acting in bad faith. It means the agreement you’re signing is not designed for the environment you’re working in.

5. The private practice exception

This changes if you are in private practice.

In that setting, you or your practice are the covered entity. You can sign a BAA directly with a vendor and use the tool within your own workflows.

The key difference is control. You own the systems, the policies, and the data governance.

Once you step into a hospital, that control shifts.

6. A simple way to explain it to colleagues

You can frame it like this:

• The hospital is responsible for the entire system

• A BAA connects a vendor to that system

• An individual agreement sits outside that system

So even if the tool feels safe, it is not approved for use inside the hospital until the hospital itself signs.

7. What to do instead

Before using any AI scribe that touches patient data:

• Ask if your hospital has an enterprise agreement in place

• Route the vendor through your compliance or IT team

• Avoid entering identifiable patient information until that’s confirmed, and that includes audio recordings of your patient, since voice data itself is identifiable and cannot be treated as deidentified

It may feel slow. It protects you and your patients.

8. When you step outside your role as an agent

There’s another layer that often gets overlooked. As an employed physician, you are acting as an agent of the hospital when you deliver care within its systems and workflows.

When you independently sign a BAA and use a tool that your hospital has not approved, you may be stepping outside that agency relationship.

That has consequences.

From a HIPAA standpoint, responsibility can shift in uncomfortable ways:

• You may be viewed as acting outside the scope of your employment

• Institutional protections may not fully apply to your actions

• Liability can become more personal rather than organizational

From a practical standpoint, this can lead to:

• Internal investigations or disciplinary action

• Mandatory reporting of a privacy incident

• Exposure to civil penalties tied to improper disclosure of PHI

• Scrutiny from licensing boards, depending on the severity

Hospitals carry insurance, legal infrastructure, and compliance programs designed to manage risk at scale. Those protections are built around approved workflows and authorized vendors.

When you go outside that structure, you are operating without that safety net.

There’s a strong pull toward tools that save time and reduce documentation burden. That need is real. The way we adopt these tools still matters.

A signed checkbox is not the same thing as institutional approval.