In April 2024, the Federation of State Medical Boards (FSMB) published its policy on artificial intelligence in clinical practice. In its recommendations, was this statement about professional accountability:
“The physician is ultimately responsible for the use of AI and should be held accountable for any harms that occur.”
That expectation made sense in 2024. Physicians have always been responsible for the clinical decisions they make, including decisions informed by diagnostic tests, clinical guidelines, and decision-support software. So the use of AI would not change that expectation.
But the FSMB’s policy also acknowledged something important. Physicians may be unable to explain precisely how a complex AI system arrived at its recommendation. Despite that limitation, they remain responsible for interpreting its output and justifying their clinical decisions.
Two years later, we have a growing body of research suggesting that physicians frequently fail to recognize errors in AI-generated medical information. This raises the question: Do physicians still possess the ability to verify the information AI provides?
Physicians encounter AI-generated documentation, summaries of medical literature, and patients who arrive with diagnoses suggested by consumer chatbots. In each setting, the physician is expected to determine whether the AI’s conclusions are correct. Today, I will spend some time trying to determine if physicians can reliably do that anymore.
What Does It Mean to Verify an AI Answer?
If a colleague makes a statement about a treatment, we can find supporting literature, examine studies, and determine whether the statement is true. If a clinical guideline recommends a particular intervention, we can review the supporting evidence and assess whether it applies to our patient. AI-generated medical information introduces distinct complications.
An AI model can produce a recommendation based on several studies, clinical guidelines, and pieces of patient information. The response may contain citations, explain reasoning, and present a conclusion that sounds entirely consistent with accepted medical practice. But verifying that answer involves three steps.
Source verification: Do the cited references exist, and contain the information claimed?
Interpretation verification: Has the model correctly interpreted those sources, including study limitations, exclusion criteria, conflicting findings, and the strength of the evidence?
Summary verification: Does the model’s conclusion correctly follow from the combined evidence, and is that conclusion appropriate for the specific patient being evaluated?
The first step is straightforward. A physician can follow a citation, locate the relevant passage, and determine whether the AI accurately represented it. In fact, the AI could even quote the relevant portions of the study manuscript.
The second step is cumbersome and time-consuming. The physician must understand the underlying evidence well enough to recognize when the AI has overstated a finding, overlooked an exclusion criterion, or applied research to an inappropriate population.
The third step … well, who has the time. An AI model can accurately summarize several individual studies and still reach an incorrect conclusion when combining their findings. Every citation might be legitimate. Every individual statement might be defensible. But the final recommendation could still be wrong.
Additionally, the explanation an AI model provides for its recommendation is itself generated by the model. It shouldn’t automatically be treated as an accurate record of how the conclusion was produced. Which means a physician reviewing the answer may be evaluating a plausible explanation rather than an auditable account of the model’s reasoning.
Physicians Are Already Missing AI Errors
A September 2026 study published in npj Digital Medicine evaluated whether junior clinicians could identify hallucinations in GPT-4o-generated medical information.
Their findings showed clinicians detected only 15.8% of the hallucinations. In addition, 13% failed to identify a single hallucination across the scenarios they evaluated. And most importantly for my colleagues in the ED and ICU, detection did not improve as the clinical stakes increased.
The study examined simulated encounters and involved junior clinicians, so its findings can’t be generalized to experienced attending physicians. Nevertheless, it directly challenges the assumption that a clinician reviewing AI output will reliably identify errors before they affect patient care.
A separate randomized trial published in NEJM AI in April 2026 examined automation bias among physicians who completed 20 hours of AI literacy training.
The investigators found “Physicians demonstrate substantial automation bias when exposed to erroneous LLM recommendations”. They also found that AI literacy training and access to standard medical resources made no difference in automation bias. The test group exposed to error-prone model responses had an accuracy of 76% compared to a control group with an accuracy of 90%.
Neither study establishes that physicians are incapable of verifying AI-generated clinical reasoning. But they do demonstrate that physician review and AI education alone can’t be assumed to provide reliable protection against errors.
Unfortunately, much of the current approach to clinical AI safety depends on exactly those assumptions.
When Every Citation Is Correct, and the Answer Is Still Wrong
Another 2026 study examining evidence support in AI-generated medical assessments evaluated responses from GPT-5, OpenAI o3-mini, and GPT-3.5. It found “evidence support rates across models ranged from only 60% to 65%, with conservative hallucination rates of 34%-37%”. Those numbers suggest there is considerable validation required for conclusions reached by these models.
Physicians must determine whether the model has appropriately interpreted the evidence, accounted for competing explanations, and applied the findings to the clinical circumstances. In effect, the physician may need to independently reconstruct the argument that produced the recommendation.
That creates a practical dilemma which I’ve discussed a few times in prior articles.
If a physician must read the relevant studies, independently interpret their findings, evaluate competing evidence, and formulate a clinical conclusion to determine whether the AI’s recommendation is correct, how much of the original cognitive work has actually been reduced? Why use an AI at all?
There may be circumstances in which AI can help physicians locate relevant information, organize evidence, or identify diagnoses they might otherwise overlook. But the verification requirement shines a light on just how much independent work is necessary before that answer can be trusted.
We have relatively little direct evidence measuring physicians’ ability to identify errors in AI-generated synthesis when all the individual citations are accurate.
The Patient Arrives With an AI Diagnosis
So far we’ve talked about the physician side of the equation, but consider the patient who arrives in the clinic or emergency department after spending an hour discussing symptoms with a consumer AI chatbot.
In that chatbot conversation, the patient described their symptoms, answered follow-up questions, maybe entered laboratory results, or even uploaded information from their medical record.
The chatbot provided a diagnosis, explained its reasoning, and recommended a particular course of treatment. By the time the patient sees a physician, they may have considerable confidence in that explanation.
Now, the physician has to evaluate the patient’s symptoms while also addressing the conclusions of an AI system they never interacted with. The patient remembers only the diagnosis and a few of the chatbot’s explanations, leaving the physician with no access to the original conversation, the information supplied to the model, or the assumptions that shaped its response.
Even if the patient provides the complete conversation, reviewing it takes time and effort. The physician has to determine whether the patient entered accurate information, whether the model misunderstood something, and whether its conclusions are clinically justified.
There is also the possibility that the AI’s diagnosis has influenced how the patient describes their symptoms. A patient who has been told that their chest discomfort is probably gastrointestinal may emphasize features that support that explanation. Another patient who has been told they have a serious neurological disorder may interpret nonspecific symptoms through that lens.
Now the physician is evaluating both the patient’s clinical presentation and the effects of an AI interaction. Of course, physicians can perform their own assessment, obtain an independent history, examine the patient, and order appropriate testing. They can arrive at a sound clinical decision without auditing every statement made by the chatbot. But the original AI conclusion remains unverified.
This creates an interesting distinction between independently evaluating a patient and validating the AI-generated information that influenced the encounter.
As patient-facing AI becomes more common, physicians may increasingly be expected to address conclusions whose origins and supporting evidence are unavailable.
The research examining this specific verification problem is virtually non-existent. We need to understand how often patient-generated AI conclusions affect clinical encounters, whether physicians can identify errors within those conclusions, and how the interaction influences subsequent decision-making.
Have We Defined What Physicians Need to Know?
Physicians have skills that are relevant to evaluating AI-generated medical information. We receive extensive training in clinical reasoning and evidence-based medicine. We learn to interpret diagnostic tests, evaluate research, recognize cognitive biases, and apply population-level findings to individual patients. But auditing an AI-generated clinical recommendation may require additional competencies.
A physician may need to separate a response into individual factual claims, locate the supporting evidence, identify unsupported inferences, recognize omitted information, and determine whether the conclusion is justified. Those are related tasks, but proficiency in one does not guarantee proficiency in the others.
An October 2026 paper on teaching hallucination literacy in medical education proposes treating verification of AI-generated information as a distinct educational competency.
The authors describe a structured process that includes identifying claims, examining evidence, and making an explicit verification decision.
That’s a useful framework that also exposes how little standardization currently exists around what physicians are expected to do when told to verify AI output.
What constitutes adequate verification of an AI-generated differential diagnosis? How should a physician evaluate a recommendation based on multiple studies with different populations and outcomes? What level of independent review is necessary before an AI-generated clinical summary can safely influence management?
And perhaps most importantly, how do we determine whether a physician has demonstrated competence in performing those tasks?
The NEJM AI study we discussed earlier suggests that education alone may be insufficient. Even physicians who completed substantial AI literacy training remained vulnerable to erroneous recommendations.
We need studies that measure actual verification performance, including the ability to identify unsupported conclusions built from accurate source material. Until those competencies are defined and tested, instructions to verify AI-generated information remain difficult to operationalize.
Who Is Verifying the Verification?
Physicians have always been expected to evaluate the information they use in clinical decision-making. That responsibility remains reasonable. But AI has introduced a new problem: its conclusions may depend on a synthesis of evidence that cannot be reconstructed simply by checking the references it provides.
A physician may recognize an obviously incorrect recommendation. They may independently arrive at the correct diagnosis or treatment plan. But neither establishes that they can reliably verify the AI’s underlying analysis, particularly when every individual citation appears legitimate.
The studies discussed here suggest that physicians already struggle to identify errors in AI-generated information. What remains uncertain is whether additional training, better access to supporting evidence, or more transparent systems can overcome those limitations.
And perhaps there is a point at which an AI-generated conclusion becomes sufficiently complex that independent verification requires essentially repeating the entire analysis.
Have we reached a point where physicians are being asked to verify conclusions they can no longer reliably verify?