7 Things NOT To Do On OpenEvidence
Or Any AI Clinical Decision Support Tool
If you have been a regular reader of this newsletter, you know I’ve focused a lot on the failures of clinical AI tools. But I still find them clinically useful. To help my colleagues and friends avoid the traps of these tools, I developed a short checklist of things to keep in mind. Download it and read more about the reasoning behind each item below.
As always, if you enjoy reading this newsletter, subscribe and tell a friend.
Sam
Clinical AI tools have become part of daily practice for many physicians. OpenEvidence, Doximity GPT, AI scribes, and clinical decision support tools can improve efficiency and provide valuable assistance. At the same time, these tools create new risks that most physicians never encountered during training.
I have written about the limitations, regulatory concerns, and real-world testing of clinical AI systems. The most common problems I see are not dramatic AI hallucinations. They are workflow mistakes made by clinicians who assume these tools are safer, more accurate, or more legally protected than they actually are.
To help physician leaders educate their clinical staff, I created the following Clinical AI Safety Checklist. You can download it here and read more about each item below.
1. Don’t Upload ECGs or X-Rays
Most general-purpose clinical AI tools are not FDA-cleared devices for interpreting ECGs, radiographs, CT scans, MRIs, or other diagnostic images.
My own testing of multiple systems has demonstrated substantial errors in ECG and radiology interpretation. These errors can be subtle and dangerous because the AI often presents its conclusions with confidence. Often, there is no warning that the system can not accurately read them, and the presence of an image upload feature is misleading.
If a clinical AI platform lacks FDA authorization for diagnostic image interpretation, physicians should avoid using it for that purpose.
Dive Deeper:
2. Don’t Sign a BAA if You Work in a Hospital System
Many physicians assume that signing a HIPAA Business Associate Agreement solves privacy concerns. After all, it’s a convenient button click in the AI tool, and it reassures the user with a “HIPAA Compliant” banner.
In reality, employed physicians do not own the data and are not the entity responsible for it. In the law’s eyes, the protected health data is stored by the hospital, which has the responsibility of safeguarding it. For that reason, the BAA has to occur between the AI service and the hospital. So clicking that little HIPAA BAA agreement box only puts you at risk for acting as an “agent” of the hospital without authority. Unless you are in private practice, own the practice, and use the AI tool only on those patients, better to avoid this trap.
Dive Deeper:
3. Don’t Upload PHI
Even when a platform offers a BAA, physicians should think carefully before uploading identifiable patient information.
The question is not simply whether the technology can accept PHI. The question is whether you are authorized to disclose that information under your organization’s policies and contractual arrangements.
When in doubt, de-identify the information or avoid uploading it altogether. If you don’t know what’s involved in de-identifying the information, read more at the link below. It’s a critical skill that will keep you (and your hospital) out of a lawsuit.
4. Don't Trust AI-Generated Insights Without Verification
Many clinical AI systems now generate summaries, assessments, risk predictions, or observations across multiple visits. These insights can be useful. They can also be wrong.
An AI may only be looking at a subset of encounters, incomplete documentation, or fragmented records. As a result, it may generate conclusions that appear reasonable while missing critical context.
Physicians should treat AI-generated insights the same way they would treat recommendations from a trainee: useful starting points that require independent verification.
Dive Deeper:
5. Don't Use AI-Informed Clinical Reasoning Without Documenting It
This may be the most overlooked issue in clinical AI today.
If an AI-generated insight influences your diagnosis, treatment plan, referral decision, or other aspect of care, that reasoning should be documented in the patient’s official medical record.
Patients can review information contained in the medical record and request corrections when appropriate.
Information that exists only inside an AI platform may influence care without the transparency and accountability legally required by HIPAA and the Cures Act.
If the AI helped drive a clinical decision, document the relevant information in the chart.
Deeper Dive:
6. Don't Sign AI-Generated Notes Without Reviewing Medications and Mental Health History
When physicians think about AI scribe errors, they often focus on hallucinations. But a report earlier this year from the Ontario Auditor General’s office found that AI scribes committed these errors most often:
45% hallucinated treatment plans, blood tests, or referrals that were never discussed
60% documented incorrect medication names or dosages
85% omitted critical aspects of mental health history
AI scribes frequently produce notes that appear polished and complete while leaving out clinically important details. Before signing any AI-generated note, carefully review medications, mental health history, and other high-risk sections of the chart for omissions.
Your signature confirms the accuracy of the documentation.
Deeper Dive:
7. Don't Delete AI Chat History Without Understanding the Consequences
This issue cuts both ways.
Maintaining AI chat history may provide evidence regarding what information was presented to the physician and what recommendations were generated by the system. Deleting that history may remove information that could later be relevant when evaluating clinical decisions. AI chat histories may also become discoverable during litigation or investigations. Physicians should understand their organization’s policies and think carefully before deciding whether to retain or delete AI interactions.
Most importantly, any information that materially influences patient care should be documented in the medical record rather than existing solely within an AI conversation.
Deeper Dive:
Final Thoughts
Clinical AI tools are becoming a permanent part of healthcare delivery.
The greatest risks are rarely the ones featured in headlines. Most arise from privacy misunderstandings, documentation shortcuts, incomplete records, misplaced trust, and workflow decisions made by clinicians under pressure.
Technology will continue to improve.
Professional responsibility remains unchanged.
Your license. Your responsibility.
The pattern I would add is to treat the tool's answer as a workflow input, not a decision endpoint. In OR-adjacent work, the useful habit is asking what source the system used, what uncertainty remains, and which human owns the next action before the output changes care or documentation.