The Hidden Medical Record
When Does AI Memory Become A Medical Record?
In this article, I’m looking at the latest feature from some of the most popular medical AI models… persistent patient memory. It’s a feature that seems super helpful, but is it creating a new legal challenge?
As always, if you enjoy reading this newsletter, consider subscribing and telling a friend.
Sam
Over the last several years, healthcare has undergone a remarkable shift in transparency. Not long ago, patients waited days or weeks to receive laboratory results, imaging reports, pathology findings, and physician notes. Some health systems intentionally delayed access to information until a physician could review it first. Others required patients to submit formal records requests to obtain information that already existed electronically.
The 21st Century Cures Act and subsequent Information Blocking regulations accelerated a different vision of healthcare. The guiding principle became increasingly clear: if patient information exists electronically, patients should have access to it.
Healthcare spent years debating how quickly information should be released. We may soon be debating something very similar. What happens when information influencing clinical decisions is created not by a physician, laboratory, or radiologist, but by an AI system?
From Documentation to Memory
Ambient scribes listen to patient encounters, generate notes, and reduce clerical burden. Whether one uses Heidi, Glass, Abridge, Suki, or another platform, the core value proposition is largely the same: document what happens during the encounter.
The next generation of tools is beginning to do something different. Many AI platforms now maintain context across encounters, synthesize information from multiple visits, and generate longitudinal patient summaries. OpenEvidence can associate patient-specific searches and information with an individual patient. Heidi and Glass can pull together information spanning multiple encounters. Several platforms can generate summaries that span months or years of clinical history.
These capabilities are often described as contextual awareness, patient memory, longitudinal synthesis, or pre-visit intelligence. Whatever terminology is used, the underlying shift is significant. The industry is moving from AI that documents encounters to AI that remembers patients.
When Does Memory Become a Record?
At first glance, this seems like a distinction without a difference. After all, physicians have always reviewed prior notes, laboratory results, imaging studies, and discharge summaries. AI simply makes that process faster. But there is an important progression worth examining.
At the most basic level, an AI system retrieves information. “Show me the last three notes.” Few would view that as creating a new medical record.
The next level is summarization. “Summarize the last three notes.” Again, the system is organizing information that already exists.
Then comes another level entirely as the AI prompts the physician with “This patient demonstrates progressive cognitive decline and increasing medication nonadherence.” Now the AI has created a patient-specific conclusion that may influence future clinical decisions.
At that point, the system is doing more than retrieving information. It is generating and retaining patient-specific knowledge. If clinicians rely on that information, what exactly is it?
A Possible Future
Imagine opening a patient’s chart five years from now. Before you review a single note, an AI-generated summary appears:
Longitudinal Patient Summary
Progressive decline in renal function over two years
Multiple episodes of medication nonadherence
Increasing emergency department utilization
Missed specialist referrals
High likelihood of care fragmentation
The summary immediately shapes your thinking. You order additional testing. You spend more time discussing medication adherence. You prioritize care coordination. The AI-generated summary influenced your clinical decision-making within seconds.
Yet no physician wrote that summary.
No laboratory generated it.
No radiologist signed it.
No individual encounter contains it.
The information was synthesized by software and retained over time. Is that simply a software feature? Or is it more like a clinical record?
The Legal Framework Was Built for a Different World
Current law does not provide a clear answer. HIPAA provides patients with the right to access protected health information. The regulation states that individuals have a right to “inspect and obtain a copy of protected health information about the individual in a designated record set.”
The key phrase is designated record set. The phrase encompasses all records used to make medical decisions about individuals. Most clinicians intuitively understand what belongs in the medical record when information originates from a physician, laboratory, radiologist, or pharmacist. The answer becomes less obvious when information is generated by an AI system and retained across encounters.
The Cures Act and Information Blocking regulations add another dimension. The Information Blocking Rule defines information blocking as a practice that is likely to interfere with, prevent, or materially discourage access, exchange, or use of electronic health information. Yet current AI systems do not provide patient portal access or a mechanism for patients to view their own information.
Federal policy has been moving steadily toward greater transparency and fewer barriers to information access. Yet neither HIPAA nor the Cures Act was written for a world in which software could develop and retain its own understanding of a patient over time.
If a physician routinely relies on an AI-generated patient summary, should patients be able to access it? Is that summary part of the designated record set? These are the questions the law will have to answer soon.
The Governance Challenge
The regulatory questions may ultimately prove easier than the governance questions.
Consider a few practical issues.
Who owns AI-generated patient memory?
Who is responsible for correcting errors?
How long should it be retained?
What happens when an AI-generated summary conflicts with the underlying chart?
Should these systems maintain audit trails?
Should patients be informed that longitudinal AI memory exists?
Should patients have access to it?
Could it become discoverable during litigation?
Unlike a lab result, which is binary and objective, AI-synthesized 'memory' is interpretative. If an AI incorrectly tags a patient as 'medication nonadherent' based on a misinterpreted data point, that 'memory' can color every future clinical interaction. We don’t have a clear mechanism for patients to challenge or 'edit' these persistent algorithmic conclusions, raising a critical question: how do we protect patients from automated bias that the legal system has not yet classified as part of the medical record?
Health systems are increasingly developing governance frameworks for AI-generated documentation. Far fewer appear to be discussing governance frameworks for AI-generated memory, but the distinction is important.
Documentation captures what happened. Memory influences what happens next.
The Hidden Medical Record
AI may be creating a new category of information. Patient-specific knowledge generated by software, retained across encounters, and used to inform future care.
That information lives somewhere. It may influence clinical decisions. It may persist for years. In many cases, patients may not know it exists.
Healthcare is approaching a new and largely unexamined boundary. For the past decade, we debated who should have access to the medical record. The next decade may be spent defining what the medical record actually is.
Before deploying AI memory systems at scale, health systems should begin asking a few questions:
Is AI-generated patient memory part of the medical record?
Would we be comfortable if a patient requested access to it?
Would we be comfortable if it became discoverable in litigation?
Do we even know what our AI systems are storing, synthesizing, and retaining?
The answers may shape the next chapter of healthcare transparency.