What It Takes to Build a Medical Database
From fragmented patient records to Al-ready insights.
A recent Nature Medicine commentary proposed a pan-Nordic AI-health data infrastructure that links registries across six countries to train foundation models on 30+ million lives. It brought up a topic I want to discuss in today’s article: What has to happen to patient records before they are collected into a database for AI models to digest?
Commercial and public databases are now advertising longitudinal data, sometimes covering the entire lifetime of a patient. How does that data get assembled if it is de-identified? How do you link records from multiple sources belonging to a single patient while maintaining anonymity? And does that still allow you to sell it without patient consent? Let’s take a closer look.
As always, if you enjoy reading, subscribe and tell a friend.
Sam
Building the database
A medical database is just a collection of clinical events like diagnoses, labs, meds, procedures, notes, images, and vitals, from multiple systems, reorganized into something a researcher (or a model) can actually use.
Two things about that process are important to examine if you are considering contributing your data or purchasing access.
Where the data came from determines who’s in it. A database built from hospital EHRs only contains people who were sick enough and able to get to those hospitals. One built from insurance claims only contains people with health insurance. One built from a volunteer research cohort only contains people who chose to sign up. And we know that volunteers are healthier, wealthier, and more educated than the general population they’re supposed to represent. A neutral, unbiased sampling is extremely difficult to collect. Every database has some limitation.
Consent isn’t uniform. Prospective cohorts like UK Biobank, All of Us, and the Nurses’ Health Study enroll people who opt in and know their data will be used for research. But most commercial products don’t work that way. They’re built from clinical data that already existed because a patient was treated under HIPAA rules, with no separate research consent involved. The patient whose ED visit becomes a row in Truveta or Optum or Oracle Health’s dataset almost certainly has never heard of any of those companies.
What “de-identified” means
Under HIPAA, “de-identified” has a specific legal definition with two methods listed in 45 CFR §164.514.
Safe Harbor means you strip all 18 specified identifiers: names, any geography smaller than a state, every date more specific than year, phone numbers, SSNs, medical record numbers, device IDs, full-face photos, and more. Also, you must have no knowledge that what’s left could still identify someone.
Expert Determination is the other route. A qualified statistician determines that the risk of re-identifying anyone in the set is “very small.” This option allows more granular detail to remain in the database, like month-level dates and smaller geography.
If you can accomplish either method, the data is no longer Protected Health Information. Once health data is de-identified, the Privacy Rule doesn’t apply to it anymore. No authorization is needed to use it, sell it, or share it because those rights only attach to PHI. So a patient’s ED visit or hospital stay, once properly de-identified, can be aggregated, licensed, and sold to a pharma company or an AI developer, and that patient has no way to find out it happened.
The longitudinal problem
When trying to collect data over a long period of time (longitudinal data), there are additional complications.
A longitudinal database links the same patient’s records across time: the ED visit in 2019, the diagnosis in 2021, the hospitalization in 2023, all tied to one underlying person so you can see a trajectory instead of individual snapshots. That linkage is what makes disease-progression modeling possible. It’s the “digital twin” concept mentioned in the Nordic paper.
But linkage needs something to tie the records together. And that something is, by definition, an identifier. Which means the more rigorously you strip identity, the harder it gets to prove two records belong to the same person.
HIPAA planned for this under §164.514(c). A covered entity can assign a re-identification code (or token) to de-identified records, essentially letting them be linked as long as the code isn’t derived from anything about the individual. It may not be a portion of a SSN, a birthdate, or MRN. That code also may not be disclosed to whoever gets the data. That way the dataset stays legally de-identified even though a linking key exists somewhere behind a firewall. This is basically how tokenization services like Datavant work. They generate non-PHI-derived tokens so separate data sources can recognize “same patient” without ever exposing who that patient actually is. Atropos Health’s evidence network runs on this kind of tokenized linkage, by its own description.
Now compare that to the Nordic model in the paper I mentioned earlier. Nordic countries link registries using government-issued personal ID numbers. These identifiers often encode birthdate and are held by national authorities specifically so records can be traced back to identity. Under GDPR, that’s not de-identification. It’s “pseudonymization,” a legally separate, lesser category. GDPR Article 4(5) and Recital 26 are very specific about it: if re-identification remains possible through a separately held key, the data is still “personal data” and GDPR still fully applies.
So, HIPAA treats "unique identifier" and "de-identified" as interchangeable, but that's not GDPR's approach. By GDPR definition, an identifier that enables linkage is evidence that the data isn't anonymous. It's a privacy improvement over raw identifiable data, and it's the mechanism that makes cross-registry Nordic research possible.
That leaves us with two ways to get a longitudinally linked, de-identified dataset.
Keep a compliant re-identification key around, like the tokenization approach above, so the database can keep linking new records to existing patients indefinitely. That’s how vendors like Truveta and Oracle Health can advertise daily updates.
Assemble the whole dataset first, using real identifiers in a locked-down environment to do all the patient-level linking, and then strip identifiers as a final, one-way step before anything leaves the building. That produces a dataset that’s genuinely de-identified with no lingering key anywhere. But it means all the linking work has to be done before de-identification happens, which means what gets released is a closed, static snapshot. Once the identifiers come off, you can’t add a new record for an existing patient without reopening the identified stage and starting over. No version of this approach gives you a “living,” continuously updated database. You can only produce a new static version periodically.
That’s why MIMIC, NHANES, SEER, and the National Inpatient Sample ship as periodic releases instead of live updates. Meanwhile, the commercial products marketed as daily or real-time updates rely on some kind of retained linking key. Neither approach is wrong. They’re just not the same privacy posture, and “de-identified” alone doesn’t tell you which one you’re looking at.
So what’s actually out there
Not everything called a “medical database” is the same. Some are commercial products with a sales team. Some are federally funded public resources. Some are prospective research cohorts that have been running for decades. Below is a breakdown of nineteen of the big databases.
Two philosophies, same underlying risk
Another way to sort databases on the list is by where the data lives.
Centralized (pooled) means the data physically or virtually lives in one place the vendor controls. Truveta, Optum, MarketScan, Oracle Health, Flatiron, and UK Biobank are examples of this strategy.
Federated is the opposite. The data never leaves the institution that generated it. A query gets sent to the data, and only aggregated results are returned. This is the model the Nordic commentary is suggesting. PCORnet and OpenSAFELY are the cleanest examples of it. TriNetX and Atropos run hybrid versions with a federated query / tokenized-network layer sitting on top of member institutions’ own systems.
The Nordic paper frames this as a design choice, proposing federated approaches for the most sensitive data, pooling for standardized and less sensitive data, and synthetic data for pretraining where either option is too risky. It’s a layered strategy, depending on the data needed and the use case.
Centralized storage has real strengths. It supports multimodal model training across notes, images (radiology), and waveforms (EEG, ECG). One unified data model, so a single researcher can iterate fast. But it also means a single point of possible security failure. UK Biobank, despite a strong governance reputation, recently had a data breach that the Nordic authors cited as a cautionary tale. It requires enormous cross-institutional trust and legal paperwork to move data across borders. And it concentrates security and the exact de-identification methodology inside one vendor.
Federated storage changes most of that. Data never crosses institutional or national borders, which avoids a lot of cross-border transfer risk under GDPR or the EU’s European Health Data Space. It preserves local control, and it tends to earn higher public trust for exactly that reason. The trade-off is that it’s hard to train large multimodal foundation models this way because you can’t combine raw pixel (image) or waveform (EEG, ECG) data across sites. So you’re stuck with averaged or aggregate query results. Site-to-site differences in EHR vendors and coding practices make joint analysis messier. And it’s structurally harder for anyone outside the network to check what’s in the underlying records.
What trips people up
If you’re a researcher picking a data source, or a clinician trying to figure out what a new AI tool was actually trained on, the marketing page rarely answers the following important questions:
Bias: Who’s in the database. Hospital EHR data skews toward people sick enough to seek care. Claims data cuts out the uninsured entirely and skews toward employed populations. MarketScan describes its data as commercially insured workers and dependents. Volunteer cohorts carry the opposite bias: UK Biobank, the Nurses’ Health Study, and All of Us recruit people healthier, wealthier, and more health-literate than the general population. Single-site or single-region resources tell you something true and useful about a very narrow slice of the world: MIMIC is one Boston ICU, OpenSAFELY is England only, and the Nordic proposal is 30 million largely homogeneous Northern Europeans.
Verifiability: This is really one question asked in two parts.
At the research stage: Can anyone check how the thing was built? Open resources like MIMIC/PhysioNet, NHANES, SEER, HCUP, and OpenSAFELY publish their cohort-construction and de-identification methodology. OpenSAFELY goes a step further by publishing its analysis code for anyone to inspect. Commercial vendors treat the exact tokenization and linkage methods as a trade secret. So a peer reviewer can look at the output but can’t audit the data.
At publication: Journals increasingly expect data and code availability statements, but licensed commercial (purchased) data generally can't be redistributed alongside a paper. An independent lab can't rerun your exact analysis on your data. Open resources solve both problems by staying accessible under the same public terms to anyone who wants to check.
Access: This determines what kind of work you can even attempt. If you can download it to your own servers (MIMIC, NHANES, SEER*Stat, NIS), you get full control, but you’re now on the hook for securing it. Cloud-locked platforms (UK Biobank’s RAP, All of Us‘s Researcher Workbench, Truveta Studio) let you analyze but not export raw records, which rules out a lot of machine learning (transfer-learning and fine-tuning workflows) that needs raw data. And federated-query-only databases (PCORnet, OpenSAFELY, TriNetX) never give you patient-level data at all. They are great for privacy, but you can’t train a multimodal model on it.
And then there are these other issues:
Fees can run $10,000 to $25,000 or more per study for commercial claims data. This favors well-funded institutions and industry sponsors over independent academic labs.
Claims-based sources tend to run one to two years behind current practice.
Probabilistic matching introduces real errors when tokenized records from data sources that were never designed to talk to each other get linked anyway.
HIPAA, GDPR, and the UK’s Data Protection Act each define “de-identified” a little differently, which is exactly the harmonization problem the Nordic Comment’s roadmap lists as a step that still needs solving.
Takeaway
The Nordic paper says that the scientific infrastructure “is not aspirational: it is substantially in place.” Read against everything above, that claim seems premature for a plan that still has to establish data standardization, sort out cross-border regulatory alignment, and prove clinical feasibility.
For me, that gap isn’t unique to the Nordic proposal. It’s sitting underneath every “trained on X million de-identified patients” claim we’ll read this year. The architecture, the access model, the update cadence- none of that should be a footnote. It predicts exactly where the resulting model will hold up and exactly where it’ll quietly fail.