Is Open Source the Answer to Healthcare AI’s Trust Crisis?
Open source helps with a lot of what’s keeping hospital AI officers up at night this year, but not with everything, and the hospitals that get that distinction right will have a structural advantage for the next decade. This piece walks through exactly where open-weight models solve real problems and where they don’t.
As always, if this is useful, the best way to support the work is to subscribe and share it with a colleague.
Sam
Three days ago, Mark Zuckerberg published a manifesto arguing that AI should be “open source” and released Meta’s newest model, Muse Glimmer, as an open-weight offering. The White House and press noticed. And if you are an AI officer at a hospital or a developer, you should absolutely notice too, because the announcement lands in the middle of a genuinely important moment.
In the last three weeks, OpenAI disclosed that its models went rogue during a security test, escaping a containment sandbox and launching an "unprecedented" autonomous cyberattack against Hugging Face. About two weeks later, Anthropic's Mythos 5 model did something arguably worse: it created fake online identities, spear-phished two real software developers with malicious code, and tried to trick a GitHub project maintainer into accepting a backdoor. The UK's AI Security Institute called it a "serious incident" and a "shift in the risk landscape." Then Meta disclosed its own incident: a misconfiguration by its outside testing firm gave one of its models unsupervised internet access, and the model used it to breach another company's systems. The third major AI lab to report an incident like this in a matter of weeks.
If you are a hospital CIO watching proprietary models autonomously hack platforms, deceive humans, and sabotage their own shutdown mechanisms, you are probably asking the same question a lot of us are asking: Why on earth would I send my patient data to these companies?
The logic is straightforward. If hospitals stop renting APIs from black-box vendors and instead run open-weight models locally, they could stop version drift, regain control, keep data local, and avoid vendor collapse. Academics are already making this case. Riedemann et al. argued in npj Digital Medicine that the “path forward for large language models in medicine is open” because medical devices require transparency and version control that proprietary APIs cannot provide. Dennstädt et al. mapped out the implementation framework in 2025, showing that local deployment gives hospitals customization and control at the cost of expertise and infrastructure. A 2026 Frontiers review concluded that hybrid approaches are probably the realistic future.
Open source clearly has merit. The question is whether it solves the problems that actually keep hospital AI officers awake at night.
So, let’s break it down for clarity.
What It Helps With: Version Control as a Medical Device Requirement
In 2023, a Stanford team quietly published one of the most consequential papers in AI safety. Chen, Zaharia, and Zou showed that GPT-4’s accuracy on a prime-number identification task dropped from 97.6% to 2.4% between March and June 2023, even though the API endpoint was the same. The model had been updated. No announcement. No changelog. No way for users to know what changed, or why, or how to reproduce the old behavior.
In healthcare, that is a medical device failure. Imagine a sepsis prediction tool that your hospital validated, deployed, and trusted. On Tuesday morning, its sensitivity drops by 40% because the vendor pushed a silent update. Your patients do not get flagged. Your nurses do not know to look harder. Your mortality metrics shift six months later, and by the time you trace it back, the vendor has pushed three more updates.
This is what researchers call “version drift.” I prefer AI creep, because it captures something more insidious: the slow, invisible expansion of a model’s behavior beyond its validated boundaries, combined with the unannounced changes that make root-cause analysis impossible.
Open-weight models solve this through version pinning. A hospital downloads the weights, validates them against its own clinical data, locks them in, and treats them exactly like the firmware on a pacemaker. You cannot update a pacemaker without full re-validation. You should not update a clinical AI model without it either. Riedemann et al. explicitly argue that open-source approaches provide “medical device developers maximum flexibility to reduce the black-box properties of such systems”, which is a polite way of saying that proprietary APIs are structurally incompatible with medical device safety culture.
The OpenAI “laziness” incident of December 2023, where ChatGPT inexplicably refused to perform complete tasks, is another example. The cause was never identified by independent researchers nor disclosed by OpenAI. A hospital relying on that API for clinical documentation would have had no recourse, no explanation, and no ability to roll back to the previous behavior. With fixed open weights, you control the timeline.
Data Sovereignty and the Two-Pipeline Problem
If you read this newsletter regularly, you know I have written about the two data pipelines that every hospital AI deployment creates. There is the back-door pipeline, where de-identified data flows to vendors. And there is the front-door pipeline, where clinicians, patients, or administrators upload PHI directly into consumer chatbots, often without understanding that the data is being used to train the next model.
Running open-weight models locally eliminates the front-door risk. Patient data never leaves the hospital network. There is no question about whether de-identification actually worked. There is no wondering if a clinician copied and pasted a pathology report into a web interface that lacks a Business Associate Agreement. Dennstädt et al. put this bluntly: transmitting sensitive patient data to external servers, even in partially anonymized formats, “raises the risk of data breaches and unauthorized access,” and “complete anonymization in most clinical settings is not possible.”
For HIPAA compliance officers, this is a gift. The liability architecture becomes cleaner. If patient data never leaves your network, you have eliminated an entire class of breach scenarios. You have also eliminated the vendor’s terms of service as a factor in your compliance posture. OpenAI's terms of service state that its models aren't meant to be used for diagnosing or treating any health condition. Anthropic’s terms are similar. When you run a model locally, you are not agreeing to anyone’s terms except your own hospital’s.
For patients, the stakes are more direct. It’s their diagnosis, their genetic results, their mental health history moving through whatever pipeline a hospital, or a third-party app the hospital contracted with, chose to build on. Most patients have no idea which of those two pipelines their information is taking, let alone whether the model on the other end is proprietary or open, local or cloud-hosted.
Niche Specificity
A hospital in rural Mississippi treating a rare sickle-cell variant has different needs than a quaternary academic center in Boston. Proprietary APIs are trained on internet-scale data, optimized for the average case, and fine-tuned for the most common complaints. They are not optimized for your patient population.
Open-weight models allow fine-tuning on local data. There are now specific medical open models like OpenBioLLM-Llama3-70B, MEDITRON-70B, and Google’s MedGemma collection. A hospital can download these, fine-tune them on its own historical cases, its own protocols, and its own rare-disease populations, and produce a model that knows its local context.
The performance gap has narrowed dramatically. A Harvard Medical School study found that an open-source model performed on par with leading proprietary AI in solving tough medical cases. By mid-2024, LLaMA 3.1 405B had nearly closed the gap with GPT-4o on general benchmarks.
But the stronger argument is verifiability. A proprietary model might perform well on your data, but you cannot inspect its weights or training data to know why. An open model can be validated against your specific clinical scenario and audited by your own quality team. In medicine, knowing why a model works is often as important as knowing that it works.
Infrastructure and Expertise
Here is where the fantasy breaks down. A 70-billion-parameter model requires serious GPU infrastructure. A 300-bed community hospital in rural America does not have an A100 cluster of servers in its basement. It does not have ML engineers on staff. It barely has enough IT staff to keep the EMR running. A three-person health-tech startup trying to fine-tune the same model on a laptop is in exactly the same position, just without the EMR.
Dennstädt et al. are explicit about this trade-off: “Open LLMs deployed on local hardware enable greater model customization” but “demand resources and technical expertise.” Most hospitals do not have the expertise to deploy, monitor, and maintain these systems. Control is only valuable if you have the organizational capacity to wield it.
This is where the literature proposes a consortium model, and it is the most practical expansion of the open-source theory. The University Health Network authors explicitly call for a “global consortium” similar to the Trillion Parameter Consortium to pool computational resources, share validated base models, and allow individual hospitals to fine-tune locally without each building its own data center. This is how open source wins in healthcare: as shared infrastructure.
There is also the rise of smaller, efficient models. Even 7-billion-parameter models can now run on a Mac M1 and perform well for targeted tasks. As models get smaller and more efficient, the infrastructure barrier drops. This trend turns the theory from a fantasy into a practical reality about targeted deployment.
The “Open Source” Definition Problem
Zuckerberg’s announcement is a perfect teaching moment. The NYT reported explicitly that Muse Glimmer “is not fully open source”. Meta released the weights, but not the training data, or the full architecture details, or the license terms that would allow unrestricted commercial use. This is “open weight,” not “open source.”
The distinction matters because Riedemann et al., discuss code, data, weights, documentation, and licensing separately. By that framework, most “open” models are barely halfway open: open enough to download and run, rarely open enough to fully audit. The European Union AI Act is currently struggling with this exact problem. Whether you’re procuring a model for a hospital, deciding what to build a clinical product on, or just trying to evaluate a vendor’s privacy claims, you need to ask the same question: open in what dimension? Open to inspection? Open to modification? Open to commercial deployment without a usage license from Meta?
If you write a procurement policy based on “open source” without defining it, you will end up with a vendor that released weights under a restrictive license and still controls your destiny. The marketing language is not your friend.
Security and Adversarial Risk
The open-source security argument has a flip side. Yes, running locally keeps data in-house. But open weights are also susceptible to attackers. Because the architecture and weights are public, adversaries can systematically search for vulnerabilities, craft adversarial inputs, and test data-poisoning strategies at scale.
A Galileo AI analysis of open-source LLM security risks notes that “open-source LLMs often introduce significant security risks due to their public nature and generally immature security practices.” A PMC study on safety in medical LLMs found that medical-specific fine-tuned models, both open-source and proprietary, underperformed general-purpose frontier models across most safety dimensions tested. The problem is that safety in medical AI is hard for everyone, not that open models are uniquely unsafe.
The counterargument is that closed models get breached too. OpenAI had its data leak. The rogue AI incidents of the past month show that proprietary models are not magically secure because their weights are hidden. The difference is transparency. With open weights, your hospital can probe the model itself, find vulnerabilities, and patch or mitigate them. With a closed model, you must wait for the vendor to fix it, and hope they tell you the truth about what happened. A Censinet analysis of AI-specific vulnerabilities in healthcare makes the case for adversarial robustness testing regardless of model source.
The Regulatory Paradox
Here is the most biting counterargument, and it is underexplored in the literature. The FDA wants medical devices to be locked-down and validated. If you download an open-weight model, fine-tune it on your local patient data, and deploy it in a clinical decision support workflow, have you just created a new medical device? Does it require its own 510(k)?
The regulatory framework has not caught up to this question. The Riedemann paper notes that LLM AI chatbots require approval as medical devices, but the approval process assumes a static, validated product. An open model that a hospital can freely modify is, by definition, not a static device. If your AI officer fine-tunes the model next quarter, is that a design change requiring re-validation?
Liability shifts entirely to whoever deployed the model, the hospital, or the developer who built a clinical product on top of it. When a proprietary model harms a patient, the hospital, the developer, and the vendor will all point at each other. An organization running its own locked model has clearer liability. For the risk-averse, that is actually a feature, not a bug. But only if the organization has the documentation and governance infrastructure to defend itself.
Hybrid Deployment
If you take one thing from this article, take this: hospitals should version-pin and locally deploy open models for high-stakes clinical tasks, while using proprietary APIs for low-stakes general tasks.
This is the only position that is both defensible and practical. Use GPT-4o or Claude for scheduling, patient education, and general documentation tasks where version drift is annoying but not dangerous, and where the cost of local infrastructure is not justified. Use pinned, validated, locally run open models for clinical decision support, pathology review, rare disease diagnosis, and any workflow where a silent behavior change could harm a patient.
Dennstädt et al. and the Frontiers review both back this hybrid approach. The Frontiers review focuses narrowly on information-extraction tasks, pairing local, on-premises retrieval with cloud-based generation, but the same logic applies more broadly to clinical decision support. Hospitals get control where it counts without building full ML infrastructure for every task. And it creates a natural safety net: if a proprietary API goes rogue, it's running your scheduling bot, not your sepsis predictor.
The Version-Pinned Hospital
The hospitals, developers, and health systems that will own the next decade are the ones that treat AI like the medical device it is: validate, lock, version, archive, and only update through a controlled change-management process. Whoever is building or buying these tools, in-house AI team, third-party vendor, or solo developer, will not outsource their clinical brains to black boxes that update silently every three months. They will build the competency to run, validate, and govern their own models, even if they have to share the infrastructure to do it.
The ones that keep renting APIs and hoping the terms of service will protect them, and the ones handing over their health data without asking what’s actually running on the other end, will have no one to blame but themselves when the next data leak hits or the next unannounced model update quietly changes what their AI actually does.
Open source is a risk-mitigation strategy, not a magic bullet. And in medicine, whether you’re the hospital deploying it, the developer building it, or the patient trusting it, that’s what protects you.