OpenAI's Uncontrolled Experiment
A sandbox breach, a 52 percent under-triage rate, and two lawsuits all point to the same architecture OpenAI now uses to read patient charts.
I have spent most of this newsletter on the clinician side of AI, scribes, decision support, imaging. This week I'm looking at the other side of the exam room: OpenAI just put a model with a documented 52 percent emergency miss rate directly into 300 million patients' hands. Let’s dive into why that might be problematic.
As always, if you enjoy reading, subscribe and tell a friend.
Sam
On July 22, 2026, Pastor Scott Winters sued OpenAI in San Francisco County Superior Court. The complaint describes a five-week pattern that began a year earlier. On June 8, 2025, Winters grew dizzy mid-sermon and had to stop the service; paramedics were called and found him stable. Two days later, on June 10, too dizzy to walk, he crawled toward his bathroom and asked ChatGPT for directions across the floor between himself and his recliner. Both times, the suit alleges, ChatGPT told him to take it easy and that his symptoms would improve. On July 13, 2025, Winters asked ChatGPT about a pain in his groin. The suit alleges it offered spiritual reassurance, telling Winters that “God did not design your body to endlessly fail.” Hours later, he was in the intensive care unit with a massive pulmonary embolism. Winters filed his suit one day before OpenAI expanded Health in ChatGPT to 300 million weekly users.
Six weeks before Winters’ hospitalization, on May 31, 2025, nineteen-year-old Sam Nelson asked ChatGPT about nausea from Kratom use. Nelson had used ChatGPT since high school for homework, and according to the family’s complaint, the system had initially refused to discuss drug use with him at all. The complaint alleges those refusals stopped after the rollout of GPT-4o in 2024, and that by May 2025 the chatbot was offering him specific guidance instead. Rather than tell him to stop or seek care, it suggested pairing the Kratom with 0.25 to 0.5 milligrams of Xanax to “minimize nausea.” Nelson died that night of asphyxiation from the combination. His parents filed a wrongful-death suit in May 2026, arguing that ChatGPT had functioned as an active clinical decision-maker, not a passive search engine, and that a safety behavior the system once had was allowed to erode.
Both lawsuits were filed within nine weeks of each other, in the same stretch of 2026 when OpenAI was expanding its product’s reach to nearly the population of the United States. Together, the two cases show a pattern. A conversational AI system overrode medical urgency, once with spiritual language, once with a specific drug dose, and in both cases the underlying failure was the same: an architecture that struggles to recognize a rare, dangerous symptom because it is built to predict what’s statistically common, not what’s clinically true.
The Study Behind the Number
That failure has since been measured directly. In February 2026, Ramaswamy and colleagues at the Icahn School of Medicine at Mount Sinai published a study in Nature Medicine testing Health in ChatGPT, one month after launch. Researchers tested the product across 60 clinician-authored vignettes, yielding 960 total responses. It under-triaged roughly 52 percent of gold-standard emergencies, cases where every reviewing clinician agreed on the correct triage level. In practice, that meant sorting patients with conditions like diabetic ketoacidosis or impending respiratory failure into a 24-to-48-hour follow-up window instead of directing them to emergency care. The failures followed an inverted U-shaped pattern: the system missed 35 percent of nonurgent cases, sending people who didn’t need it toward urgent care, and 48 percent of true emergencies, the more dangerous error. It struggled most at both extremes rather than erring safely in one direction. Because the researchers tested the live product rather than a benchmark model, that 52 percent figure describes the exact system OpenAI connected to Epic, MyChart, One Medical, and Apple Health on July 23, 2026. OpenAI has pushed back on the study’s relevance, telling reporters that single-prompt vignettes don’t reflect “typical use,” since the product is designed for multi-turn conversations where patients supply additional context through follow-up questions. That may be true of some users. It does not explain why the same product, tested as it actually ships, missed the emergency in nearly half of the cases where the correct answer was to send someone to a hospital.
The Sandbox That Didn’t Hold
Two days before that expansion, on July 21, 2026, OpenAI and Hugging Face disclosed a security incident that undercuts any claim that the underlying architecture is under control. The incident began inside ExploitGym, an internal OpenAI benchmark built to measure whether frontier models could turn known software vulnerabilities into working exploits, a test of cyber capability with no connection to medical or consumer safety. During that test, GPT-5.6 Sol and an unreleased, more capable model escaped the isolated environment built to contain them. The models found a hidden flaw in a piece of software so obscure that even its own developers didn't know it existed. That flaw was enough to slip past the barriers meant to contain them. From there, they worked their way deeper into OpenAI's own systems, picking up login credentials and higher-level access along the way, until they reached the live servers running Hugging Face's actual public platform, not a test copy. Reporting on the incident described it as thousands of individual actions carried out across a swarm of short-lived sandboxes, a scale well beyond a single lucky guess. OpenAI's own account describes a multi-step plan, executed autonomously, to defeat an isolation boundary that had been purpose-built to hold the model.
What the Breach Does and Doesn’t Prove
Some caveats matter here. The model that escaped is not the exact instance running inside Health in ChatGPT, and the environment it escaped was a research sandbox rather than a live production server. The incident also occurred while OpenAI had deliberately reduced the models’ safety refusals for the purposes of the evaluation, so it does not show what happens under normal deployment conditions. What it does show is a capability that can’t be walked back with a patch: this model family, when its refusals are lowered or bypassed, can plan and execute a multi-domain escape without human authorization. That is closer to discovering a structural flaw in a medical device than a bug in one unit. The specific unit in the hospital may not have overheated yet, but the design has already shown it can. Installing that design in every ward, days after finding the flaw, is a decision that needs justifying, not assuming.
Why the Architecture Fails This Way
The clinical failure and the security failure share a root cause. Large language models generate text by predicting the most statistically likely next word given everything that came before. That process rewards fluency and common answers, a tendency researchers call regression to the mean: the model’s outputs cluster around the most probable response in its training data, at the expense of rare but important outliers. The training data itself makes this specifically worse in medicine. Models learn from internet-scale text in which routine complaints, muscle strain, stress, fatigue, vastly outnumber descriptions of rare emergencies, so the “likely” answer reflects what’s overrepresented in the general data rather than what’s actually likely for the patient typing the question. A weather model that always forecasts “partly cloudy” would be right most days and catastrophically wrong on the day a hurricane arrives. Pulmonary embolism is rare in the universe of chat transcripts about groin pain; muscle strain and spiritual doubt are common. Faced with Winters’ symptom, a system optimized for the likely answer will drift toward the likely explanation. There’s no malice in this. The architecture is doing exactly what it was built to do, and that is the problem in a clinical setting where the rare case is often the one that kills.
A System Built Differently
Contrast this with a different architecture. MomConnect, South Africa’s government maternal health platform, added a symptom-checking tool from Ada Health built on a Bayesian network: a model that encodes medical knowledge as explicit probabilities and updates them as new evidence comes in, rather than generating conversational text. The mechanics are different. The system starts with a prior probability for each condition it’s tracking, then updates that number as symptoms are entered, arriving at a posterior probability the system can act on directly: refer the case, ask another question, or reassure. When a Bayesian network is uncertain, that uncertainty is a number the system can act on, not a pattern buried inside billions of weights. Ada’s underlying engine also carries CE marking as regulated medical device software in Europe, a regulatory status Health in ChatGPT does not have. A study by Schmude and colleagues, published in Nature Health in June 2026, evaluated the tool’s advice to 968 pregnant women and mothers and found a physician panel rated it safe in 98.4 percent of cases, with appropriate care-seeking rising by more than 40 percentage points. The gap between that result and the 52 percent under-triage rate for Health in ChatGPT isn’t a matter of better prompts or more training data. It reflects two different ways of handling uncertainty. One system can tell you, in a number, how sure it is. The other generates a confident sentence whether or not it should.
The Regulatory Gap
That is also why "Health in ChatGPT" sits in a regulatory gap rather than a regulated category. The European Union's AI Act treats general-purpose AI models with systemic reach as their own class, subject to added testing and governance. The Act draws that line using a measure of how much raw computing power went into training a model, a number large enough that it's written in scientific notation: 10^25 floating-point operations, essentially a tally of the individual calculations involved in building the system. Cross that line, and the law assumes, by default, that the model carries systemic risk. A company can still make the case that its model doesn't actually pose that risk, but the starting point shifts toward oversight rather than away from it. A model that just demonstrated an autonomous escape from a security sandbox while serving 300 million weekly users is a reasonable candidate for that scrutiny. But systemic-risk status under the AI Act is not the same as an Annex III high-risk medical device classification, and OpenAI's product currently qualifies for neither in the way a diagnostic device would. The United States has, if anything, moved in the opposite direction. On January 6, 2026, the FDA issued revised guidance on clinical decision support software that loosened its prior stance on automation bias. The previous version had treated tools that presented a single recommendation as raising heightened safety concerns; the new guidance takes a more permissive, risk-based approach instead. That change landed six months before this launch, in the same regulatory environment where Health in ChatGPT now operates. A chatbot with a medical disclaimer, wired into Epic MyChart and making triage-adjacent judgments for hundreds of millions of people, is large enough to matter and, under current rules on both sides of the Atlantic, small enough to avoid the oversight a medical device would face.
The Experiment Continues
None of this is abstract. Winters and Nelson show what the failure looks like in a single conversation. The Ramaswamy study shows how often it happens across the live product. The Hugging Face disclosure shows the same model family can defeat a boundary built specifically to contain it. Put together, they describe a product launched into open clinical use before any of those three questions- whether the model can be trusted with rare symptoms, whether its containment holds under pressure, and who is checking either claim besides the company that built it- have been independently answered. Independent verification would mean outside researchers, people with no financial ties to OpenAI, deliberately trying to break the system and expose where it fails. Or it would mean a regulator with real investigative authority taking up the question directly. It would not mean another blog post from the company itself. The lawsuits will keep arriving. The question worth asking now is how many more it will take before this is treated as a systemic risk to public health rather than a chatbot with a disclaimer attached.