What You Should Know
- The Study: In a paper published today in The Lancet Digital Health [10.1016/j.landig.2025.100949], researchers at the Icahn School of Medicine at Mount Sinai analyzed over one million prompts across nine leading Large Language Models (LLMs) to test their susceptibility to medical misinformation.
- The Vulnerability: The study found that AI models frequently repeat false medical claims—such as advising patients with bleeding to “drink cold milk”—if the lie is embedded in realistic hospital notes or professional-sounding language.
- The Takeaway: Current safeguards are failing to distinguish fact from fiction when the fiction “sounds” like a doctor. For these models, the style of the writing (confident, clinical) often overrides the truth of the content.
The “Cold Milk” Fallacy
To test the systems, the research team exposed nine leading LLMs to over one million prompts. They took real hospital discharge summaries (from the MIMIC database) and injected them with single, fabricated recommendations.
The results were sobering. In one specific example, a discharge note for a patient with esophagitis-related bleeding falsely advised them to “drink cold milk to soothe the symptoms”—a recommendation that is clinically unsafe.
Instead of flagging this as dangerous, several models accepted the statement as fact. They processed it, repeated it, and treated it like ordinary medical guidance simply because it appeared in a format that looked like a valid hospital note.
Style Over Substance
“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” said Dr. Eyal Klang, Chief of Generative AI at Mount Sinai.
This exposes a fundamental flaw in how current LLMs operate in healthcare. They are not necessarily verifying the medical accuracy of a claim against a database of truth; they are predicting the next word based on context. If the context is a highly realistic, professional discharge summary, the model assumes the content within it is accurate.
“For these models, what matters is less whether a claim is correct than how it is written,” Klang added.
The “Stress Test” Solution
The implications for clinical deployment are massive. If an AI summarizer is used to condense patient records, and one of those records contains a human error (or a hallucination from a previous AI), the system might amplify that error rather than catch it.
Dr. Mahmud Omar, the study’s first author, argues that we need a new standard for validation. “Instead of assuming a model is safe, you can measure how often it passes on a lie,” he said. The authors propose using their dataset as a standard “stress test” for any medical AI before it is allowed near a patient.
