LLMs Show Variable Safety in Clinical Medical Testing
A new study reveals that popular models differ significantly in guideline-concordance and safety when evaluating rare conditions.
Updated on Oct. 28, 2026 in Artificial Intelligence
Live Poll
Do you trust artificial intelligence tools to provide accurate medical information for your health concerns?
Researchers evaluated the clinical accuracy and safety of four public large language models (LLMs) regarding Charcot neuro-osteoarthropathy. The study found significant disparities, with ChatGPT outperforming Claude, DeepSeek, and Doubao in guideline adherence and safety.
Why it matters
Inaccurate AI responses regarding Charcot neuro-osteoarthropathy may delay critical clinical actions like offloading and imaging. Establishing benchmarked performance for these models is essential for assessing their role in professional medical support.
In a 80-response test across 20 clinical questions, ChatGPT maintained a 5.0 median guideline-concordance score and a 0% safety-critical error rate. In contrast, Claude, DeepSeek, and Doubao recorded error rates of 70%, 80%, and 100%, respectively.
The players
ChatGPT
A conversational AI model developed by OpenAI that uses large-scale transformer architectures to generate human-like text.
Claude
An AI assistant built by Anthropic, designed with a focus on constitutional AI and safety-aligned performance.
DeepSeek
A research-focused AI lab and large language model provider based in China.
Doubao
An AI chatbot service developed by ByteDance that integrates large-scale language model capabilities into consumer-facing applications.
The details
Two clinicians independently evaluated platform-masked LLM responses using established clinical benchmarks, with a senior clinician adjudicating discrepancies. The study tested 20 standardized questions across 10 clinical domains related to Charcot neuro-osteoarthropathy—a progressive condition affecting the bones and joints of the foot in patients with diabetic neuropathy. ChatGPT provided authentic sources with direct claim-source linkage in every tested instance, while Doubao recorded a 15% major error rate.
Timeline
October 28, 2026: Article publication date.
The Tech Race
This study aligns with the broader push to validate large language models against established clinical quality and safety benchmarks. It highlights the wide variance in performance currently existing between top-tier models when applied to specific medical domains.
Clinicians and health professionals should note that current LLMs demonstrate significant variability in clinical safety and reliability for complex conditions. These findings suggest that output verification remains a necessary requirement for any professional medical workflow utilizing AI assistance.
The takeaway
The study demonstrates that LLM performance in clinical settings varies drastically, with only some models currently providing reliable source-linked medical information. Readers should track upcoming prospective studies that aim to test whether these safety gaps persist in real-world patient care-seeking scenarios.
What happens next
Researchers plan to conduct further prospective evaluations of public large language models specifically focused on health literacy and care-seeking behaviors.
Further reading
Explore more developments in Artificial Intelligence as systems increasingly intersect with clinical medicine.
Live Poll
Do you trust artificial intelligence tools to provide accurate medical information for your health concerns?
