While Large Language Models (LLMs) are now passing medical licensing examinations and rivaling physicians in curated diagnostic tasks, a new perspective paper highlights a critical safety gap in their autonomous application. The study, published on arXiv, argues that while LLMs excel at text prediction, they lack the specific reasoning required for autonomous triage of patients without a clinician in the loop.
The Probability vs. Safety Paradox
The core issue identified is not a lack of medical knowledge, but the nature of how LLMs function. A model optimized to generate the most probable text is not inherently optimized for safety. In clinical triage, safety is not about identifying the most likely outcome; it is a sequential decision process under "asymmetric cost." In this context, a single catastrophic miss of an improbable but high-harm diagnosis outweighs the cost of many false alarms. LLMs, by design, may prioritize the statistical likelihood over the "must-not-miss" clinical exception.
Information Gathering Under Uncertainty
The research points out that LLMs struggle with active information gathering when presented with incomplete patient histories. Safe triage requires specific behaviors that LLMs often fail to exhibit, such as:
- Broadening the differential diagnosis when data is sparse.
- Proactively seeking missing "red flags."
- Lowering the threshold for escalation to human experts.
- Deferring judgment until sufficient information is obtained.
Behavioral Biases in AI Assistants
The paper also warns of "assistant-like" behaviors that can compromise clinical integrity. Attributes such as credulity, agreeableness, and miscalibration can lead the model to follow a patient's lead rather than applying rigorous clinical triage logic. These failure modes are often masked in current evaluations because researchers frequently use well-curated, complete simulations rather than the messy, undifferentiated data encountered in real-world care.
