While humans naturally express uncertainty through verbal markers such as "possible" or "likely," most Large Language Model (LLM) uncertainty quantification (UQ) has traditionally relied on likelihood- or consistency-based signals. A new study published on ArXiv investigates the gap between how humans and AI interpret these linguistic cues.
The Metacognitive Divide
From a cognitive perspective, accurate verbal uncertainty reflects metacognitive monitoring—representing the boundaries of one's knowledge, or "knowing that you don't know." Researchers curated a corpus of human uncertainty markers from psychology and decision-science literature to benchmark LLMs against human standards. They observed that LLMs encode verbal uncertainty using numerical levels that differ substantially from human interpretations.
METHODNAME: A Novel Optimization Algorithm
To address this divergence, the researchers introduced METHODNAME, an optimization-based algorithm designed to learn an optimal uncertainty profile directly from LLM outputs. By fitting a marker-uncertainty mapping to best explain empirical correctness, METHODNAME discovers exactly how much probability mass each verbal marker should convey, moving away from traditional repeated sampling methods.
This approach enables a direct, marker-level comparison of confidence semantics between humans and LLMs. It effectively disentangles linguistic mismatches and reveals systematic confidence disparities in how AI models use verbal expressions to quantify their own certainty.
