Evaluating Large Language Models (LLMs) often relies on benchmarks designed to measure specific traits like safety or reasoning. However, new research introduces BenchMIRT, a method for auditing these benchmarks at the level of individual prompts to see what is actually driving a model's score.

Psychometrics Meets AI

BenchMIRT draws from Item Response Theory (IRT), a psychometric technique used to measure abilities from patterns of test responses. Instead of looking only at aggregate scores, it analyzes individual tasks to estimate which underlying capabilities are most closely associated with getting them right. By applying multidimensional IRT across 100 models and 34,000 questions, the analysis independently identified two dominant dimensions: safety and general reasoning.

Signals Inside Popular Benchmarks

For many benchmarks, BenchMIRT confirmed their intended focus. However, it revealed a more complicated picture in certain evaluations:

  • BBQ: Designed to test social bias and often grouped with safety benchmarks, BBQ aligned much more strongly with general reasoning. A low score may reflect difficulty reasoning through evidence rather than safety behavior alone.
  • WMDP: This benchmark for dangerous dual-use knowledge (such as biology and cybersecurity) was more strongly associated with general reasoning than safety. Stronger reasoning was linked to lower scores because the benchmark rewards refusing to provide dangerous information.
  • HarmBench: While its standard questions aligned with safety, its copyright-related tasks (like generating song lyrics) were more closely associated with general reasoning.

Efficiency and Potential Risks

BenchMIRT can also identify which questions are most informative. Researchers found that keeping only 10% of the most effective questions preserved nearly the same picture of model capabilities as the full set. While this allows for leaner evaluations, it presents a risk: the same transparency could be used to remove informative questions, creating weakened benchmarks that an unsafe model might pass. Despite this, the method offers a step toward more focused and interpretable LLM evaluation.