While state-of-the-art AI models report execution accuracy exceeding 89% on academic benchmarks like Spider and BIRD, translating this success to complex enterprise environments remains a hurdle. A new study introduces ESQ-Bench, an Oracle-first benchmark designed to evaluate NL2SQL dialect generalization and "silent semantic divergence."

Enterprise Complexity vs. Academic Schemas

ESQ-Bench departs from traditional benchmarks by utilizing six fully populated schemas comprising 465 tables and 164,682 rows. The evaluation harness includes 550 gold-validated question-query pairs distributed across three systematic complexity tiers. These schemas were deployed with identical seed data across Oracle, PostgreSQL, MySQL, and SQL Server to test cross-dialect performance.

Model Performance and the Accuracy Gap

The research highlights a monotonic degradation in execution match (EX) as schema complexity increases:

  • Claude Sonnet 4.6 emerged as the top performer, reaching 87.4%, 74.9%, and 68.7% EX across the tiers, outperforming GPT-4o in every category.
  • GPT-4o saw its performance drop from 79.8% to 57.2% at the highest complexity tier when using schema-linked prompting.
  • The open-weight Llama 3.2 reached only 13.3% accuracy bank-wide, highlighting a significant gap between proprietary API models and local alternatives in enterprise Oracle settings.

The Threat of Silent Semantic Divergence

A critical finding of the study is the prevalence of "silent semantic divergence." This occurs when the generated SQL executes successfully but returns incorrect data. In higher complexity tiers, this phenomenon was observed in 73% to 99% of queries that passed the execution test, suggesting that "working" code does not equate to "correct" answers in an enterprise context.