Evaluating long-horizon strategic reasoning remains a significant hurdle for large language models (LLMs). While agentic models now tackle complex tasks, most benchmarks are short and episodic, making it difficult to distinguish between stored knowledge, reasoning, and luck. Researchers have introduced SpireBench, a framework where models play the complete commercial deck-builder Slay the Spire (StS), a game where early-game decisions dictate long-term survival.
The Gap Between AI and Experts
Across 20 LLMs tested at the game's highest difficulty setting, frontier models successfully outperformed heuristic baselines and matched the performance of average human players. However, they remain significantly behind human experts. This performance gap widens as the run progresses; while the strongest models can clear the first act as reliably as experts, their success rates plummet in later stages of the game.
Misjudging Risk and Growth
The study identifies that the primary differentiator between strong and weak models is their tactical proficiency in individual fights. Yet, a universal failure among all tested models is the misjudgment of risk over the course of a full run. Instead of strategically spending health to acquire power-ups and grow stronger—a key expert tactic—models tend to be overly cautious, preserving health at the expense of long-term viability.
Prior Knowledge and Memory Failures
To determine if models were truly reasoning or simply recalling game guides, researchers implemented a comprehensive renaming of all game terms. This revealed that a portion of their performance relies on prior knowledge of the game. Furthermore, the models did not show improvement from the memories they recorded across different runs; these self-written notes were frequently inaccurate and hindered performance by being overly cautious.
