Long-horizon tasks represent a significant frontier in the evaluation of Large Language Models (LLMs). When each step in a process depends on the success of the last, per-step accuracy often decays catastrophically as errors cascade. A new study addresses this by isolating state-tracking capabilities through a clean, deterministic test: the step-by-step execution of the MD5 cryptographic hash.
The MD5 Benchmark
The researchers implemented MD5 from scratch based on RFC 1321, requiring the model to navigate a sequence of 196 dependent tool calls across 64 rounds. To succeed, the LLM had to carry four 32-bit words (a, b, c, d) within its own context from one call to the next. Unlike typical agentic benchmarks, this method prevents shortcuts like hallucinating a final answer, as every single bit must be tracked and transformed correctly according to the ground-truth trace.
Key Findings and Model Performance
The study utilized gpt-oss-120b, a mixture-of-experts model featuring approximately 5.5B active parameters per token. At temperature 0, the model successfully carried the full state across all 196 calls, returning the correct digest in a majority of completed runs. The researchers identified two critical ingredients for success that do not require changing the model's weights:
- Reasoning Retention: Keeping the model's own reasoning process within its context during each turn.
- Worker Voting: Implementing voting over a 'thinking-enabled' worker to eliminate modular-arithmetic slips.
In its most challenging configuration, the experiment replaced every primitive tool with a second LLM. This meant a driver and a worker LLM computed the entire hash from scratch without an exact-arithmetic oracle in the loop, proving that LLMs can maintain rigorous bookkeeping over extended sequences when properly structured.
