Acquiring access to AI models via API involves more than selecting a name; it constitutes a time-specific agreement. A recent study suggests that these contracts encompass the specific model version, reasoning effort settings, safety constraints, and pricing structures.
The Sonnet 5 Experiment
Researchers analyzed this framework using the Sonnet 5 model, comparing requests with explicit high-effort settings against those where the effort parameter was left out. The experiment utilized 30 problems from the AIME 2026 mathematics competition, performing five API calls for each item.
The data highlighted significant differences in financial and performance outcomes:
- Increased Costs: The average cost per call was $0.01031 higher when high effort was explicitly requested.
- Efficiency Gap: The cost for each correct response reached $0.08665 under the high-effort contract, compared to $0.07662 when the effort term was omitted.
- Accuracy Findings: While a slight accuracy increase of 0.0133 was observed, it was not statistically significant. However, the study notes that a gain of up to 4.67 percentage points cannot be entirely ruled out.
Model-Specific Semantics
The investigation found that the implications of omitting reasoning effort vary by model, even among products from the same provider. When the underlying response structure is unclear, performance claims are categorized as "documentation grade." To maintain scientific rigor, the study's methodology—including the analysis pipeline and statistical plan—was finalized before the results were reviewed. Consequently, the findings are specific to the model, tasks, and timeframe analyzed.
