The launch of OpenAI’s GPT-6 Astra model has been marked by an unusual series of revisions to performance data published on the company’s official blog. Since the afternoon of September 3, the metrics defining the model’s capabilities have fluctuated, with some adjustments favoring Astra over its primary rival, Anthropic.
A Turbulent Rollout
The announcement’s publication was delayed by nearly two hours due to technical issues, which CEO Sam Altman described as a "little snag." However, OpenAI later admitted it briefly retracted the post for undisclosed reasons, though it maintained these were unrelated to benchmark figures. Upon republication, the metrics had already begun to shift.
Fluctuating Hallucination and Math Scores
One of the most striking revisions involved Astra’s hallucination rate. Initially reported at 4.2%, the figure was halved to 2% in a subsequent version of the page, only to eventually revert to the original 4.2%. Similar volatility was seen in mathematics performance (FrontierMath), where scores for Anthropic’s Fable 5.1 model were briefly lowered from 87.8% to 78% before settling at 83%.
The Rise of 'Benchmaxxing'
The practice of repeatedly running evaluations under optimal conditions to achieve the highest possible score, often termed "benchmaxxing," has drawn scrutiny. Researchers from Stanford University noted that Astra’s system card lacks sufficient technical detail regarding how specific internal evaluations were conducted, such as the number of test items used.
OpenAI defended the revisions, stating that evaluations inherently contain "noise" and that adjustments were made to ensure numbers represent the best estimate of model performance. Nevertheless, the lack of metric stability complicates the narrative for investors and customers, particularly as the company eyes a potential IPO in 2027.
