ARC Prize Foundation Challenges GPT-6 Astra Benchmark
The foundation rejected a 99.9 percent claim, citing an audit that placed the model's performance at 62.7 percent.
Updated on Oct. 24, 2026 in Artificial Intelligence
Live Poll
Do you trust that current AI advancements reflect genuine human-level intelligence?
The ARC Prize Foundation has rejected OpenAI's claim of a 99.9 percent benchmark score for its GPT-6 Astra model, citing the use of a non-standard testing harness. Verified testing under a neutral harness yielded a score of 62.7 percent.
Why it matters
The discrepancy highlights how custom execution environments can inflate AI performance claims on benchmarks designed to measure general reasoning rather than brute-force memorization.
OpenAI achieved its reported 99.9 percent score using a $19,000 custom tools-enabled harness that preserves reasoning state, while the neutral ARC-AGI-3 harness resulted in a 62.7 percent score.
The players
ARC Prize Foundation
An organization focused on creating rigorous benchmarks to measure Artificial General Intelligence and test reasoning capabilities.
OpenAI
An AI research and deployment company known for its GPT series of large language models.
Jensen Huang
The CEO of Nvidia, a semiconductor company leading in GPU architecture and AI compute infrastructure.
The details
The ARC-AGI-3 benchmark is engineered to test general intelligence, resisting gaming and memorization by requiring models to solve novel tasks. OpenAI’s custom harness maintains reasoning state across multiple actions, which the foundation argues creates an unfair performance advantage. In contrast, standard testing requires models to function within a restricted environment where they cannot leverage these external tools to persist state.
Timeline
March 25, 2026: The ARC Prize Foundation released the ARC-AGI-3 benchmark.
September 3, 2026: OpenAI announced a 99.9 percent score for GPT-6 Astra.
September 6, 2026: Jensen Huang posted on X that AGI had arrived.
Early October 2026: The open-source Prime Agent harness raised public model scores.
Late October 2026: The ARC Prize Foundation published its audit of the leaderboard figures.
The Tech Race
The dispute underscores the ongoing tension between independent performance auditing and corporate model benchmarking. It follows a rapid progression of AI capabilities, where even open-source agents have pushed performance on the ARC-AGI-3 benchmark from 30 percent to over 95 percent.
Users should treat high-percentage AI benchmark claims with skepticism until they are verified by neutral, standardized testing environments. Performance figures will continue to fluctuate as researchers develop new orchestration methods to push model capabilities toward the 100 percent benchmark limit.
The takeaway
The gap between corporate announcements and verified benchmark scores signals a need for standardized testing that accounts for external tool integration. Monitor upcoming leaderboard updates from the ARC Prize Foundation to see how standardized scores adjust throughout the remainder of 2026.
Further reading
For more context on how current models perform, read the latest analysis in Artificial Intelligence.
Source note: This article includes information reported by Startup Fortune.
Live Poll
Do you trust that current AI advancements reflect genuine human-level intelligence?
