New AI Index Highlights Widening Performance Gap
Artificial Analysis updates its benchmark to use private test sets, revealing a 12-point divide between proprietary and open models.
Updated on Oct. 28, 2026 in Artificial Intelligence
Live Poll
Do you believe open-source artificial intelligence models will eventually outperform proprietary versions?
Artificial Analysis has released version 5 of its Intelligence Index, shifting 60 percent of its benchmark to private test sets to prevent model training on evaluation data. The index tracks performance across coding, science, knowledge work, and general knowledge, revealing a widening gap between proprietary and open-weight models.
Why it matters
The transition to private, held-out test sets addresses concerns that AI models are becoming overfitted to public benchmarks. This change in methodology highlights a significant performance divergence as proprietary models maintain a lead over open-weight alternatives.
The index utilizes a median of 59,000 output tokens per task to evaluate performance. Claude Opus 5.5 leads the rankings with 58 points, compared to the highest-scoring open-weight model, Muse Spark Open, which achieved 25 points.
The players
Artificial Analysis
An independent benchmarking firm that evaluates large language model capabilities and performance metrics.
The details
The index evaluates capabilities across coding, science, knowledge work, and general knowledge. By utilizing private test sets—data that the models have not been exposed to during their training phase—the firm aims to ensure that scores reflect reasoning capabilities rather than rote memorization. Models are measured on their ability to generate accurate outputs over a high volume of tokens per request.
Timeline
October 28, 2026: Artificial Analysis released version 5 of its Intelligence Index.
The Tech Race
This release marks a significant departure from public benchmarking standards to counter data contamination. The widening 12-point gap between proprietary systems and open-weight models suggests that the largest compute-intensive models are continuing to pull away from their open-source competitors.
Developers and organizations using the index to select models should note that results now prioritize held-out data to better simulate real-world reasoning tasks. The cost discrepancy between top models, such as $10 per task for Claude Opus 5.5 versus $1.27 for GPT-6.1 Sol, remains a key factor for deployment.
The takeaway
The move to private test sets signals that performance benchmarks are moving toward more rigorous, adversarial testing to verify genuine intelligence gains. Industry observers should watch for how open-weight developers respond to this widening performance gap in upcoming model releases.
Further reading
For more on the current state of model benchmarking, see Artificial Intelligence.
Live Poll
Do you believe open-source artificial intelligence models will eventually outperform proprietary versions?
