Enabling Two Settings Tripled GPT-5.6 Sol’s ARC-AGI-3 Scores: A Deep Dive into AI Harness Optimization

When we first encountered GPT-5.6 Sol’s underwhelming scores on the ARC-AGI-3 benchmark, it was puzzling. This model has already solved complex mathematical problems like the cycle double cover conjecture and conquered games like Pokémon FireRed. Yet, on ARC-AGI-3-a series of 2D puzzle games-GPT-5.6 Sol managed only a 7.8% score, with GPT-5.5 struggling even more at 0.4%.

What changed? The key was in the harness design. By enabling two API settings-retained reasoning and compaction-the scores tripled to an impressive 38.3%, while also cutting output tokens by a factor of six. This breakthrough underscores the importance of how we measure AI performance, not just what models we use.

Harnessing the Power: Retained Reasoning and Compaction

ARC-AGI-3 is designed to test how well AI agents can learn and reason in unfamiliar environments without explicit instructions. The benchmark uses a generic harness to ensure that model shortcomings are more visible and comparisons are fair. However, this simplicity can also mask the potential of advanced models.

Retained Reasoning

Retained reasoning allows the model to maintain its thought process across multiple steps. In traditional setups, each API call is treated as an independent query, leading to a loss of context. By retaining reasoning, GPT-5.6 Sol could build on previous insights, much like a human would.

Compaction

Compaction reduces the amount of redundant information in the model's output. This optimization cuts down the number of tokens needed to describe actions, making the model more efficient and focused on relevant details.

Here’s a breakdown of how these settings affected performance:

• Initial Scores: GPT-5.6 Sol scored 13.3% on the ARC-AGI-3 public set using the official harness.
• Enhanced Harness: With retained reasoning and compaction, the score jumped to 38.3%, a significant improvement.

Implementation Details

To implement these settings, OpenAI made the following adjustments:

• Retained Reasoning: Modified the API calls to maintain state across interactions, allowing the model to build on previous actions.
• Compaction: Applied algorithms to compress output tokens, reducing redundancy and improving efficiency.

These changes were relatively simple but had a profound impact. The average human tester scored around 48%, based on official gameplay logs, providing a benchmark for comparison. GPT-5.6 Sol’s enhanced performance brings it closer to human-level efficiency in solving these puzzles.

Key Takeaways

1. Harness Design Matters: The way we measure AI performance can significantly influence results. A well-designed harness that leverages model strengths can unlock hidden potential.
2. Context and Efficiency: Retained reasoning and compaction are powerful tools for enhancing AI performance. By maintaining context and reducing redundancy, models can tackle complex tasks more effectively.
3. Benchmarking Challenges: Benchmarks like ARC-AGI-3 are crucial for understanding AI capabilities but must be designed to account for different model features and optimizations.

This case study from OpenAI highlights the importance of considering harness design in AI benchmarks. As we continue to develop and evaluate advanced models, optimizing how these models interact with their environments will be just as critical as the models themselves.