Reaching New Heights on ARC-AGI with Multi-Agent Collaboration and Evolutionary Test-Time Compute

In a significant update to the ongoing AI research landscape, Jeremy Berman has once again claimed the top spot on the ARC-AGI benchmark. This time around, his approach not only outperforms previous records but also introduces a novel method that leverages multi-agent collaboration and evolutionary test-time compute (ETTC). The new system achieves a 79.6% score on ARC v1 at $8.42 per task-25 times more efficient than the previous best-and sets a new state-of-the-art (SoTA) of 29.4% on ARC v2.

What Changed?

Evolutionary Test-Time Compute (ETTC)

Berman’s latest achievement builds upon his earlier work with ETTC, which he used to win ARC-AGI v1 last December. However, this time he replaced Python functions with plain English instructions, a move that significantly enhances the system's flexibility and efficiency.

How It Works

• Grok-4 as the Central Agent: The core of Berman’s system is Grok-4, a large language model (LLM) that generates natural language instructions for solving each task.
• Subagent Testing: These instructions are tested against training examples by subagents. Each subagent evaluates the accuracy of the instructions and scores them accordingly.
• Evolutionary Cycles: The best-performing instructions are used to spawn new generations of refined solutions. This process is repeated through multiple evolutionary cycles, allowing the system to iteratively improve its performance.
• Dynamic Prompts: The system uses 36 dynamic prompts per task to generate up to 40 candidate instructions. These prompts help in exploring a wide range of potential solutions and ensure that the system can adapt to different types of tasks.

Key Benchmarks

• ARC v1 Score: 79.6% at $8.42 per task, which is 25 times more efficient than o3’s previous record of 75.7% at $200 per task.
• ARC v2 Score: 29.4%, setting a new SoTA and surpassing the previous best of 25%.

Technical Details

• Language Model: Grok-4, which is capable of generating high-quality natural language instructions.
• Subagents: These are smaller models or processes that test the generated instructions against training examples to evaluate their accuracy.
• Evolutionary Cycles: The system undergoes multiple cycles of instruction generation and refinement, each cycle improving the overall performance.
• Dynamic Prompts: 36 prompts per task to generate a diverse set of candidate solutions.

Why It Matters

The success of Berman’s approach highlights several important points:

• Generalization: Current LLMs struggle with tasks that require reasoning about things they weren't trained on. By using natural language instructions and evolutionary cycles, the system can better generalize to new and unseen tasks.
• Efficiency: The 25x improvement in cost efficiency over o3 demonstrates the potential for more scalable and practical AI solutions.
• Flexibility: Replacing Python functions with English instructions makes the system more adaptable and easier to modify.

Future Implications

Berman’s work suggests that combining multi-agent collaboration with evolutionary test-time compute can lead to significant advancements in AI reasoning. As LLMs continue to improve, this approach could pave the way for more robust and versatile AI systems capable of tackling a wide range of complex tasks.