HEADLINE: Grok 4 Surges to Top of AI Benchmarks, Leading Competitors in Reasoning and Performance

xAI has released Grok 4, the latest iteration of their reasoning-focused large language model (LLM), and it's making waves. After running a comprehensive suite of benchmarks, we can confirm that Grok 4 is now the leading AI model, surpassing competitors like OpenAI o3, Google Gemini 2.5 Pro, Anthropic Claude 4 Opus, and DeepSeek R1 0528.

Key Highlights:

• Artificial Analysis Intelligence Index: Grok 4 scores 73, outpacing OpenAI o3 (70), Google Gemini 2.5 Pro (70), Anthropic Claude 4 Opus (64), and DeepSeek R1 0528 (68).
• Coding and Math Indices: Leads in both the Coding Index (LiveCodeBench & SciCode) and Math Index (AIME24 & MATH-500).
• GPQA Diamond: Achieves an all-time high score of 88%, a significant leap from Gemini 2.5 Pro’s previous record of 84%.
• Humanity's Last Exam (HLE): Scores 24%, beating the previous high of 21% set by Gemini 2.5 Pro.

Technical Details and Architecture

Grok 4 is designed to "think" before providing answers, a feature that sets it apart from other LLMs. This reasoning capability is evident in its strong performance across various benchmarks:

• Context Window: Grok 4 supports a context window of 256k tokens, which is smaller than Gemini 2.5 Pro’s 1 million tokens but larger than Claude 4 Sonnet and Opus (200k tokens), o3 (200k tokens), and R1 0528.
• Speed: Generates output at a rate of 75 tokens per second, which is slower compared to o3 (188 tokens/s), Gemini 2.5 Pro (142 tokens/s), and Claude 4 Sonnet Thinking (85 tokens/s) but faster than Claude 4 Opus Thinking (66 tokens/s).
• Pricing: The cost remains the same as Grok 3 at $3/$15 per 1M input/output tokens ($0.75 per 1M cached input tokens). This pricing is on par with Claude 4 Sonnet but more expensive than Gemini 2.5 Pro ($1.25/$10 for <200K input tokens) and o3 ($2/$8 after a recent price decrease).

Benchmark Breakdown

• Artificial Analysis Intelligence Index: A comprehensive measure of AI performance, where Grok 4’s score of 73 is the highest recorded.
• Coding Index: High scores in LiveCodeBench and SciCode, indicating strong coding capabilities.
• Math Index: Top scores in AIME24 and MATH-500, showcasing robust mathematical reasoning.
• GPQA Diamond: An all-time high score of 88%, reflecting improved general knowledge and reasoning.
• Humanity's Last Exam (HLE): A significant improvement to 24%, demonstrating enhanced understanding of complex human concepts.

Availability and Deployment

Grok 4 is currently available via the xAI API. The version deployed on X/Twitter may differ slightly due to additional instructions and logic that can affect style and behavior. We also anticipate Grok 4 being available through Microsoft Azure AI Foundry, following the footsteps of Grok 3 and Grok 3 mini.

Conclusion

Grok 4 marks a significant milestone for xAI, placing them at the forefront of the AI landscape. Its strong performance in reasoning, coding, and mathematical tasks, combined with competitive pricing, makes it an attractive choice for developers and researchers alike. As we continue to explore its capabilities, it will be interesting to see how Grok 4 evolves and influences future advancements in AI.