March 17, 2024

Open Release of Grok-1

xAI has just announced the release of Grok-1, a massive 314 billion parameter Mixture-of-Experts (MoE) model. This open-source release includes both the weights and the architecture of the base model, providing researchers and developers with unprecedented access to one of the largest language models currently available.

What Changed Technically?

The core technical advancement here is the release of a fully trained 314 billion parameter MoE model. Unlike monolithic large language models (LLMs), Grok-1 employs a Mixture-of-Experts approach, where only a subset of the total parameters are activated for any given input token. Specifically:

• 314B Parameters: The model is significantly larger than many existing LLMs.
• 25% Activation Rate: Only 25% of the weights are active on a per-token basis, which helps in managing computational resources and maintaining efficiency.

This architecture allows Grok-1 to scale more effectively while keeping inference costs manageable. It's particularly useful for tasks that require deep contextual understanding without overwhelming resource consumption.

Why Does This Matter?

For practitioners, this release is significant for several reasons:

• Open Source: The model is released under the Apache 2.0 license, allowing anyone to use, modify, and distribute it.
• Custom Training Stack: xAI used a custom training stack built on top of JAX and Rust, which might offer insights into optimizing large-scale model training.
• Base Model Checkpoint: The release includes the raw base model checkpoint from the pre-training phase, which concluded in October 2023. This means it's not fine-tuned for specific tasks like dialogue, making it a versatile foundation for various applications.

Getting Started

To start using Grok-1, you can follow the detailed instructions provided on the GitHub repository. The repo includes setup guides, sample code, and other resources to help you integrate Grok-1 into your projects.

Key Model Details

• Training Data: The model was trained on a large corpus of text data.
• Architecture:
  • Mixture-of-Experts (MoE): Only 25% of the weights are active per token, enhancing efficiency and scalability.
  • Custom Stack: Built using JAX and Rust for optimized performance.
• License: Released under the Apache 2.0 license.

Try Grok On

If you want to explore what Grok-1 can do without setting up your own environment, xAI provides several options:

• Web: Grok.com
• iOS: App Store
• Android: Google Play
• 𝕏 (X Platform): Grok on X

Products and APIs

xAI offers a range of products and APIs to leverage Grok-1:

• Grok: The primary language model.
• 𝕏 (X Platform): Integration with the X platform for broader use cases.
• Grok Enterprise: For business users, offering more robust support and features.
• Grokipedia: A knowledge base powered by Grok-1.

APIs include:

• Overview: x.ai API
• Voice API: x.ai Voice API
• Imagine API: x.ai Imagine API
• Pricing: Detailed Pricing
• API Console Login: Console Access
• Documentation: [Developer