INTELLECT-1: The First 10B Parameter Model Trained Globally with PRIME Framework

INTELLECT-1 Release: A Milestone in Distributed AI Training

Today, Prime Intellect is proud to release INTELLECT-1, the first 10 billion parameter language model trained collaboratively across five countries and three continents. This achievement represents a significant scale-up from our previous research, demonstrating that large-scale model training can be effectively decentralized and community-driven.

Key Releases:

• Detailed Technical Report: PDF
• INTELLECT-1 Base Model, Intermediate Checkpoints, and Post-Trained Model: Hugging Face
  • Chat Interface: chat.primeintellect.ai
• Pre-training Dataset: Hugging Face Collection
• Post-training Datasets by Arcee AI: EvolKit-75K
• PRIME Framework: GitHub Repository

Scaling Globally-Distributed Training

INTELLECT-1 marks the first large-scale experiment in globally distributed training. The model was trained on up to 112 H100 GPUs, achieving an overall compute utilization of 83% across continents and 96% when exclusively using nodes within the United States. This minimal overhead compared to centralized training approaches is a significant achievement.

• Compute Utilization:
  • 83% across continents
  • 96% within the U.S.
• Training Convergence: Maintained despite severe bandwidth constraints and node volatility

These results open new possibilities for community-driven training of frontier foundation models, proving that large-scale AI can be democratized.

Technical Innovations in PRIME

The PRIME framework was pivotal in achieving this milestone. Key innovations include:

• ElasticDeviceMesh: Manages dynamic global process groups for fault-tolerant communication across the internet and local process groups for intra-node communication.
• Live Checkpoint Recovery: Enables seamless recovery from node failures without significant downtime.
• Custom Kernels: Optimized for efficient communication and computation.
• Hybrid DiLoCo-FSDP2 Implementation: Combines data parallelism and model parallelism to reduce communication overhead.

Using PRIME with DiLoCo and a custom int8 all-reduce, we achieved an overall 400x reduction in communication bandwidth compared to traditional data-parallel training settings while maintaining comparable performance at the 10B scale.

Training Details and Datasets

INTELLECT-1 is based on the Llama-3 architecture with the following specifications:

• Layers: 42
• Hidden Dimension: 4,096
• Attention Heads: 32
• Sequence Length: 8,192
• Vocabulary Size: 128,256

The model was trained on a carefully curated 1 trillion token dataset mix:

• FineWeb-Edu: 55%
• Stack v2: 20%
• FineWeb: 10%
• DCLM-baseline: 10%
• OpenWebMath: 5%

Training completed over 42 days using a WSD learning rate schedule.

What This Means for Practitioners

The release of INTELLECT-1 and the PRIME framework is a game-changer for the AI community. It demonstrates that:

• Decentralized Training is Viable: Large-scale models can be trained collaboratively, reducing the barrier to entry for researchers and organizations.
• High Performance with Limited Resources: Even with bandwidth constraints and node volatility, high compute utilization and training convergence can be achieved.
• Open Source Potential: The open-source nature of PR