Cerebras Achieves Trillion-Parameter Model Training on a Single CS-3 System at NeurIPS 2024

---

SUNNYVALE, CA AND VANCOUVER, December 10, 2024 – At NeurIPS 2024, Cerebras Systems, the leader in AI acceleration, announced a significant breakthrough: training a 1 trillion parameter model on a single CS-3 system. This achievement, in collaboration with Sandia National Laboratories, marks a major milestone in the development of large language models (LLMs), which typically require thousands of GPUs and extensive infrastructure.

What Changed Technically?

Traditionally, training a trillion-parameter model is an enormous undertaking:

• Hardware Requirements: Thousands of GPUs
• Infrastructure Complexity: Significant setup and configuration
• Expertise Needed: Teams of AI infrastructure experts

Cerebras' Wafer Scale Cluster technology has changed this paradigm by enabling the same task on a single CS-3 system. Here’s how they did it:

• MemoryX Technology: Cerebras introduced MemoryX, a terabyte-scale external memory device that stores model weights. This allows trillion-parameter models to be trained as easily as smaller models on a single GPU.
• Scalability: The model was initially trained on one CS-3 system and then seamlessly scaled up to 16 systems with no code changes.

Key Technical Details

• MemoryX Configuration: For Sandia’s trillion-parameter training run, Cerebras configured a 55 terabyte MemoryX device. This was achieved using commodity DDR5 memory in a 1U server format.
• Initial Training Steps: AI researchers ran initial training steps and observed improving loss and stable training dynamics on the single CS-3 system.
• Scaling Up: After the single-system run, the model was scaled to two and sixteen CS-3 nodes. The cluster exhibited near-linear scaling with a 15.3x speedup on sixteen systems.

Why It Matters

• Simplified Infrastructure: No need for thousands of GPUs or extensive hardware setup.
• Reduced Time and Cost: Procuring and configuring the hardware took mere days, compared to weeks or months for traditional setups.
• Energy Efficiency: The system uses significantly less power than thousands of GPUs.

Practical Implications

For practitioners, this means:

• Faster Prototyping: Researchers can quickly test and iterate on large models without waiting for extensive setup times.
• Cost Savings: Reduced hardware and energy costs make it feasible to train larger models with fewer resources.
• Improved Performance: The near-linear scalability ensures that performance scales efficiently as more systems are added.

Expert Insights

Siva Rajamanickam, a researcher at Sandia National Laboratories, commented on the achievement: "Traditionally, training a model of this scale would require thousands of GPUs, significant infrastructure complexity, and a team of AI infrastructure experts. With the Cerebras CS-3, the team was able to achieve this feat on a single system with no changes to model or infrastructure code."

Conclusion

Cerebras' achievement at NeurIPS 2024 demonstrates the power and flexibility of their Wafer Scale Cluster technology. By simplifying the training process for trillion-parameter models, Cerebras is setting a new standard in AI research and development.