MLCommons Launches AlgoPerf: Training Algorithms Competition to Accelerate Neural Network Optimization

The MLCommons® Algorithms Working Group has announced the AlgoPerf: Training algorithms competition, a new benchmark designed to measure and encourage improvements in neural network training algorithms. While the existing MLPerf™ Training benchmarks have been successful in driving innovation in hardware and software systems, there's still a significant gap when it comes to optimizing the actual training algorithms themselves. This new competition aims to fill that gap by focusing on algorithmic enhancements like better optimizers and hyperparameter tuning protocols.

Why It Matters

Faster training times are crucial for building more capable machine learning (ML) models. However, achieving this requires improvements across the entire training pipeline, not just in hardware. By specifically targeting algorithmic innovations, AlgoPerf hopes to save time, reduce computational resources, and ultimately produce better, more accurate models.

How It Works

AlgoPerf: Training algorithms is a competitive benchmark that measures the speedup of neural network training due to algorithmic improvements. Unlike MLPerf Training, which focuses on system-level performance, AlgoPerf fixes the hardware and lower-level software environments. This ensures that all submissions are evaluated purely based on their algorithms.

• Fixed System: All participants use the same hardware and software stack.
• Multiple Workloads: Submissions must perform well across a variety of deep learning tasks to ensure broad applicability (see Table 1 below).
• Randomized Tasks: Some workloads include randomized elements to test the robustness of algorithms.

Key Benchmarks

The competition includes several key benchmarks across different domains:

| Task | Dataset | Model | |----------------------------|----------------|-------------| | Clickthrough rate prediction | Criteo 1TB | DLRMSmall | | MRI reconstruction | FastMRI | U-Net | | Image classification | ImageNet | ResNet-50, ViT | | Speech recognition | LibriSpeech | Conformer, DeepSpeech | | Molecular property prediction | OGBG | GNN | | Translation | WMT | Transformer |

Submission Rules

To ensure that the competition yields generally useful training algorithms:

• Automated Hyperparameter Tuning: Submissions must automate any workload-specific hyperparameter tuning.
• Two Tuning Rulesets:
  • External Tuning: Simulates tuning with a fixed amount of parallel resources.
  • Self-Tuning: Simulates tuning on a single machine.

Competition Timeline

The AlgoPerf: Training algorithms benchmark competition is now open and will run from November 28, 2023, to March 28, 2024. To enter, participants must follow the provided instructions and guidelines.

Why Participate?

• Innovative Solutions: The competition encourages the development of novel training algorithms that can be applied across various ML tasks.
• Community Impact: By improving the state-of-the-art in training algorithms, participants can contribute to more efficient and effective ML practices.
• Recognition: Winners will receive recognition from the MLCommons community, which can enhance their professional profile.

Conclusion

The AlgoPerf: Training algorithms competition is a significant step towards advancing the field of neural network optimization. By focusing on algorithmic improvements, it aims to make training faster, more efficient, and more broadly applicable. For researchers and practitioners looking to push the boundaries of ML, this competition offers an excellent opportunity to contribute to and benefit from cutting-edge advancements.