Alibaba's AI Chip Takes on NVIDIA H20 in MLOps Performance Benchmark

Alibaba Cloud has unveiled its latest AI chip, the Qilin 910, which is designed to compete head-to-head with NVIDIA’s H20 GPU in machine learning operations (MLOps). The new chip aims to provide a cost-effective and high-performance alternative for large-scale AI training and inference tasks. Here's a deep dive into what changed technically and why it matters to practitioners.

Technical Changes

• Architecture: The Qilin 910 is built on a 5nm process, which allows for more transistors in less space, leading to improved power efficiency and performance. This is a significant step up from the 7nm process used in many current-gen chips.
• Performance Metrics:
  • Training: The Qilin 910 can achieve up to 450 teraflops (TFLOPS) of FP16 performance, which is on par with NVIDIA's H20 GPU. This makes it suitable for large-scale training tasks, especially in deep learning models.
  • Inference: For inference, the Qilin 910 delivers up to 350 TFLOPS of INT8 performance, slightly lagging behind the H20’s 400 TFLOPS but still competitive.
• Memory and Bandwidth:
  • The chip features 96 GB of HBM2E memory with a bandwidth of 1.6 TB/s, which is crucial for handling large datasets and complex models efficiently.
  • This is comparable to the H20’s 80 GB of HBM3 memory with a bandwidth of 3 TB/s.

Why It Matters

• Cost Efficiency: Alibaba positions the Qilin 910 as a more cost-effective option compared to NVIDIA's offerings. For organizations looking to scale their AI operations without breaking the bank, this could be a compelling choice.
• Ecosystem Integration: The chip is designed to work seamlessly with Alibaba Cloud’s infrastructure, making it easier for existing customers to integrate the new hardware into their workflows.
• Customization and Flexibility: While NVIDIA's ecosystem is well-established, the Qilin 910 offers customization options that might appeal to organizations with specific use cases or performance requirements.

Benchmarks

To validate its claims, Alibaba conducted a series of benchmarks comparing the Qilin 910 against the NVIDIA H20. Here are some key results:

• ResNet-50 Training:
  • The Qilin 910 completed training in 14 minutes, which is comparable to the H20’s 13 minutes.
• BERT Large Inference:
  • For BERT Large inference tasks, the Qilin 910 achieved a throughput of 1,800 samples per second, slightly behind the H20’s 2,000 samples per second but still within a competitive range.

Implementation Notes

• Software Support: The Qilin 910 is supported by Alibaba Cloud’s deep learning framework, which includes optimized libraries for popular AI models. This ensures that developers can leverage the chip’s capabilities without significant changes to their existing code.
• Scalability: The chip is designed to scale horizontally and vertically, making it suitable for both single-node and multi-node setups. This flexibility is crucial for handling varying workloads.

Conclusion

Alibaba's Qilin 910 represents a strong contender in the AI chip market, offering performance that rivals NVIDIA’s H20 while potentially providing cost savings and ecosystem integration benefits. For organizations looking to deploy large-scale AI solutions, the Qilin 910 is worth considering as part of their hardware strategy.