Databricks Unveils DBRX: A New Open LLM with State-of-the-Art Performance and Efficiency

Databricks has just introduced DBRX, a new open-source large language model (LLM) that sets a new benchmark for performance and efficiency among existing open models. This is significant news for both the research community and enterprises looking to build or enhance their own AI capabilities.

What Changed Technically?

DBRX stands out in several key areas:

• Performance: DBRX outperforms established open-source LLMs like GPT-3.5 and is competitive with Gemini 1.0 Pro.
• Efficiency: It leverages a fine-grained mixture-of-experts (MoE) architecture, which makes it more efficient in both training and inference compared to dense models.
• Size: DBRX is about 40% smaller than Grok-1 in terms of total and active parameter counts.
• Inference Speed: It can generate text at up to 150 tokens per second per user when hosted on Databricks Model Serving.
• Training Efficiency: Training MoEs is approximately 2x more FLOP-efficient than training dense models for the same final model quality.

Key Features and Benchmarks

• Language Understanding (MMLU): DBRX outperforms other open-source models in language understanding tasks.
• Programming (HumanEval): It surpasses specialized code models like CodeLLaMA-70B, making it a strong choice for programming tasks.
• Mathematical Reasoning (GSM8K): DBRX excels in solving complex mathematical problems.

Architecture Details

DBRX’s success is largely attributed to its MoE architecture. Here are the key points:

• Mixture-of-Experts (MoE): This approach allows for more efficient use of parameters by activating only a subset of experts during inference, reducing computational overhead.
• Fine-Grained MoE: Unlike traditional MoE models, DBRX uses a finer-grained approach, which enhances both performance and efficiency.
• Inference Performance: The model is up to 2x faster than LLaMA2-70B in generating text, thanks to its optimized architecture.

Implementation Notes

• Training Stack: Databricks had to overcome significant scientific and performance challenges to develop a robust training pipeline for MoE models. This stack allows enterprises to train world-class MoE foundation models from scratch.
• Compute Efficiency: The overall recipe for DBRX, including pretraining data, model architecture, and optimization strategies, can achieve the same quality as previous-generation MPT models with nearly 4x less compute.

Availability and Use Cases

• Open Source: The weights of both the base model (DBRX Base) and the finetuned model (DBRX Instruct) are available on Hugging Face under an open license.
• Databricks Integration: DBRX is already integrated into Databricks’ GenAI-powered products, where it has surpassed GPT-3.5 Turbo in applications like SQL and is challenging GPT-4 Turbo.
• Customer Access: Databricks customers can use DBRX via APIs and train their own DBRX-class models from scratch or continue training on top of existing checkpoints using the same tools and science that Databricks used.

Conclusion

DBRX represents a significant step forward in the development of open-source LLMs. Its combination of state-of-the-art performance, efficiency, and versatility makes it a valuable asset for both researchers and enterprises looking to leverage advanced AI capabilities.