Tülu 3 405B Surpasses DeepSeek V3 with Enhanced Post-Training Recipes

January 30, 2025

Following the successful launch of Tülu 3 in November, we are excited to announce Tülu 3 405B-a significant milestone in applying fully open post-training recipes to the largest open-weight models. This new release not only scales our post-training techniques to a massive 405 billion parameters but also outperforms DeepSeek V3 and other leading models on various benchmarks, particularly in safety and mathematical problem-solving.

Key Technical Changes and Why They Matter

• Scaling Post-Training Recipes: Tülu 3 405B is the first application of our novel post-training techniques to a model of this scale. This demonstrates that our methods can handle large-scale models effectively.
• Enhanced Performance: The model achieves competitive or superior performance compared to DeepSeek V3 and GPT-4o, while surpassing other open-weight post-trained models like Llama 3.1 405B Instruct and Nous Hermes 3 405B on many standard benchmarks.
• Reinforcement Learning from Verifiable Rewards (RLVR): Our RLVR framework showed particularly significant improvements in MATH performance at the 405B scale, similar to findings in DeepSeek-R1.

Scaling the Tülu 3 Recipe

The primary goal of this release was to test and refine our RLVR approach and training infrastructure at a large scale. The training recipe for the 405B model closely follows the methods used for the 8B and 70B models:

• Careful Data Curation and Synthesis: We focused on data that targets core skills, ensuring the model is well-rounded.
• Supervised Finetuning (SFT): This involves training the model on a carefully selected mix of prompts and their completions to improve its performance on specific tasks.
• Direct Preference Optimization (DPO): Using both off-policy and on-policy preference data to refine the model’s outputs.
• Reinforcement Learning with Verifiable Rewards (RLVR): This novel method enhances specific skills by providing verifiable rewards, which is particularly effective for mathematical problem-solving and instruction following.
• Standardized Evaluation Suite: A comprehensive set of benchmarks for development, decontamination, and final evaluation to ensure the model’s robustness and reliability.

RLVR Training

Reinforcement Learning with Verifiable Rewards (RLVR) is a key component of our post-training recipe. This method trains language models on tasks with verifiable outcomes, such as solving mathematical problems or following instructions accurately.

To scale RLVR at 405B parameters, we deployed the model using vLLM with 16-way tensor parallelism. The remaining 240 GPUs were used for training. After each RLVR iteration, the weights are synchronized to the vLLM engine using NCCL broadcast, which ensures efficient and consistent updates.

Performance Benchmarks

• Mathematical Problem-Solving: Tülu 3 405B showed significant improvements in MATH performance compared to its smaller counterparts (70B and 8B), aligning with the findings from DeepSeek-R1.
• Safety Benchmarks: The model consistently outperforms DeepSeek V3, especially on safety benchmarks, which is crucial for real-world applications.

Conclusion

Tülu 3 405B represents a major step forward in scaling post-training techniques to large models. By leveraging our RLVR framework and other advanced training methods, we have achieved superior performance across various benchmarks, making this model a strong contender in the field of AI research and application.