HelpSteer2: A Compact, Open-Source Dataset for Training Top-Performing Reward Models

---

High-quality preference datasets are crucial for training reward models that can guide large language models (LLMs) to generate responses aligned with human preferences. As LLMs become more sophisticated and better aligned, the need for up-to-date, permissively licensed preference datasets has grown. Existing datasets like Open Assistant, HH-RLHF, and HelpSteer have limitations, especially when it comes to commercial usage and data efficiency.

To address these challenges, a team of researchers from NVIDIA and other institutions has introduced HelpSteer2, an open-source dataset (licensed under CC-BY-4.0) designed specifically for training top-performing reward models. This new dataset not only improves upon the quality of generated responses but also enhances attribute labeling, making it highly effective for aligning LLMs.

Key Features of HelpSteer2

• Compact Size: HelpSteer2 consists of only ten thousand response pairs, which is an order of magnitude smaller than existing preference datasets like HH-RLHF. This compact size makes it highly efficient for training reward models without sacrificing performance.
• High-Quality Data: The dataset includes high-quality responses and attribute labels, ensuring that the trained reward models can effectively guide LLMs to generate aligned and meaningful outputs.
• Permissive Licensing: HelpSteer2 is released under the CC-BY-4.0 license, allowing for broad use in both research and commercial applications without restrictive limitations.

Technical Details

The researchers used a powerful internal base model trained on HelpSteer2 to achieve state-of-the-art (SOTA) performance on Reward-Bench's primary dataset. Here are some key technical details:

• Performance: The reward models trained on HelpSteer2 achieved a score of 92.0% on Reward-Bench, outperforming both open and proprietary models as of June 12th, 2024.
• Training Efficiency: Despite its small size, HelpSteer2 is highly efficient for training reward models. The compact dataset reduces the computational resources required, making it accessible to a wider range of practitioners.
• Model Alignment Approach: The team also introduced SteerLM 2.0, an alignment approach that leverages the rich multi-attribute scores predicted by the reward models. This method enhances the ability of LLMs to generate responses that are not only high-quality but also aligned with multiple human preferences.

Implementation and Availability

HelpSteer2 is available for download from Hugging Face, and the accompanying code can be found on GitHub at NVIDIA/NeMo-Aligner. The dataset and code are designed to be user-friendly, making it easy for researchers and practitioners to integrate HelpSteer2 into their projects.

Why It Matters

For practitioners working in the field of LLM alignment, HelpSteer2 offers several key advantages:

• Efficiency: The compact size of the dataset reduces training time and resource requirements.
• Quality: High-quality data ensures that trained reward models can effectively guide LLMs to generate aligned responses.
• Flexibility: The permissive license allows for a wide range of applications, from academic research to commercial products.

Conclusion

HelpSteer2 represents a significant advancement in the field of LLM alignment. By providing a compact, high-quality dataset under a permissive license, it addresses key challenges faced by researchers and practitioners. With its state-of-the-art performance and efficient training capabilities, HelpSteer2 is poised to become a valuable resource for anyone working on reward models and LLM alignment.