Snorkel AI Releases Aligned Mistral Model with New Benchmark Results

Snorkel AI has released a new aligned version of the Mistral language model, dubbed Snorkel-Mistral-PairRM-DPO. This model is designed to improve alignment and performance in chat-based applications. The release includes benchmark results that demonstrate the effectiveness of Snorkel's approach to LLM (Large Language Model) alignment.

Technical Changes and Why They Matter

The key technical changes introduced by Snorkel AI involve a novel training methodology that combines response reranking with Direct Preference Optimization (DPO). This approach aims to refine the model's responses to be more aligned with human preferences, making it particularly suitable for chat applications where nuanced and contextually appropriate responses are crucial.

• Training Dataset: The model is trained using prompts from the UltraFeedback dataset, which contains binarized feedback on LLM responses. Importantly, no external LLM responses were used during training.
• Response Generation and Reranking: For each prompt, five response variations are generated using the Mistral-7B-Instruct-v0.2 model. These responses are then reranked using PairRM, a technique that evaluates and ranks the quality of responses.
• Direct Preference Optimization (DPO): The top-ranked (chosen) and bottom-ranked (rejected) responses from the reranking step are used to update the LLM through DPO. This process is repeated for three iterations to refine the model further.

Implementation Details

The Snorkel-Mistral-PairRM-DPO model can be accessed via multiple endpoints, including Hugging Face and Together AI:

• Hugging Face Inference Endpoint:
  • API URL: https://t1q6ks6fusyg1qq7.us-east-1.aws.endpoints.huggingface.cloud
  • Initial Activation: The endpoint may take a few minutes to activate but will eventually operate at the standard speed of Hugging Face's 7B model text inference endpoint.
  • Usage Example:

    import requests
    
    API_URL = "https://t1q6ks6fusyg1qq7.us-east-1.aws.endpoints.huggingface.cloud"
    headers = {
        "Accept" : "application/json",
        "Content-Type": "application/json"
    }
    

def query(payload):
    response = requests.post(API_URL, headers=headers, json=payload)
    return response.json()

output = query({
    "inputs": "[INST] Recommend me some Hollywood movies [/INST]",
    "parameters": {}
})
```

• Together AI Playground and API:
  • Playground: You can try the model in the Together AI playground at https://api.together.xyz/playground/chat/snorkelai/Snorkel-Mistral-PairRM-DPO.
  • API: The model is also available through the Together AI API using the model string snorkelai/Snorkel-Mistral-PairRM-DPO.

Dataset and Training Recipe

• Dataset: The training dataset, Snorkel-Mistral-PairRM-DPO-Dataset, is derived from the UltraFeedback dataset. It contains prompts but no external LLM responses.
• Training Recipe: The data is formatted to be compatible with Hugging Face's Zephyr recipe. Each DPO iteration is executed using the "train/test_iteration_{n}" format.

Key Premises and Future Work

The key premise of Snorkel AI's approach is that specialization in alignment techniques can lead to better performance in specific applications like chat. The team plans to release more detailed results and findings on their blog in the coming weeks, providing deeper insights into the training process and benchmark results.

If you're interested in learning more about the model and its capabilities, check out the [Snorkel AI Blog](https://snorkel.ai/new-benchmark-results-demonstrate-value-of-snorkel-