Understanding Behavioral Differences Between Base and Chat Models Through Model Diffing

When it comes to AI models, understanding the nuances between base models and their fine-tuned chat versions is crucial. A recent study by researchers at Anthropic delves into this topic, revealing some fascinating insights about how these models differ and what techniques can help us capture those differences more effectively.

Background

Base models are typically large language models (LLMs) trained on diverse datasets to generate coherent text. Chat models, on the other hand, are often fine-tuned versions of base models designed for interactive conversations. The goal is to make these chat models more contextually aware and user-friendly. However, identifying the specific changes that occur during this fine-tuning process can be challenging.

The Problem: Most "Model-Specific" Latents Aren't

One common approach to understanding model differences is through latent space analysis. Latent spaces are lower-dimensional representations of the model's internal state, which can help us identify key features and behaviors. However, researchers found that most latents identified as "model-specific" were not actually unique to a particular model.

• Key Finding: Most latents thought to be specific to either base or chat models were actually shared across both.
• Implication: This suggests that the differences between base and chat models are more subtle and require more sophisticated techniques to detect.

The Fix: BatchTopK Crosscoders

To address this issue, the researchers introduced a new technique called BatchTopK Crosscoders. This method involves:

• Batch Processing: Analyzing multiple inputs simultaneously to capture batch-specific patterns.
• Top-K Selection: Focusing on the top-k most relevant features in the latent space.
• Crosscoding: Mapping latents between different models to highlight unique characteristics.

Maybe We Don’t Need Crosscoders?

While BatchTopK Crosscoders are effective, the researchers also explored simpler techniques. They found that:

• Sparse Autoencoders (SAEs): These can capture behavioral differences without the need for crosscoding.
• Latent Scaling: Adjusting the scale of latents to better highlight model-specific features.

diff-SAEs Excel at Capturing Behavioral Differences

The study introduced a variant of SAEs called diff-SAEs, which are particularly good at capturing the subtle differences between base and chat models. Key points include:

• Behavioral Focus: diff-SAEs are trained to focus on behavioral aspects rather than just textual output.
• Performance Metrics: They outperform traditional SAEs in terms of identifying model-specific behaviors.
• Implementation Details: The architecture involves a combination of convolutional layers and attention mechanisms to effectively capture latent features.

Conclusion

Understanding the differences between base and chat models is essential for improving AI systems. Techniques like BatchTopK Crosscoders and diff-SAEs provide valuable tools for this task, helping us identify and leverage the unique characteristics of each model. As we continue to refine these methods, we can expect more precise and effective fine-tuning processes, leading to better-performing chat models.

Appendix

A) Other Techniques in the Model Diffing Toolkit

• Latent Space Clustering: Grouping similar latents to identify common patterns.
• Feature Importance Analysis: Determining which features are most influential in model behavior.

B) Closed Form Solution for Latent Scaling

• Mathematical Derivation: A closed-form solution for scaling latents can be derived using linear algebra techniques.
• Practical Application: This method helps in normalizing latent representations across different models, making it easier to compare and analyze them.