Linearizing LLMs for Efficiency and Quality: The LoLCATs Approach

In Part 1, we introduced LoLCATs, a novel method to linearize large language models (LLMs) while maintaining high quality and subquadratic efficiency. In this part, we dive deeper into the technical details and results of our approach.

What is Linearizing?

Linearizing LLMs involves converting existing pretrained Transformer models into more efficient architectures by replacing their self-attention mechanisms with linear attentions. This process allows us to achieve competitive performance without the need for extensive retraining on massive datasets, which can be computationally prohibitive.

The LoLCATs Challenge

When we embarked on this project, our primary goals were:

• Competitive Quality: Ensure that the linearized models perform as well as their original counterparts.
• Subquadratic Efficiency: Reduce computational complexity to make these models more accessible.
• Feasibility: Develop a method that can be implemented with limited compute resources.

Traditional approaches to training high-quality LLMs often require training 7B+ parameters on trillions of tokens. This is not feasible for most researchers, especially those without access to large GPU clusters (like the 64 A100s we were competing for).

Prior Work and Limitations

Before LoLCATs, several methods attempted to linearize LLMs:

• Hedgehog (Zhang et al., 2024): Linearized Llama 2 7B for summarization tasks.
• TRI's Work (Arora et al., 2024): Demonstrated general zero-shot capabilities by linearizing Llama 2 7B and Mistral 7B.
• Mamba Architectures (Chalamala et al., 2023, Singhal et al., 2024): Distilled 1.3B and 8B Transformer LLMs into more efficient models.

However, these methods still required extensive retraining after swapping out the attention mechanisms, which was both time-consuming and resource-intensive. The quality of the linearized models also varied, often falling short of their original counterparts.

LoLCATs: A New Approach

LoLCATs addresses these limitations by:

• Efficient Linearization: We use a combination of fine-tuning and layer-wise adjustments to minimize the need for full retraining.
• Quality Preservation: Our method ensures that the linearized models maintain high performance across various tasks.

Key Technical Details

• Layer Replacement:

  • Replace self-attention layers with linear attention mechanisms.
  • Use a gradual replacement strategy to avoid sudden drops in model quality.

• Fine-Tuning:

  • Fine-tune the model on a smaller dataset (e.g., 100M tokens) to adapt to the new architecture.
  • Use layer-wise learning rates to balance convergence and stability.

• Evaluation:

  • Benchmark performance on standard NLP tasks (e.g., GLUE, SQuAD).
  • Compare linearized models against their original counterparts in terms of quality and efficiency.

Results

Our experiments show that LoLCATs can achieve the following:

• Competitive Quality: Linearized LLMs perform comparably to their pretrained versions on various tasks.
• Subquadratic Efficiency: The computational complexity is significantly reduced, making these models more accessible for deployment.
• Feasibility: The method requires minimal compute resources, enabling researchers with limited budgets to develop high-quality LLMs.

Conclusion

LoLCATs represents a significant step forward in the linearization of LLMs. By combining efficient layer replacement and fine-tuning