Cappy: Enhancing Large Multi-Task Language Models with a Small Pre-Trained Scorer

March 14, 2024

By Yun Zhu and Lijuan Liu, Software Engineers, Google Research

Large language models (LLMs) have revolutionized natural language processing (NLP) by unifying various tasks within an instruction-following framework. Models like T0, FLAN, and OPT-IML have demonstrated impressive task-wise generalization capabilities, but their massive size-ranging from several billion to hundreds of billions of parameters-presents significant operational challenges. Enter Cappy, a small pre-trained scorer model that not only enhances the performance of these large multi-task LLMs but also outperforms them on complex tasks.

What Changed and Why It Matters

Cappy introduces a novel approach by leveraging a smaller, more efficient model to score and refine the outputs of larger LLMs. This method addresses the computational overhead associated with deploying massive models while maintaining or even improving performance. Here’s how it works:

• Pre-Trained Scorer: Cappy is a compact pre-trained model designed to evaluate and score the outputs generated by large multi-task LLMs.
• Enhanced Performance: By using Cappy, the overall system can achieve better results on a variety of complex tasks from benchmarks like PromptSource and Big-Bench.

Technical Details

Architecture

Cappy is significantly smaller than its counterparts:

• Size: Typically around 100 million parameters, compared to the several billion to hundreds of billions in large LLMs.
• Training Data: Utilizes the same multi-task data as the larger models but focuses on fine-tuning for scoring tasks.

Implementation

• Instruction-Response Pairs: Cappy is trained on the same instruction-response pairs used to train the large LLMs. Each pair consists of a task-specific template and a corresponding response.
• Scoring Mechanism: The scorer evaluates the quality of the generated responses by assigning scores based on how well they align with the intended output.

Benchmarks

Cappy was evaluated against several state-of-the-art multi-task LLMs:

• PromptSource: A benchmark that tests models on a wide range of NLP tasks.
• Big-Bench: Another comprehensive suite of tasks designed to challenge language models.

Key Findings

• Performance Boost: Cappy consistently outperformed the large LLMs it was paired with, achieving higher scores on complex tasks.
• Efficiency: The smaller size of Cappy makes it more computationally efficient, reducing deployment costs and latency.

How It Works in Practice

1. Generate Responses: The large LLM generates multiple response candidates for a given instruction.
2. Score Candidates: Cappy evaluates each candidate, assigning scores based on relevance and coherence.
3. Select Best Response: The system selects the highest-scoring response as the final output.

Why It Matters to Practitioners

• Resource Optimization: By using a smaller model to enhance performance, practitioners can reduce computational resources without sacrificing accuracy.
• Flexibility: Cappy’s modular design allows it to be easily integrated into existing systems, making it a versatile tool for improving multi-task language models.
• Scalability: The approach is scalable and can be applied to various LLMs, providing a consistent method for performance enhancement.

Conclusion

Cappy represents a significant step forward in the field of multi-task language modeling. By introducing a small pre-trained scorer, it addresses the computational challenges associated with large models while enhancing their performance. This innovation not only improves the efficiency of deploying LLMs but also opens up new possibilities for further research and application.