Verbalized Sampling: A New Method to Boost LLM Diversity and Mitigate Mode Collapse

The Problem: Alignment Causes Mode Collapse

You ask your favorite language model (LLM) for a joke about coffee. You ask again. You get the same joke, no matter which model you try. You ask for a story, and it always begins with "Once upon a time..." The brainstorming ideas feel generic, the outputs repetitive. This frustrating phenomenon is known as mode collapse, where LLMs produce homogeneous responses despite being capable of more diverse outputs.

The Cause: Human Typicality Bias

While previous studies have attributed mode collapse to algorithmic limitations, researchers from Northeastern University and Stanford University have identified a more fundamental cause: human typicality bias. Annotators, when providing preference data for model alignment, tend to favor familiar and conventional responses. This systematic bias trains models to be less diverse, leading to the same repetitive outputs across different LLMs.

The Solution: Verbalized Sampling (VS)

To address this issue, a team of researchers led by Jiayi Zhang, Simon Yu, Derek Chong, and Anthony Sicilia introduced Verbalized Sampling (VS). This training-free prompting method asks models to output an explicit probability distribution over responses. For example, instead of just generating a single joke, the model might generate five jokes along with their corresponding probabilities.

How Verbalized Sampling Works

• Prompting for Diversity: Instead of directly asking the model to generate a response (e.g., "Generate a joke"), VS prompts the model to provide multiple responses and their associated probabilities (e.g., "Generate 5 jokes with their corresponding probabilities").
• Mitigating Mode Collapse: By explicitly asking for a distribution of responses, VS encourages the model to explore a wider range of outputs, reducing the likelihood of settling into a single mode.
• Tunable Diversity: The method allows for adjustable diversity across various tasks, including creative writing, social simulation, open-ended QA, and synthetic data generation.

Results and Impact

• Diversity Boost: Verbalized Sampling increases diversity by 1.6 to 2.1 times in creative writing tasks.
• Human Evaluation: It improves human evaluation scores by 25.7%, indicating that the generated outputs are not only more diverse but also of higher quality.
• Pre-alignment Recovery: VS recovers 66.8% of the base model’s pre-alignment diversity, effectively mitigating the negative impact of alignment on output variety.
• Safety and Quality: Importantly, Verbalized Sampling achieves these improvements without compromising safety or overall response quality.

Implementation Details

• Prompt Design: The VS prompt structure is crucial. It typically includes a request for multiple responses and their probabilities, e.g., "Generate 5 jokes with their corresponding probabilities."
• Model Compatibility: VS can be applied to various LLMs, including those from different vendors and architectures.
• Benchmarking: The researchers conducted extensive benchmarking across diverse tasks, including creative writing, social simulation, and open-ended QA, demonstrating consistent improvements in diversity and quality.

Example: Verbalized Sampling in Action

Direct Prompting (Mode Collapse):

• "Generate a joke about coffee."
  • Output: "Why don't coffee and tea ever get into arguments? Because they like to keep things steeped."

Verbalized Sampling:

• "Generate 5 jokes about coffee with their corresponding probabilities."
  • Joke 1: "Why did the coffee file a police report? It got mugged." (Probability: 0.3)
  • Joke 2: "Why don't coffee and tea ever get into arguments? Because they like to keep things steeped." (Probability: 0.25)
  • Joke 3: "What did the coffee say to the donut? You look dough-nut!" (Probability: 0.2)
  • Joke 4: "Why did the coffee go to the doctor? It was feeling a bit ground down." (Probability: 0.15)
  • Joke 5: "What do you call a coffee that’s always making jokes? A latte laugh!" (Probability: 0.1)

Conclusion

Verbalized Sampling offers a practical and effective solution to the