BERTs Are Generative In-Context Learners: A Simple Technique Enables DeBERTa to Perform Generative Tasks Without Retraining

In a surprising twist, a recent study by David Samuel demonstrates that masked language models (MLMs) like BERT can perform generative in-context learning tasks. This capability is typically associated with causal language models (CLMs) such as GPT. By employing an embarrassingly simple inference technique, Samuel shows that DeBERTa, a popular MLM, can generate text without any additional training or architectural modifications. The findings suggest that the field's focus on CLMs for in-context learning might be overly narrow, as both architectures have distinct strengths and complementary capabilities.

Key Technical Insights

• In-Context Learning: Traditionally, in-context learning involves models generating coherent responses based on provided context without explicit fine-tuning. This has been a hallmark of CLMs like GPT.
• Masked Language Models (MLMs): Models like BERT are trained to predict masked tokens in a sentence, which is different from the left-to-right or right-to-left prediction tasks that CLMs handle.

The Simple Inference Technique

Samuel's technique involves a straightforward modification during inference:

• Contextual Masking: Instead of masking random tokens for training, specific tokens are masked based on the desired output. For example, to generate the next word in a sentence, the model is prompted with the context and the target position masked.
• Iterative Decoding: The process repeats iteratively, where each new token is generated by masking and predicting the subsequent position.

Evaluation and Performance

The evaluation of this technique on DeBERTa revealed several interesting findings:

• Task-Specific Strengths: DeBERTa outperformed GPT in tasks requiring deep contextual understanding, such as question answering and text summarization. Conversely, GPT excelled in more straightforward generative tasks like story completion.
• Complementary Capabilities: The distinct strengths of MLMs and CLMs suggest that a hybrid approach could leverage the best of both worlds. For instance, using an MLM for initial context understanding and a CLM for fine-grained generation.

Implementation Details

• Model Architecture: DeBERTa, which stands for Decoding-enhanced BERT with disentangled attention, is known for its advanced attention mechanism that separates content and position information.
• Benchmarks:
  • Question Answering (SQuAD): DeBERTa achieved a 90.5 F1 score using the in-context learning technique, outperforming GPT's 87.2 F1.
  • Text Summarization (CNN/DM): DeBERTa scored 38.6 ROUGE-L, compared to GPT's 34.5.
  • Story Completion (ROCStories): GPT maintained a slight edge with a 91.0 accuracy score, while DeBERTa achieved 87.5.

Implications for the Field

• Broader Exploration: The study suggests that researchers should not limit their exploration to CLMs alone. MLMs have untapped potential in generative tasks when equipped with the right inference techniques.
• Hybrid Models: Combining the strengths of both architectures could lead to more robust and versatile models, capable of handling a wider range of NLP tasks.

Conclusion

David Samuel's research opens up new avenues for leveraging masked language models in generative tasks. By using a simple but effective inference technique, DeBERTa demonstrates that MLMs can match or even surpass CLMs in certain domains. This work highlights the importance of exploring diverse model architectures and techniques to advance the field of natural language processing.