Researchy Questions: A New Dataset for Multi-Perspective, Decompositional QA Challenges

Introduction

The landscape of question answering (QA) has seen significant advancements with the rise of Large Language Models (LLMs). However, traditional benchmarks like TriviaQA and NaturalQuestions are becoming less challenging for these powerful models. This has led to a false sense of security in their capabilities. In response, researchers from various institutions have introduced "Researchy Questions," a dataset designed to address this gap by providing non-factoid, multi-perspective questions that simulate real-world complexity.

What Changed and Why It Matters

Traditional QA Benchmarks vs. Researchy Questions:

• Traditional Datasets: These benchmarks primarily focus on "known unknowns" where the missing information is clear and easily identifiable.

  • Examples: TriviaQA, NaturalQuestions, ELI5, HotpotQA
  • Challenges: They are becoming too easy for modern LLMs like GPT-4.

• Researchy Questions:

  • Focus: Non-factoid, decompositional, and multi-perspective questions.
  • Source: Filtered from search engine logs.
  • Complexity: Involves "unknown unknowns" where the information needs are unclear and require deeper cognitive processes.

Key Features of Researchy Questions

• Non-Factoid Queries:
  • These questions go beyond simple factual recall and often require synthesis of multiple sources or perspectives.
• Decompositional Nature:
  • Questions can be broken down into sub-questions, making them more manageable but also more complex to answer holistically.
• Multi-Perspective:
  • They consider various viewpoints, adding a layer of depth that is not present in traditional factoid questions.

Methodology and Findings

• Data Collection:

  • The dataset was compiled from search engine logs, which are typically rich with user intent but often dominated by factoid queries.
  • Researchers filtered out non-factoid questions through a tedious process, ensuring the final set of questions met their criteria for complexity and depth.

• User Effort Analysis:

  • Users spend significantly more time on these questions, as indicated by metrics like session length and number of clicks.
  • This suggests that Researchy Questions are indeed challenging and require more cognitive effort to resolve.

• Model Performance:

  • GPT-4, one of the most advanced LLMs, struggles with these questions, highlighting their difficulty.
  • "Slow thinking" techniques, such as breaking down the question into sub-questions, show better performance compared to direct answering methods.

Dataset Details

• Size: Approximately 100,000 Researchy Questions
• Additional Resources:
  • Clueweb22 URLs that users clicked while trying to answer these questions.
  • This provides a rich context for further research and model training.

Implications for Practitioners

For researchers and practitioners in the field of NLP and QA, this dataset offers several benefits:

• Benchmarking: A more rigorous testbed for evaluating the capabilities of LLMs.
• Model Improvement: Insights into how models can be improved to handle complex, multi-perspective questions.
• User Experience: Better understanding of user behavior and needs in real-world scenarios.

Conclusion

Researchy Questions is a significant step forward in creating more challenging and realistic benchmarks for QA systems. By focusing on non-factoid, decompositional, and multi-perspective questions, it pushes the boundaries of what LLMs can achieve and provides valuable data for future research and development.