OpenScholar: Retrieval-Augmented LMs for Scientific Literature Synthesis

November 19, 2024

Akari Asai - Ai2

Share

---

Models Tech Report Code Demo

Update 2/4: Our paper has been accepted to Nature. Read more in our news release.

---

OpenScholar: A Step Forward in Scientific Literature Synthesis

Scientific progress depends on the ability to find, synthesize, and build upon relevant knowledge from a vast and rapidly growing body of literature. With millions of papers published annually, it's becoming increasingly challenging for researchers to stay updated or even locate crucial information within their subfields. To address this, we've developed OpenScholar, a fully open retrieval-augmented language model (LM) designed to help scientists navigate and synthesize scientific literature more effectively.

Key Features and Performance

OpenScholar leverages a two-step process: first, it retrieves relevant papers from a large corpus of academic literature; then, it generates responses grounded in those retrieved sources. This approach ensures that the generated answers are factually accurate and well-cited, unlike models that may hallucinate citations.

• Benchmark Performance: On ScholarQABench, our new benchmark for open-ended scientific questions, OpenScholar's 8B LM sets a new state-of-the-art in factuality and citation accuracy. For biomedical research questions, GPT-4o (a variant of GPT-4) hallucinated over 90% of the cited papers, whereas our 8B model remains grounded in real, retrieved papers.
• User Evaluation: We conducted a study with 20 scientists working in computer science, biomedicine, and physics. They evaluated OpenScholar's responses against expert-written answers. Across these disciplines, OpenScholar’s 8B LM was considered more useful for the majority of questions.

Technical Details

Architecture

• Retriever: The system uses a dense retriever to search for relevant papers in a large corpus of academic literature. This retriever is trained on a dataset of scientific papers and their embeddings.
• Re-ranker: After retrieving candidate papers, a re-ranker refines the selection based on relevance to the user's query.
• Language Model: The 8B LM generates responses using the retrieved papers as context. This ensures that the generated text is grounded in real sources.

Training and Data

• Training Data: The language model and retriever are trained on a diverse dataset of scientific papers, ensuring broad coverage across various disciplines.
• Retrieval Index: A large-scale retrieval index is built from this dataset, enabling efficient search during inference.
• Evaluation Data: ScholarQABench, our new benchmark, consists of open-ended scientific questions designed to test the model's ability to synthesize and cite relevant information accurately.

Open Source Release

To support further research in this area, we have open-sourced all components of OpenScholar:

• Code: Available on GitHub.
• Model Checkpoints: Including the 8B LM, retriever, and re-ranker.
• Retrieval Index: The large-scale index used for efficient paper retrieval.
• Training Data: The dataset of scientific papers and their embeddings.
• Evaluation Data: ScholarQABench, our new benchmark.

This is the first open release of a complete pipeline for a scientific assistant LM, from data to training recipes to model checkpoints. We're excited to see how the community builds upon this work.

Try It Out

Check out the demo at openschilm.allen.ai to see OpenScholar in action!