LiveCodeBench: A Holistic and Contamination-Free Evaluation Framework for Code-Generating LLMs

Large Language Models (LLMs) have become increasingly adept at handling code-related tasks, sparking significant interest from both academia and industry. However, as these models continue to evolve, existing evaluation benchmarks like HumanEval and MBPP are falling short in providing a comprehensive assessment of their capabilities. Enter LiveCodeBench-a new framework that aims to address this gap by offering a contamination-free and holistic evaluation of LLMs for code.

What Changed Technically?

LiveCodeBench introduces several key innovations:

• Continuous Problem Collection: Unlike static benchmarks, LiveCodeBench dynamically collects new coding problems from popular competition platforms like LeetCode, AtCoder, and CodeForces. This ensures the benchmark remains up-to-date and relevant.
• Broader Capability Assessment: The framework evaluates LLMs on a wider range of tasks beyond just code generation. It includes self-repair (fixing bugs), code execution, and test output prediction.
• Contamination-Free Evaluation: By using recently published problems (May 2023 to May 2024), LiveCodeBench minimizes the risk of model contamination, where models might have been trained or fine-tuned on similar data.

Why It Matters

For practitioners and researchers, LiveCodeBench offers a more robust and fair way to assess LLMs. Here’s why:

• Real-World Relevance: The dynamic collection of problems ensures that the tasks are representative of current coding challenges.
• Comprehensive Assessment: Evaluating models on multiple dimensions (code generation, self-repair, execution, and test output) provides a more holistic view of their capabilities.
• Fair Comparison: By using recent and unseen problems, LiveCodeBench reduces the risk of overfitting and ensures that performance metrics are reliable.

Key Details

• Problem Set: LiveCodeBench currently hosts 400 high-quality coding problems collected from May 2023 to May 2024.
• Model Evaluation: The framework has been used to evaluate 18 base LLMs and 34 instruction-tuned LLMs.
• Empirical Findings:
  • Contamination Analysis: LiveCodeBench provides insights into the contamination levels of existing benchmarks, highlighting potential overfitting issues.
  • Performance Comparisons: Detailed comparisons of model performance across different tasks and problem sets are presented.
  • Model-Specific Insights: The framework offers individual model evaluations, helping to identify strengths and weaknesses.

Implementation Notes

• Data Collection: LiveCodeBench uses web scraping techniques to gather problems from the specified platforms. This process is automated to ensure continuous updates.
• Evaluation Metrics: The framework employs a combination of accuracy metrics (e.g., pass rate for test cases) and qualitative assessments (e.g., code quality, readability).
• Toolkit for Extensibility: A general toolkit is provided to facilitate the addition of new scenarios and models. This makes it easier for the community to contribute and expand the benchmark.

Community Engagement

The authors are committed to fostering a collaborative environment. They will release all prompts and model completions for further analysis, along with the toolkit for adding new scenarios. This transparency and openness will enable the community to build upon LiveCodeBench, enhancing its utility and reliability.

In summary, LiveCodeBench represents a significant step forward in the evaluation of code-generating LLMs. By addressing the limitations of existing benchmarks and providing a more comprehensive assessment, it offers valuable insights for both researchers and practitioners.