Code Researcher: A Deep Learning Agent for Systems Code and Commit History Analysis

Large Language Model (LLM)-based coding agents have made significant strides in various coding benchmarks, but their performance on systems code has remained largely unexplored. Systems code, such as the Linux kernel, is notoriously complex and vast, making it a challenging domain for automated tools. A team of researchers from Microsoft Research has addressed this gap by introducing Code Researcher, a deep research agent designed specifically for large systems codebases and their extensive commit histories.

What Changed Technically

Code Researcher stands out because it combines multi-step reasoning with deep exploration of the codebase to generate patches for mitigating crashes. Here’s a breakdown of its key features:

• Multi-Step Reasoning: Code Researcher performs several steps of reasoning about the semantics, patterns, and commit history of the code. This allows it to gather a comprehensive context before making any changes.

  • Semantics Analysis: Understanding the meaning and structure of the code.
  • Pattern Recognition: Identifying common patterns and structures in the codebase.
  • Commit History Analysis: Leveraging historical data to understand past changes and their implications.

• Structured Memory: The gathered context is stored in a structured memory, which is then used to synthesize a patch. This ensures that all relevant information is available when generating the final solution.

• Deep Exploration: Unlike traditional agents that might only explore a few files, Code Researcher delves deep into the codebase. On average, it explores 10 files per trajectory, compared to just 1.33 files for the SWE-agent baseline.

Why It Matters

The ability to effectively analyze and modify systems code is crucial for maintaining and improving complex software projects. Here are some key points that highlight why Code Researcher is a significant advancement:

• Crash Resolution Rate: When evaluated on kBenchSyz, a benchmark of Linux kernel crashes, Code Researcher achieved a crash-resolution rate of 58%, significantly outperforming the SWE-agent (37.5%).
  • This improvement demonstrates its effectiveness in handling real-world issues.

• Generalizability: The researchers also tested Code Researcher on an open-source multimedia software to show its applicability beyond the Linux kernel. This indicates that the approach can be adapted to other large systems codebases.

Implementation Details

Code Researcher’s architecture is designed to handle the complexity of systems code:

• Data Collection: It starts by collecting data from the codebase and commit history.

  • Code Parsing: The agent parses the code to extract semantic information.
  • Commit History Analysis: It analyzes past commits to understand historical changes.

• Reasoning Engine: The core of Code Researcher is its reasoning engine, which performs multi-step reasoning:

  • Context Building: It builds a structured memory by combining semantic, pattern, and commit history data.
  • Patch Synthesis: Using the structured memory, it generates a patch that addresses the identified issues.

• Evaluation Metrics: The performance of Code Researcher is evaluated using several metrics:

  • Crash-Resolution Rate: The percentage of crashes successfully mitigated.
  • Files Explored: The number of files explored during each trajectory to understand the depth of exploration.

Conclusion

Code Researcher represents a significant step forward in the field of automated systems code analysis and patch generation. By combining multi-step reasoning with deep exploration, it effectively handles the complexities of large systems codebases. This research not only improves the efficiency of maintaining such projects but also opens up new possibilities for automated software development tools.