Optimizing AGENTS.md for Better Code Generation: Patterns and Pitfalls

If you’ve been working with coding agents, you know that the quality of your AGENTS.md files can make a significant difference. We recently conducted an in-depth study on how these files impact code generation, and the results were eye-opening. The best AGENTS.md files improved our agent's output as much as upgrading from Haiku to Opus, while the worst ones made things worse than having no file at all.

Key Findings

• Task-Specific Impact: A single AGENTS.md file can boost performance on one task by 25% and reduce it by 30% on another.
• Patterns Matter: Most content in these files either doesn't help or actively hurts. Effective patterns are specific and learnable.

The Same File, Different Tasks

A AGENTS.md file isn’t uniformly good or bad; its impact varies depending on the task. For example:

• Bug Fix: A decision table for choosing between two similar data-fetching approaches helped the agent pick the right pattern immediately and stay within codebase standards. This boosted best_practices by 25%.
• Complex Feature Task: The same file caused the agent to get pulled into a reference section, open dozens of other markdown files trying to verify its approach, create unnecessary abstractions, and ship an incomplete solution. This dropped completeness by 30%.

How We Measured

We used AuggieBench, one of our internal evaluation suites, to assess agent performance on typical day-to-day tasks from a large repository. Here’s the process:

• Baseline: Start with high-quality PRs that reflect common tasks.
• Environment Setup: Set up the environment and prompts for the agent.
• Task Execution: Ask the agent to perform the same task.
• Comparison: Compare the agent's output against the golden PR, which is the version that actually landed after review by multiple senior engineers.

Effective Patterns

To help you optimize your AGENTS.md files, here are some patterns that work and some that fail:

What Works

1. Decision Tables

   • Why: They provide clear, structured guidance on common decisions.
   • Example: A table outlining when to use different data-fetching methods.

2. Code Snippets

   • Why: Concrete examples help the agent understand expected patterns and styles.
   • Example: Sample code for handling edge cases in a specific module.

3. Task-Specific Guidelines

   • Why: Tailored advice for particular tasks ensures the agent stays on track.
   • Example: A section dedicated to best practices for API integration.

4. Reference Links with Context

   • Why: When necessary, provide links but include a brief context or summary.
   • Example: Link to a detailed design document with a one-liner explaining its relevance.

What Fails

1. Overly Detailed References

   • Why: The agent may get lost in the details and lose focus.
   • Example: A long list of links to various guidelines without clear guidance on which ones are most relevant.

2. Ambiguous Instructions

   • Why: Vague or overly broad instructions can lead to incorrect assumptions.
   • Example: Telling the agent to "follow best practices" without specifying what those practices are.

3. Repetitive Content

   • Why: Redundant information can confuse the agent and slow down its decision-making process.
   • Example: Repeating the same guidelines in multiple sections of the file.

4. Lack of Structure

   • Why: Unorganized content makes it difficult for the agent to find what it needs quickly.
   • Example: A long, unformatted text block without headings or clear sections.

How to Tell Which Patterns Work for Your Codebase

1. Experiment and Measure

   • Use your own evaluation suite (or a similar one) to test different patterns in your AGENTS.md files.
   • Compare the agent's output against golden PRs to see what works best.

2. Gather Feedback

   • Get input from senior engineers on which patterns are most effective.
   • Use their insights to refine and improve your