An MCP server that has to prove itself: citations, abstention and entitlements as CI gates

The demo, in one screen

Here's the eval scorecard for a small MCP server on a green build:

retrieval   (staff, n=22):  hit@1 1.00   recall@5 1.00   MRR 1.00
abstention  (n=10):         rate  1.00   (target 1.00)
leakage     : 0 (must be 0)
redaction   : 0 (must be 0)
EVAL GATE: PASS

Now I weaken one thing. Two environment variables drop the server's relevance bar to zero, the setting where it returns its best guess for every question instead of admitting defeat:

abstention  (n=10):         rate  0.40   (target 1.00)
  FALSE-ANSWER  'customer refund policy' -> internal/hr/expenses-policy.md (score 4.29)
  FALSE-ANSWER  'office dog policy'      -> internal/hr/expenses-policy.md (score 4.29)
EVAL GATE: FAIL

Exit code 1. The build is red. Nobody had to notice the server had started making things up, because the pipeline noticed first.

Why bother?

The Model Context Protocol has made it absurdly easy to hand an agent your knowledge base. There are hundreds of servers that expose notes, wikis and docs as searchable tools, and most of them work fine as plumbing. But before you let an agent answer questions out of your company handbook, there are three questions the plumbing never answers.

Can you trust what comes back? When the server returns a snippet, can the agent cite where it came from, precisely enough that a human can check?

Does it know when it doesn't know? Ask a vault about a policy that isn't in it and BM25 will cheerfully return the closest-sounding note anyway. The agent, primed to trust its tools, turns that into a confident answer about a document that does not exist.

And who is allowed to see what? If your vault has a restricted/ folder, "the search tool filters those results out" is not an answer. It's the start of an incident report.

So I built grounded-mcp: a small Python MCP server over a folder of markdown (Obsidian works as-is) with three guardrails. Every content-bearing response carries a stable citation id, path plus heading plus line span. Search abstains, explicitly and structurally, when nothing clears the relevance bar. And entitlements are enforced at index level: content a profile can't see is never indexed for it in the first place.

None of which would be worth writing about, except for one decision: the repo ships with its own eval suite, and the suite is the CI gate. That decision paid for itself on the first run.

The first honest run found a real bug

The eval suite runs four families against a committed demo vault (a fictional company handbook with public, internal and restricted zones): a golden retrieval set, a set of questions whose answers are deliberately not in the vault, a leakage set answerable only from restricted content, and a redaction check against a seeded fake credential.

First run: retrieval perfect, leakage zero, redaction zero. Abstention: 0.30. Seven of ten unanswerable questions came back with confident wrong answers.

The interesting part is why. I had assumed a BM25 score threshold would separate real answers from plausible noise. The data said no:

true answers:    scores 2.44 to 14.68
false answers:   scores 2.12 to 4.29

The ranges overlap. "How long is probation" scores 2.44 against the note that genuinely answers it, while "customer refund policy" scores 4.29 against an expenses policy that says nothing about refunds. One common word landing in a heavily weighted title field looks exactly like relevance. No threshold exists that keeps the first and rejects the second.

What does separate them is coverage: what fraction of the query's content words the note actually contains. The false answers ride on one matching word ("policy") and their coverage collapses once you stop counting scaffolding like "what is the". So abstention became a double gate: score above threshold AND coverage above half. That took the rate from 0.30 to 1.00 with zero retrieval loss.

Then the gate caught two more bugs, in my fix. Naive substring matching decided "code" doesn't appear in a note that only says "coding", and a hyphenated "AI-generated" hid both of its parts from a query about AI. Light suffix-stripping and splitting compounds fixed both, and I know the fixes didn't break anything else because 24 golden queries and the leakage set re-ran on every attempt. The eval suite was the code review.

Leakage is a ranking problem, not a filter problem

The entitlements design deserves one more paragraph, because the obvious implementation is quietly wrong. Most access control on search works by filtering results: retrieve everything, then drop what the caller shouldn't see. That leaks. A filtered result still influenced rankings. Its terms still matched. In some systems the result count still moves, and now your search tool is an existence oracle for the documents you hid.

grounded-mcp builds a separate index per entitlements profile, containing only the notes that profile may see. Denied content is never scored, so there is nothing to filter and nothing to leak. The eval suite's leakage family checks this at the harshest setting, thresholds off entirely, and any hit from a denied path fails the build. Same principle for direct reads: a denied note and a nonexistent note return byte-identical responses.

What it doesn't do

Honesty section. Over stdio, client and server run as the same user, so profiles demonstrate the deployment pattern rather than defend against a hostile peer; real per-client enforcement arrives with the HTTP transport in v0.2. Retrieval is plain BM25, deliberately: it's deterministic, dependency-free and measurable, and when hybrid semantic retrieval lands it has to publish its eval delta against this baseline or it doesn't land. The redaction patterns catch key-shaped strings, not every secret. And there are no write tools. A server that can quote your vault but never rewrite it is a trust feature, not a gap in the roadmap.

Try it

The repo is github.com/rich-atkins/grounded-mcp, MIT licensed. pip install -e ., point GROUNDED_VAULT at your notes, wire it into Claude Code or Claude Desktop with a five-line config. Run the evals, then run the sabotage demo and watch the gate go red.

Better still: write ten questions your vault can't answer and add them to the abstention set. If your knowledge server passes that test today, you're doing better than mine was on Friday morning.