Measurement tools need evals too

A tool that refuses to answer

Here is one of my tools declining to do the thing it exists to do.

$ git-habits scan --from-export thin.tsv

  selection
    window            2026-03-10 to 2026-03-12  (3d span, 3 active)
    commits           3   insufficient (n=3 commits, need >=5)
    changed lines     120
    lines/commit      insufficient (n=3 commits, need >=5)
    moved (reuse)     insufficient (n=120 changed lines, need >=200)
    legacy touch      insufficient (n=120 changed lines, need >=200)
    rework <=14d      insufficient (n=120 changed lines, need >=200)
    AI co-authored    insufficient (n=3 commits, need >=5)

  ! only 3 commits in range: rates are volatile, treat as indicative

Three commits went in. The tool prints the raw counts, then refuses to turn them into rates. That is not an error and it is not a zero. It is a stated refusal, with the reason and the shortfall sitting exactly where the number would have gone, so you can see why there is no figure rather than being handed a confident one built on nothing. Its companion behaves the same way when you ask it to compare against a window that turns out to be empty.

$ diff-habits compare --repo . --split 2020-01-01
comparison refused: the 'before' window has no added lines. Check
--author/--since/--until/--split; an empty window compared against a real
one produces fiction, not deltas.
$ echo $?
1

That refusal, and the discipline behind it, is the whole of what v0.2 adds to git-habits and diff-habits, two small command-line tools I keep in the open at github.com/rich-atkins. No new metric. They measure what they measured before: commit sizes, rework, moved lines, error-masking constructs, the shape of how a codebase gets written over time. What changed is the set of things they are willing to say out loud, plus a test suite that proves they can still catch what they claim to catch. If that pairing sounds familiar, it should. It is what a decent AI eval does.

The rule I keep arriving at

These two tools point at other people's code. That is their reason to exist: git-habits reads commit metadata and never opens a source file, so you can run it against an employer's repository without a conversation first. A number that comes out of a tool like that carries weight. You will see it pasted into a review, a decision, a slide. So a wrong number is worse than no number, and an unknown quietly rendered as a zero is the worst outcome of all, because it looks like a finding.

Evals for language models taught me the shape of the fix, from the other end of my work. You do two things. You fail closed on insufficient evidence, so a thin sample produces a refusal rather than a reading. And you prove the thing you built actually detects what it says it detects, rather than assuming it does because the code looks right. v0.2 is those two ideas pushed into a pair of measurement tools that had neither.

The floors are the first idea made concrete. Every rate now knows its denominator and the point below which it declines to be a number. In git-habits that is five commits for the per-commit rates and 200 changed lines for the churn-based ones. In diff-habits it is 500 added lines, set higher on purpose: masking events are rare, running at roughly 0.2 to 4 per thousand added lines in the samples I have, so a per-thousand rate computed over a couple of hundred lines is noise wearing a number's clothes. Below the floor you get the refusal. Above it you get the figure, still with the old volatility caveats attached. The raw counts stay on screen throughout, because you are owed the evidence, not just the verdict you were going to quote.

Proving the detector detects

The second idea is a test suite that builds real git repositories on every run and then tries to fool the tools. There are two of them. One is PLANTED: a clear regime change after a fixed date, where commits swell to roughly six times their earlier size, cadence rises and AI co-author trailers start appearing. The other is CONTROL: the same shape from start to finish, no shift at all. The planted repository must be flagged. The control must stay quiet.

Both directions are asserted, and that is the part you skip when you are proud of the thing. A detector that always fires is as broken as one that never does, and a demo only ever shows you the first kind of failure. So the control test is the one that earns its keep. It fails if the tool cries wolf.

The three bugs it caught on day one

I wrote the self-test to protect work I believed was already correct. It caught three bugs the first time it ran, and the way it caught them is the entire argument for writing it.

The first was a windowing question I had answered wrongly and confidently. When you split a history into before and after at an instant, which window does a commit landing exactly on the split second belong to? My review predicted the bug would be double-counting, the boundary commit falling into both sides and inflating the totals. The boundary test proved something subtler. Git's --until is inclusive while its --since is exclusive, so v0.1 had quietly put every boundary commit in the before window only, and in nobody's after window. Not doubled. Dropped from one side. A plausible story about the bug, refuted by a test before it could become a plausible story in an article.

The second bug was the fix for the first. My initial correction shifted a single window edge, which felt right and read right. The test refused it, because moving one edge dropped the boundary commit out of both windows instead of one. The commit had gone from appearing once in the wrong place to appearing nowhere at all. The corrected fix anchors both edges one second before the split, so a boundary commit now lands exactly once, in the after window, matching git-habits' convention so the two tools agree about the same commit. Plausible fix, test proves it wrong, corrected fix pinned in place. I would have shipped the first attempt.

The third bug was the most humbling, because it exposed a claim I had been making for a month. Both tools advertised support back to Python 3.10. Neither had ever parsed a real git date under it. Git emits a trailing Z on UTC timestamps, and datetime.fromisoformat only accepts that Z from Python 3.11 onward. The unit tests fed synthetic logs that never carried the suffix, so the gap sat latent in both repositories since v0.1. The moment the sabotage suite built actual repositories and ran the real pipeline against them, the 3.10 job on CI failed. The floor I had promised was never a floor. It had simply never been tested.

Where this came from, and where it fails

None of this is new to me, which is why it stung. v0.1 of git-habits once reported "0% AI co-authored" for a repository whose trailers had never been captured, an unknown printed as a zero, the exact mistake the floors now forbid. And I watched the same shape appear somewhere with far higher stakes: an LLM eval gate in another project that returned PASS on a run where zero articles were actually judged, because zero judgments contain zero failures. A gate that cannot fail is not a gate. A report that cannot say "not enough evidence" will, given time, lie to you. I have now met that lesson three times, in a metrics tool, in a masking detector and in a model eval, and I have stopped treating them as separate.

Two honesty notes I will not bury. The floors are defaults with reasons behind them, not universal truths; five commits and 200 lines are low bars for "a number at all", and you should raise them for your own context rather than trust mine. And everything the tools could not tell you before, they still cannot. They do not detect AI, a missing trailer does not prove a human wrote the code, and a young repository cannot hold year-old code. The self-test proves the detector fires on a planted change. It does not turn a coarse signal into a fine one.

If you maintain anything that measures other people's work, whether a metrics script, a CI check or an eval harness, run one experiment this week. Feed it an input with nothing in it, and watch what it prints. If it hands you a clean zero and a passing exit code instead of refusing, you have not found a quiet week. You have found the day your tool learned to lie politely, and you get to fix it before it does so in front of someone who believes it.