Field Notes

Measurement tools need evals too

I shipped a version of two small measurement tools whose only new feature was a willingness to say "I don't know". The self-test I added to prove they still worked caught three real bugs on its first day, and two of them had been hiding since v0.1.

One LLM judge is an opinion. Two model families agreeing is evidence.

I needed to know whether cloud rewrites matched my local model's voice, and I didn't trust a single LLM judge to tell me. So two judges from different model families scored the same work blind, and the interesting part isn't that they agreed. It's the bug the panel's own gate was hiding.

The migration the data ordered: moving a production LLM pipeline off the Mac Mini under my desk

A year ago I measured whether my local LLM pipeline was cheaper than the cloud, published the break-even, and kept running it locally anyway. This is the sequel, where the pipeline actually moves, the bill lands at five cents an article, and the surprises are all operational.

An MCP server that has to prove itself: citations, abstention and entitlements as CI gates

Hundreds of MCP servers will happily expose your notes to an agent. Almost none can prove the agent should believe what comes back. This is a knowledge-vault server whose trust properties are CI-gated numbers, and the first eval run found a real bug in under a minute.

The code that hides the evidence

An empty catch block does not handle a failure, it deletes the report of one. Error-masking constructs are measurable, they accumulate quietly, and almost nobody counts them. Here is how to count yours.

Measure your own coding habits before you believe anyone else's numbers

The industry says refactoring is collapsing and duplication is climbing. Those are global averages, and a global average has never told anyone what is happening in their repository. Here is how to check yours, and the five ways I nearly fooled myself doing it.

The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

Running your LLM pipeline locally to save money is mostly a myth: at real scale, batched cloud inference is pocket change. The engineering that matters is resumability, cost-aware routing, and a benchmark that measures your own local-versus-cloud crossover instead of inheriting a headline.

Stop shipping AI agents you can't measure: evals + observability from scratch

Everyone can build an AI agent. Almost nobody can prove it still works after the next prompt tweak or model bump. This is the smallest vendor-neutral way to make an agent's accuracy a CI gate, so a behavioural regression fails the build like a broken test.