One LLM judge is an opinion. Two model families agreeing is evidence.

The problem with one judge

I migrated my news pipeline's article rewriting from a local Qwen model to hosted Claude (previous piece), and before cutting over I needed an answer to a question taste can't settle at scale: does the cloud model hold each of my four writer personas as well as the local one did?

The standard answer is an LLM-as-judge: give a model a rubric, show it the article, get a score. The standard objection is also correct: a single judge is one model's opinion, with that model family's tastes baked in. And my case had a sharper version of the objection. The candidate prose was written by Claude. If Claude also judges it, every positive result carries an asterisk you can't remove: of course the judge liked its cousin's writing.

You don't fix that with a better rubric. You fix it with independence.

The panel

So the harness runs a blind, cross-family panel: Claude Opus and Gemini, each scoring the same article against the same persona rubric, five dials from 0 to 5, structured output, neither told which backend produced the text or what the other judge said. The gate takes the average of the two families and flags any article where they meaningfully split.

The blindness matters more than it sounds. The moment a judge prompt contains "this one is the new model", you're not measuring the article any more, you're measuring the judge's expectations. Strip every provenance clue before the text hits the panel.

What the panel buys you is a specific kind of defensibility. When one model says your output is good, you have a claim. When two models with different training lineages, different owners and different failure modes read the same text blind and land within a third of a point of each other, you have something a sceptic has to work much harder to dismiss.

What it found

On the full weekly set, 85 articles across all four personas, with both judges covering all 85, the verdict wasn't close: cloud 4.73 out of 5, local 2.19, and the cloud won on every persona individually:

Persona        Local   Cloud   Margin    n
The Analyst     2.17    4.70    +2.53    21
The Engineer    1.91    4.63    +2.72    27
The Optimist    2.30    4.70    +2.40     2
The Steward     2.41    4.83    +2.42    35
Overall         2.19    4.73    +2.54    85

The two families' scores differed by 0.30 out of 5 on average. Each judge run independently reaches the same verdict: Opus puts the margin at +2.45, Gemini at +2.64: they disagree slightly about how bad the old prose was, not about the answer. And the panel flagged 3 articles out of 85 where the judges genuinely split on a dial, every one of them Gemini scoring a persona trait harsher than Opus. That's not a weakness of the method; that's the method working. A panel that never disagrees isn't measuring independently.

That margin also killed a hedge I'd been keeping alive. I'd held "hosted Qwen-72B" open as an option, on the theory that the exact current voice was worth preserving. The panel data says the current voice was the weakest thing about the pipeline. Preserving it would have been sentimentality with a monthly invoice.

The bug the gate was hiding

Here's the part that earns this piece its place in a series about what survives production.

The panel's first real outing was a 12-article run. Both judges failed to start, thanks to a missing API key: the most boring failure there is. The panel logic did what half-thought-through code does: carried on, judged nothing, and the gate reported PASS, because zero judgments contained zero failures.

A gate that can't fail isn't a gate. It's a rubber stamp with extra steps.

The fix is the same principle I keep arriving at from different directions: fail closed on insufficient evidence. The gate now refuses to pass unless enough of the sample was actually judged, and a judge dropping out is a loud, visible event rather than a silent shrug.

Then reality tested the fix within hours. On the full 85-article run, Gemini's free tier (twenty requests a day) ran out after eleven articles. This time the system behaved: Opus coverage held at 85 out of 85, the run proceeded, and the degradation was visible instead of invisible. That became the production policy: if one judge fails, the panel proceeds on the survivor and flags a human; if both fail, the run fails. Degraded and honest beats complete and fictional.

If you build eval gates, test the gate itself. Feed it an empty result set and check it fails. Mine didn't, and I only found out because two different things broke on the same Saturday.

What it costs, honestly

The Opus judge came to $0.035 per article, about $3 for the full 85. And the quota story has a punchline: on the paid tier, Gemini judged all 85 articles for 22 cents. Twenty free requests a day was never an evals budget; the actual evals budget turned out to be less than a postage stamp. For continuous production use we still sample: mechanical checks (structure, length, banned phrases) run on every article for pennies, and the two-judge panel runs on a stratified dozen per week.

Caveats, because they're load-bearing: this compared new-model-plus-new-window against old-model-plus-old-limits, an era confound as much as a model one; one persona had only two samples in the set; and judge-versus-human calibration is still ahead of us (the panel agreeing with itself is necessary, not sufficient). The scores are directional evidence with unusually good provenance, not a leaderboard.

But the method stands on its own: blind the judges, cross the families, gate on the average, and above all, make sure the gate can actually say no.