The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

The gif, and the honest headline

Here's a multi-stage LLM pipeline being killed mid-run and started again:

It picks up exactly where it stopped. No duplicate model calls, no re-spend. That's the engineering payoff. The headline I want to be honest about up front is the economics one, because it runs against the usual pitch:

*This is not "run it locally and save a fortune." At a real content pipeline's scale,
batched cloud inference is genuinely cheap. The valuable thing is a reproducible way to
measure the local-vs-cloud crossover, and the answer of when local does and doesn't
pay off.*

Here's the real number, from an instrumented production run: one weekly run of this pipeline moves 641,671 input and 77,409 output tokens across nine local model stages, plus 134 locally-generated images. Run it on batched cloud models instead and that's about $0.88, under a dollar a week. So the interesting question was never "how much does local save." At this scale, cloud is already pocket change. It's when that stops being true, and how you would know.

The pains everyone hits on week two

The first version of any LLM pipeline is a for loop over a list. It works in the demo. Then production happens:

• It dies at item 800 of 1000 and you do not want to pay for the first 800 again.
• The same run, re-triggered, re-does work and double-writes downstream.
• Near-duplicate inputs get processed twice, because nothing deduplicates them.
• You have no idea what it cost, because nobody recorded the tokens.
• A rate limit or a credit ceiling hits and the run dies dirty, mid-item.

None of these is hard on its own. Together they're the difference between a script and a pipeline. `resumable-llm-pipeline` is the small, framework-light reference that wires them together, extracted (patterns only) from a real weekly content pipeline at Cedar & Bloom. It runs offline on the stdlib alone via a deterministic mock router, so you can see all of it in five minutes.

The build

Two pieces carry the weight.

`ProgressStore`, for resumability. A crash-safe {item_id -> {status, ...}} written with a temp-file-and-rename atomic swap after every item. Each stage filters to unfinished items and reads the previous stage's output. That decoupling is the resumability: kill the process anywhere and re-running recomputes nothing that already finished. The test suite proves it in numbers. Across a killed run and its resume, total model calls equal the number of kept items exactly. No item is processed twice.

`ModelRouter`, for cost-aware routing. complete(prompt, tier) and embed(text) behind one uniform response that carries token counts. Two tiers (cheap and quality) map to a backend (mock, local ollama, or hosted) by one config line. Because both local and hosted return (text, tokens_in, tokens_out), cost and metrics are backend-agnostic, and swapping local for hosted to re-run the benchmark is a one-liner. A Budget(max_tokens, max_cost) guard checks the paid stage as it runs; on breach it checkpoints and exits cleanly. Resume is the retry.

Routing between a cheap and an expensive model is not a new idea. RouteLLM (paper) showed a learned router can cut cost by around 85% on some benchmarks while holding quality. This router is the boring, explicit version: you assign each stage a tier by hand. The goal is not to be clever, it is to make the cost of that choice measurable.

The measurement: the tokens were being thrown away

Here's the detail that started this. The production pipeline it generalises calls local models via Ollama, and Ollama's response includes `prompt_eval_count` and `eval_count`, the exact input and output token counts, in the final chunk. The original code threw them away. It did real token work every week and recorded none of it.

So the router captures those counts on every call (and usage.input_tokens / usage.output_tokens on the hosted path), and a fail-open metrics module writes one JSON line per call: tokens, latency, backend, model. Fail-open matters. The thing that measures the pipeline must never be the thing that breaks it. rlp report rolls it up.

Here's the actual per-stage scorecard from one instrumented weekly run (qwen2.5:14b is the cheap tier, qwen2.5:72b the quality tier):

stage          model            calls   tok_in   tok_out
novelty        qwen2.5:14b        273   417647    12860
rewrite        qwen2.5:72b         67   112516    48673
image_prompt   qwen2.5:14b         67    52193    11094
enrich         qwen2.5:14b         67    43759     3289
cluster        qwen2.5:14b          4    10264      700
timelines      qwen2.5:14b         10     2568       60
companies      qwen2.5:72b         20     1445       661
edition_intro  qwen2.5:14b          1     1279        72
image_gen      FLUX.1-schnell     134        0        0   (images — timing only)
--------------------------------------------------------------
TOTAL                                    641671    77409

Until this run, every one of those token counts was None. The pipeline did the work weekly and recorded nothing. Notice novelty on its own: 273 calls, 417k input tokens, 65% of the entire run's input. Hold that thought, it matters for the cost.

The benchmark: measure the crossover, don't guess it

rlp benchmark reads the tokens the run actually recorded, maps each tier to its hosted equivalent, projects to an annual workload, and compares hosted cost (list price, and with the Batch API's 50% discount) against a fixed local cost. Fed the real weekly run, with the cheap tier priced as claude-haiku-4-5 ($1/$5 per 1M) and the quality tier as claude-sonnet-5 ($3/$15 per 1M):

Measured (weekly run): 641,671 input + 77,409 output tokens across 9 stages + 134 images.
Modeled:  hosted prices as above, Batch 50% off, hosted image ~$0.003/img, local $560/yr
          amortised (hardware capex + power), 52 weekly runs/yr.

| line                         | per run | per year (x52) |
|------------------------------|---------|----------------|
| hosted text (list)           | $1.75   | $91            |
| hosted text (batch, 50% off) | $0.88   | $46            |
| hosted images (modeled)      | $0.40   | $21            |
| local (fixed, amortised)     | —       | $560           |

Break-even is about 640 weekly runs per year, roughly 12x the real cadence. Below
that, batched cloud wins on cash.

Add the monthly evergreen run (measured: 47,596 in / 33,920 out, so $0.47 a run at list, $0.24 batched, roughly $3 to $6 a year) and the whole Cedar & Bloom LLM workload lands around $50 to $70 a year on batched cloud (about $110 at list). Against a $560/yr amortised local box, cloud is 8 to 12 times cheaper on cash at the real volume.

The discipline that makes this trustworthy, and the whole reason the tool exists: token counts are measured; hosted prices, the per-image cloud rate, and the local yearly cost are modeled and labelled. "The hardware's a sunk cost" is not a fair comparison. An honest break-even needs the amortised capex line, which is why it's in there.

The counter-intuitive bit: the cost isn't where the tokens are

Look again at that weekly scorecard through a cost lens:

| stage                  | share of input tokens | share of hosted cost |
|------------------------|-----------------------|----------------------|
| novelty (cheap tier)   | 65%                   | ~28%                 |
| rewrite (quality tier) | 18%                   | ~61%                 |

novelty moves two-thirds of the tokens but stays cheap, because it runs on the cheap model and emits almost nothing. rewrite is the reverse: modest token volume, but its output tokens on the quality model at $15/1M dominate the bill. The money is in output tokens on the quality tier, not where the token volume is. Knuth's old warning that "premature optimization is the root of all evil" has a neat LLM corollary: don't guess which stage is expensive. The tokens pile up in novelty; the money leaves through rewrite. You only see that by measuring, and it's what tells you which stage to move to cloud first if you ever wanted a hybrid.

One genuinely measured cloud data point

Everything above prices local tokens as if they'd run on the cloud. But one stage in the wider Cedar & Bloom stack already is hosted: SteadyPath's quiz runs on Anthropic Haiku, and instrumenting it captured 4 real calls, 1,490 input and 1,861 output tokens, actual usage from the API, not a projection. Same metrics module, same JSONL, local and hosted side by side. That's the point of the backend-agnostic router. The moment you move a stage to the cloud, its real cost lands in the same ledger as the local estimate.

The honest conclusion the data supports: at Cedar & Bloom's real scale, batched cloud would cost tens of dollars a year, and a weekly run is under a dollar. Local's edge is control, privacy, and no rate limits, and it only wins on cash at roughly 10 times the volume, or once you bolt on a token-hungry agentic layer. The number that tells you which regime you're in is one you can now measure instead of argue about.

Take the pattern

git clone https://github.com/uxdw/resumable-llm-pipeline && cd resumable-llm-pipeline
python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
rlp reset && rlp run && rlp run       # watch the 2nd run resume with 0 model calls
rlp benchmark                          # your own local-vs-cloud crossover

Point the router at your models, feed the benchmark your real per-stage tokens and verified prices, and read your own break-even. Measure your crossover. Don't inherit someone else's headline.

Built and measured at Cedar & Bloom. Written by Richard Atkins.