
Share
Gateways make model swaps a one-line config change in theory. In practice, VentureBeat's survey and interviews with engineers at Uber, PayPal, Box and T-Mobile show it's weeks of harness rework, re-tuning and evaluation before a new model earns production trust.
The pitch for AI gateways has always been simple: point your app at a different model string, and you're done. Reality is messier. VentureBeat's October survey of AI practitioners at companies with 100+ employees found that among the 47% whose companies had swapped a production model in the past year, 70% said the change took a few weeks or more of engineering work beyond flipping the model name. More than a third, 36%, said it took over a month. Only 13% called it close to a true configuration change.
The results weren't even clean wins. Of companies that made a swap, 55% said the new model matched or beat the old one. The other 45% either saw performance drop and had to rework the application, or rolled back to the original model entirely. That 45% hasn't moved in a statistically significant way since September, which tells you this isn't a problem teams are solving quickly.
Price is one obvious driver. Anthropic just dropped Claude Fable 5.2 at $8 per million input tokens and $40 per million output tokens, 20% cheaper than the version it replaces. That release landed after VentureBeat's survey closed, but it's exactly the kind of event that triggers a swap decision. Routers can shift traffic to cheaper models automatically. Adobe's marketing agent, for instance, uses a cheap model to pick context before handing the prompt to a larger one, said Anjul Bhambhri, Adobe's SVP of engineering, at VentureBeat's Agentic Infrastructure Stack Summit.
Some vendors are pushing further. Tetrate's "Auto Mode" lets customers stop specifying a model at all, according to CTO Varun Talwar. But he's candid that Tetrate isn't "fully there" on knowing which model fits which task. Customers roll Auto Mode out on one team first and only make it default once answers hold steady and costs drop.
Here's the part that trips people up: the gateway solves routing, not trust. A production app has prompts, tools and tests tuned to one model's quirks. Swap the model and those quirks shift underneath you.
Ben Kus, CTO of Box, put it bluntly: engineers tailor software to the model they started with, then "a new model just maybe acts differently." Executives assume newer means better. It often doesn't. Kus says the biggest performance swings come from the harness, the code that wraps a model, feeds it instructions, and controls which tools it can call, not from which frontier model you pick. Box now tests harness-and-model pairs together rather than evaluating models in isolation.
The numbers back this up. In Runta's FrontierHarness Eval benchmark, Moonshot AI's Kimi K3 passed 63.3% of tasks in both Claude Code and a DeepSeek Harness setup. Identical pass rate. But the median cost per passed task was $18.34 in Claude Code versus $3.28 in the DeepSeek configuration, a 5.6x gap for the same outcome. The test only covered 30 tasks across nine harnesses in 12 setups, so treat the ranking as directional rather than definitive. Still, the takeaway holds: test a model in the harness it'll actually run in before you blame the model for a harness problem.

Artificial Analysis CEO Micah Hill-Smith offered a useful nuance at the summit: for the largest frontier models, harness choice has "mattered less than we expected." Smaller models are a different story, swinging 20 to 40 points on coding benchmarks depending on whether they were trained on a particular harness's style. That's exactly the regime where companies are trying to cut costs, which means the engineering tax of a swap often lands hardest on the cheapest models.
Cost-per-run isn't the same as cost-per-good-result, either. Upwork's Sudhanshu Bist offered a clean example: a small model at 10 cents per run with a 20% success rate needs five retries on average, landing around 50 cents per good result. A larger model at 30 cents per run with 90% success costs about 33 cents per good result, cheaper despite a 3x higher per-run price. Bist also noted Upwork's gateway only tracks model-call spend. It doesn't see GPU costs for fine-tuning, human time spent fixing bad agent output, or compute for tool calls, so the "real" cost of a swap is bigger than the dashboard shows.
Fine-tuning adds another layer of lock-in. PayPal's Ren Zhang said the company fine-tuned a smaller model for its commerce agent to improve accuracy, but found that every new model release meant redoing that specialist work from scratch. PayPal has since pulled back on fine-tuning in favor of balancing out-of-the-box models with how it allocates compute. The fix some companies are landing on: keep business knowledge outside the model entirely. PayPal built a separate knowledge layer holding its business definitions, approved queries, and analysis playbooks, which it says lifted its internal data assistant's accuracy from 43% to 95% without touching the model. "It's model agnostic because I purposely did it that way," Zhang said. "When a better model comes out, I'll just plug in and play."
Not every team is chasing cheaper, either. T-Mobile CIO Jeff Simon said the company pays a premium for leading models in customer service because speed and accuracy matter more than margin there. Uber takes the opposite stance for its managed agents, agents the company builds and runs centrally. Distinguished engineer Uday Kiran Medisetty said Uber would switch to an open-weight model for AI code review "in a couple of days" if it offered better returns, since Uber already has a benchmark testing models against historical code changes. Benchmarking its automated code-cleanup job cut Uber's cost per cleanup by 90%. Sessions engineers run by hand in tools like Claude Code are much harder to move, Medisetty noted, because that requires changing how people actually work, not just swapping an API call.
Then there's governance, which has nothing to do with engineering difficulty and everything to do with organizational friction. JPMorgan Chase requires every new Claude model to clear its internal approval process before any employee can use it, Anthropic said this week. That's a separate clock running alongside the technical rework.
Snowflake's William Allen framed the real obstacle as psychological as much as technical: the sunk cost fallacy, where teams resist rebuilding because of time already invested. The companies managing swaps well share a pattern. Build an evaluation set from real requests, checked for accuracy, before you need it. Measure cost per good result, not cost per token, accounting for retries and failures. Test every model inside the harness it will actually run in, and try a second harness before writing off a model that underperformed. Databricks customers follow a similar playbook: build the first version on a frontier model, bank its answers as an eval set, then validate a cheaper model against that bar before switching. None of this makes a swap free. It just makes the weeks of work predictable instead of a surprise every time a new best model shows up.
Tags
Original Sources
Why swapping AI models takes weeks of work | VentureBeat
↗ https://venturebeat.com/infrastructure/swapping-ai-models-is-easier-said-than-done-for-most-companies-it-takes-weeks-vb-intelligence-finds
About the author
Kai built ML infrastructure at a Bay Area startup before developing an obsession with transformer architectures and inference optimisation that eventually pulled him out of product work entirely. A stint at a compute research lab sharpened his instinct for what actually matters in a model release versus what is marketing. He writes from the inside — from the perspective of someone who has debugged the systems he is describing at three in the morning. He is allergic to hype and instinctively drawn to the unglamorous plumbing questions that everyone else skips over.
More from The Engineer →This Week's Edition
13 November 2026
14 articles
Related Articles

Tether Evo's New Research Tackles BCI's Calibration Problem with Cross-Subject Models
Models & Research · 5 min

Why Some Engineers Think the Ocean Floor Is the Future of Data Centers
Environment & Climate · 5 min

Europe's Digital ID Push Tests a New Privacy Tool: Zero-Knowledge Proofs
Policy & Regulation · 5 min
Related Articles

Tether Evo's New Research Tackles BCI's Calibration Problem with Cross-Subject Models
Models & Research · 5 min

Why Some Engineers Think the Ocean Floor Is the Future of Data Centers
Environment & Climate · 5 min

Europe's Digital ID Push Tests a New Privacy Tool: Zero-Knowledge Proofs
Policy & Regulation · 5 min
More Stories
© 2026 Cedar & Bloom. All rights reserved.