The Hidden Cost Trap in Agent Routing: Why Smarter Model Switching Can Make Your Pipeline More Expensive

Most engineering teams building multi-step agent pipelines treat model routing as a straightforward optimisation lever. Route expensive reasoning steps to a frontier model, offload the rest to something cheaper, and the cost curve bends downward. The logic is intuitive. The arithmetic, in practice, frequently disagrees.

The problem is not with the routing logic itself. The problem is that cost models for agentic workloads are built around per-call pricing, while the actual cost driver in a long-running loop is context accumulation. Once you account for how prompt caching works, and what happens to cached state when a model boundary is crossed, the economics of mid-loop routing look materially different from what most teams expect.

Companion piece to our broader work on agent cost architecture. See Model Routing in Agentic Systems: Cost Architecture for routing strategy comparisons, parallel architectures, and real code benchmarks.

Why Prompt Caching Changes the Cost Baseline

Prompt caching, as implemented by the major inference providers, reduces the input token cost for repeated prefixes. When the same leading sequence of tokens appears across consecutive calls, the provider can reuse a cached KV representation rather than recomputing it. The discount is substantial, typically 75 to 90 percent off standard input token rates depending on the provider.

In a single-model agent loop, this works in your favour automatically. Each iteration carries forward the system prompt, tool definitions, and accumulated conversation history. The prefix grows, but the cached portion grows with it. By the third or fourth step of a ten-step loop, the majority of your input tokens are hitting cache, and your effective input cost per call is a fraction of the nominal rate.

This is not a minor efficiency gain. For loops with long system prompts and rich tool schemas, cached input tokens can represent sixty to eighty percent of total input volume per call. Ignoring this when modelling pipeline cost means your baseline is wrong before you even introduce routing.

The Cache Invalidation Problem at Model Boundaries

When you route mid-loop to a different model, the cache built up against the first model is abandoned. The new model has no cached prefix. Every token in the accumulated context, system prompt, tool definitions, prior turns, intermediate outputs, must be processed as uncached input at full price.

This is the counterintuitive inversion. The routing decision that looks like a cost reduction at the call level can trigger a full-context recomputation that costs more than several additional calls on the original model would have. The longer the context at the point of the switch, the worse the arithmetic gets.

The effect compounds if routing is bidirectional. A pipeline that alternates between a frontier model and a cheaper model on alternating steps is not splitting cost evenly between two price points. It is paying full uncached input rates on an ever-growing context, on every call, on both models. The caching benefit is never realised because the prefix never stabilises against either provider's cache.

Context Boundary Design as the Primary Cost Variable

The practical implication is that context architecture, specifically the decision about what state crosses a model boundary and when, matters more to total pipeline cost than the per-token rates of the models you select. A well-designed context boundary can preserve caching efficiency across a routing event. A poorly designed one destroys it regardless of how cheap the target model is.

Stable Prefix Preservation

One approach is to enforce a stable cached prefix on the receiving model before the routing event occurs. If the system prompt and tool schema for the cheaper model are pre-warmed in a prior call, the cache is already populated when the routed call arrives. This requires deliberate pipeline design, not just model selection, but it allows routing to deliver its intended cost reduction rather than cancelling it through cache miss costs.

State Summarisation at Handoff Points

An alternative is to summarise accumulated context before crossing a model boundary, rather than carrying the full conversation history forward. A compressed state representation passed to the cheaper model reduces the uncached input volume at handoff. The trade-off is information fidelity. Whether that trade-off is acceptable depends on what the receiving model step actually needs from prior context, which is a design question that must be answered before the routing architecture is fixed.

When Single-Model Pipelines Win on Cost

There are workload profiles where a single frontier model, run across all steps with caching fully engaged, is cheaper than any routing strategy. These tend to be pipelines with long, stable system prompts, high tool definition overhead, and moderate per-step output volumes. In these configurations, the caching discount on input tokens offsets much of the premium in the frontier model's per-token rate.

The break-even point shifts as context length grows. For short loops with minimal carry-over context, routing to a cheaper model on simple steps is likely to deliver the expected saving. For long loops where context accumulates substantially across steps, the cache economics increasingly favour staying on a single model and letting the prefix stabilise.

The practical test is to model both strategies with realistic token counts, not nominal rates. Take your actual system prompt length, your tool schema size, your average per-step output, and your loop depth. Apply the caching discount to the single-model scenario. Then model the routing scenario with cache invalidation at each boundary. In our experience, teams that run this calculation frequently find the routing scenario wins only in the first two to three steps of the loop, after which the single-model strategy is equal or cheaper.

What This Means for Agent Infrastructure Design

The reframe this analysis demands is to treat context as an infrastructure resource, not just a billing line. Decisions about what accumulates in context, how long it persists, and where model boundaries fall are cost architecture decisions with the same materiality as instance sizing or batching strategy in traditional ML infrastructure.

Routing logic should be designed around context lifecycle, not just task classification. Before introducing a model boundary, the relevant questions are: what is the cached prefix value at this point in the loop, what is the uncached input cost on the receiving model, and does the per-call rate difference between models exceed that penalty. If the answer to the last question is no, the routing event is a cost increase dressed as a cost reduction.

Teams that have already built routing infrastructure are not necessarily in the wrong place. The routing layer remains valuable for latency management, capability matching, and risk tiering. The adjustment is to add context cost accounting to the routing decision function, so that the system routes on economics that reflect actual pipeline behaviour rather than nominal model pricing.