Model Routing in Production: Why Letting Engineers Pick the Model Is an Ops Problem in Disguise

Most enterprise AI teams treat model selection as a decision made once, upstream, by a product manager or procurement lead. The model gets chosen, the team builds around it, and the system ships. The problem surfaces six months later when the chosen model is deprecated, a cheaper alternative arrives, or a fine-tuned variant produces better results on half the workloads. By that point, the model is not a configuration parameter. It is load-bearing infrastructure, embedded in prompt templates, evaluation harnesses, tool schemas, and latency budgets in ways that nobody fully documented.

Companion piece to our broader work on model routing architecture. See Router Architecture Pattern for Enterprise AI Pipelines for how the router pattern abstracts vendor selection from application logic at the infrastructure level.

The Real Failure Surface Is Not the Model

When a frontier model update breaks a production feature, the immediate diagnosis is usually "the model changed." The actual failure is that the application had no separation between what it was asking the model to do and how it was asking a specific model to do it.

Prompt templates accumulate model-specific idioms over time. A team that spent weeks tuning system prompts for GPT-4o will have written instructions that exploit that model's particular instruction-following behaviour, its handling of JSON output constraints, and its response to chain-of-thought framing. Those assumptions are rarely documented. They are just baked into the prompt string.

The same leakage happens in evaluation pipelines. If your benchmark was calibrated against one model's output distribution, switching models does not just change the scores. It changes what the scores mean. This is why model independence is an engineering discipline, not a preference.

How Model Assumptions Leak Into Production Systems

Instruction Tuning Divergence

Different models are fine-tuned on different instruction formats and respond to different prompting conventions. A prompt that reliably produces structured JSON from one model may produce markdown-wrapped JSON, prose explanations, or schema violations from another. The application layer typically handles this with conditional parsing logic, which then becomes model-specific branching that accumulates silently.

The mechanism is straightforward: instruction-tuned models learn to follow the patterns present in their fine-tuning data. When that data differs across providers, the same semantic instruction produces structurally different outputs. The commercial implication is that every model-specific parsing fix is technical debt that makes future model migration more expensive.

Tool and Function Call Schema Drift

Tool-calling behaviour is particularly fragile across model versions. Argument naming conventions, handling of optional parameters, and tolerance for ambiguous tool descriptions vary significantly between providers and even between major versions of the same model. Teams that built agentic pipelines on one provider's function-calling API frequently discover that equivalent schemas on a different provider require structural rewrites, not just configuration changes.

This is not a documentation problem. It reflects genuine differences in how models were trained to interpret tool schemas. The implication is that tool definitions need to be treated as model-aware contracts, not portable specifications.

Evaluation Harness Coupling

Evaluation pipelines couple to models in subtler ways. Reference outputs used for LLM-as-judge scoring, embedding similarity thresholds, and output length normalisation are all calibrated against a specific model's behaviour. When the model changes, evaluation results shift in ways that are hard to interpret without re-baselining, which requires running the full evaluation suite against the new model before any production decision can be made.

Teams that have not built this re-baselining step into their deployment process treat model updates as risky because they are operationally unprepared to validate them quickly. The risk is real, but it is an ops gap, not an inherent property of model updates.

The Architecture That Makes Model Independence Durable

The Router Layer

A model router sits between application logic and model APIs. It accepts a standardised request, which includes a task classification, a quality-cost-latency preference, and a normalised prompt, and returns a response in a standardised format. The application never calls a model endpoint directly.

This indirection has a concrete operational benefit: when a new model becomes available or a pricing change makes a different model preferable for a given task class, the routing logic changes in one place. Application code, prompt templates, and evaluation pipelines remain unchanged because they interact with the router contract, not the model.

Prompt Normalisation and Model-Specific Adapters

The router layer should include a prompt normalisation step that translates a canonical prompt representation into a model-specific format. This is the architectural equivalent of a database abstraction layer. The application writes to a schema. The adapter handles the dialect.

In practice, this means maintaining a small adapter per supported model that handles instruction formatting, tool schema serialisation, and output parsing. The adapters are isolated, testable, and replaceable without touching application logic. The cost of maintaining them is real but bounded. The cost of not maintaining them is unbounded, because it accumulates invisibly across every feature that touches the model.

Evaluation Independence

Model-independent evaluation requires separating the evaluation criteria from the model that produces the outputs being evaluated. Reference-free evaluation metrics, task-specific rubrics defined in terms of output properties rather than similarity to a specific model's outputs, and evaluation datasets that were not generated by the model under test are the minimum requirements.

Teams that invest in this early find that model updates become validation exercises rather than risk events. The evaluation suite tells them quickly whether a new model meets the criteria for a given task class. That is the operational outcome that makes model independence commercially valuable.

The Hidden MLOps Cost of Model-Picker Interfaces

Many platforms now offer UI-level model selection, where engineers or even end users can choose which model handles a request. This looks like flexibility. In practice, it distributes the model-coupling problem across every engineer who makes a selection, without any of the abstraction infrastructure needed to make those choices reversible.

When model selection is a UI control rather than a routing policy, the system has no single place where model assumptions are managed. Prompt templates diverge per model. Evaluation coverage becomes uneven. Cost attribution becomes difficult because the model mix is determined by individual choices rather than policy.

The right framing is that model selection at the UI layer is a product feature. Model routing at the infrastructure layer is an ops requirement. Both can coexist, but the infrastructure layer must own the abstraction, and the UI layer must operate within it.

Building the Abstraction Layer Incrementally

Teams rarely have the runway to retrofit full model independence onto an existing production system in a single project. The practical approach is to introduce the abstraction layer incrementally, starting with the highest-volatility integration points.

The first priority is output parsing. Centralising all model output parsing into a single module, rather than handling it inline across feature code, isolates model-specific behaviour immediately and makes adapter maintenance tractable. The second priority is prompt storage. Moving prompt templates out of application code and into a versioned prompt registry with model-specific variants creates the foundation for systematic prompt management.

Evaluation independence comes third, because it requires the most investment and cannot be rushed without producing misleading results. But it is the capability that ultimately determines how quickly the organisation can respond to model changes in production, which is the operational metric that matters.