Model Routing Inside Agent Workflows: The Cost and Governance Decision Most Orchestration Designs Get Wrong

Enterprises deploying agent workflows at scale are making a structurally expensive assumption: that the model capable of handling the hardest step in a workflow is the right model for every step. That assumption compounds across millions of inference calls, and it creates a governance surface that most compliance teams have not yet mapped. The architecture that closes both gaps routes different models to different workflow stages based on task complexity and risk profile, and it requires deliberate design to do so without introducing new auditability problems.

Companion piece to our broader work on multi-model orchestration economics. See Model Routing vs Selection: AI Cost Architecture for how routing logic changes your underlying procurement and cost structure.

Why Single-Model Defaults Create a Compounding Cost Problem

Most orchestration frameworks make it easy to wire one model to every node in a workflow graph. The path of least resistance is to pick the best-performing model available, apply it everywhere, and optimise later. In practice, "optimise later" rarely arrives.

The cost penalty is not linear. A frontier model used for document classification, intent parsing, and final synthesis within the same workflow charges frontier prices for tasks that a smaller, cheaper model would handle at equivalent accuracy. When that workflow runs thousands of times per day, the delta between appropriate model selection and default selection becomes a material line item.

The deeper problem is that cost overruns from single-model defaults are invisible until someone builds the attribution layer. Without per-step cost tracking, the expense appears as a single aggregated inference bill, which makes it nearly impossible to identify where the waste is concentrated or to justify a routing redesign to finance.

The Governance Surface That Fragmented Deployments Miss

Single-model deployments feel simpler to govern because there is only one model to audit. That intuition is wrong in an agentic context. When one model handles planning, retrieval, reasoning, and output generation in sequence, the audit trail conflates decisions that carry very different risk profiles.

A planning step that decomposes a user request into sub-tasks is a lower-stakes inference than a step that generates a customer-facing response or triggers an external API call. Treating them identically in the audit log means that when a compliance review asks which model made which decision under what conditions, the answer requires forensic reconstruction rather than structured retrieval.

Regulated industries face this acutely. Financial services, healthcare, and legal workflows increasingly require that AI-assisted decisions be traceable to a specific model version, with documented reasoning for why that model was used at that stage. A single-model deployment does not provide that granularity by default, and retrofitting it after deployment is significantly more costly than designing for it upfront.

Routing Logic: Designing for Complexity and Risk, Not Just Cost

The correct routing heuristic is not simply "use the cheap model for easy tasks." It is a two-dimensional classification: task complexity on one axis, and decision risk on the other. Those two dimensions do not always correlate.

Low Complexity, Low Risk

Extraction tasks, format normalisation, and intent classification sit in this quadrant. Smaller, fine-tuned models or even deterministic classifiers are appropriate here. Routing a frontier model to these steps is the primary source of avoidable cost in most enterprise workflows.

Low Complexity, High Risk

This quadrant is where single-model defaults create the most governance exposure. A step that looks computationally simple, such as confirming a user's identity claim or selecting a regulatory template, carries high downstream consequence. The routing decision here is not about model size but about model provenance, version pinning, and output logging. A well-governed smaller model with strict output constraints is often preferable to a general-purpose frontier model with broader capability but less predictable behaviour.

High Complexity, High Risk

Final synthesis, multi-document reasoning, and steps that produce outputs requiring human review belong here. Frontier models are justified at this stage, and the audit requirements are highest. Every inference at this node should log the model version, the input context window, and the output before any downstream action is taken.

Building the Routing Layer Without Creating New Auditability Gaps

The practical failure mode in multi-model routing is that teams optimise the cost dimension while neglecting the audit dimension. Each model transition in a workflow is a potential gap in the evidence chain unless the orchestration layer is designed to capture handoff context explicitly.

The orchestration layer needs to record four things at each routing decision: which model was selected, why it was selected (the routing rule or classifier output), what input it received, and what output it produced. That record needs to be structured and queryable, not appended to a flat log file.

Version pinning at the routing layer is non-negotiable for regulated workflows. If the routing rule selects "the current best model in class X," and that model is updated silently by a provider, the audit trail breaks. Routing configurations should reference specific model versions, with a change management process governing when those references are updated.

We have written separately about how routing architecture intersects with agent task decomposition in practice. The structural principles in our Agent Task Routing: Architecture Mistakes and Solutions piece address the upstream decomposition decisions that determine which routing quadrant each step lands in.

Measuring Whether the Routing Design Is Working

A routing architecture that cannot be measured is not an architecture; it is a hypothesis. The measurement framework needs to operate at the step level, not the workflow level.

Per-step cost attribution is the baseline requirement. Each model invocation should carry a cost tag that rolls up to a workflow cost, enabling comparison between routing configurations and identifying which steps are driving spend disproportionately. Without this, routing optimisation is directional at best.

Quality measurement at the step level is harder but necessary. For steps with deterministic outputs, automated evaluation is straightforward. For generative steps, a sampling-based evaluation pipeline that scores outputs against a rubric provides sufficient signal to detect when a cheaper model is underperforming relative to the routing assumption. The measurement cadence should match the rate of workflow change, not a fixed quarterly review cycle.