Enterprise Architecture , AI Strategy , Data science & AI Oct 23, 2026

Token Costs Are an Architecture Problem: What Enterprise AI Teams Get Wrong About Inference Economics

VECTOR Labs Team

Last updated on: Oct 23, 2026

When token spend starts accelerating past forecast, the instinct is to call procurement and renegotiate the API contract. That instinct is wrong. The rate card is rarely the problem. What drives runaway inference costs at scale is almost always a set of architectural decisions made early in the build: how context is assembled, how tasks are dispatched across models, and how the harness around the model shapes consumption. Teams that treat this as a pricing problem will keep losing ground to teams that treat it as an engineering problem.

Companion piece to our broader work on AI cost governance. See Token Budget Governance Guide for practical strategies on controlling AI agent token spend before it compounds.

The Measurement Frame Is Broken

Most engineering leaders tracking AI spend are watching cost-per-token. The metric is precise and easy to pull from billing dashboards, which makes it feel like the right lever. But cost-per-token without a corresponding measure of value-per-token tells you nothing useful about whether your architecture is performing well or burning money on low-quality outputs.

An agentic workflow that completes a task in 40,000 tokens with high reliability is cheaper than one that completes the same task in 12,000 tokens but fails 30% of the time and retries. The retry loop is invisible in cost-per-token reporting. It shows up only when you measure cost-per-successful-outcome, which requires instrumenting the harness, not just the billing API.

Engineering leaders who want to make sound architecture decisions need to instrument both sides of that equation. The mechanism is straightforward: attach task identifiers to inference calls, log completion status, and aggregate cost against outcomes rather than against raw token counts. The commercial implication is that you stop optimising for cheap calls and start optimising for efficient resolution.

Context Layer Design Is the Highest-Leverage Variable

The single largest driver of token volume in production agentic systems is not model verbosity. It is context: how much is retrieved, how it is formatted, and how much of it is redundant across calls. A retrieval-augmented system that pulls 8,000 tokens of context per query because the chunking strategy is coarse will spend more on context than on generation, and most of that context will contribute nothing to the output.

Retrieval Precision

Improving retrieval precision reduces token volume directly. Smaller, semantically tighter chunks retrieved with better relevance scoring mean less irrelevant material enters the context window. The mechanism is that the model receives a higher signal-to-noise ratio, which also tends to improve output quality, so precision improvements compound: lower cost and better results from the same change.

Pre-computation and Caching

Pre-computing context for predictable query patterns and caching prompt prefixes that repeat across sessions can eliminate a substantial share of input tokens before the model ever sees a request. This is not a new idea, but it is consistently underused in enterprise deployments because teams build the retrieval layer first and treat caching as an optimisation to add later. Adding it later is harder and more expensive than designing for it from the start.

Model Routing Is Not Optional at Scale

Running every task through the most capable model available is the architectural equivalent of using a freight truck for every delivery regardless of package size. Frontier models carry frontier prices. Many production tasks, classification, extraction, summarisation of short documents, intent detection, do not require frontier capability to meet quality thresholds.

A routing layer that dispatches tasks to the appropriate model tier based on complexity, confidence requirements, and task type can reduce inference cost significantly without degrading output quality on the tasks that matter. The mechanism is straightforward: smaller models are faster and cheaper, and for well-scoped tasks they perform comparably to larger models. The commercial implication is that the capability headroom you are paying for on simple tasks is pure waste.

Routing requires investment: you need a classifier or heuristic to triage tasks, and you need quality benchmarks per task type to set confidence thresholds. That investment pays back quickly at scale. Teams that skip it because it adds complexity to the architecture are making a decision that becomes harder to reverse as volume grows.

Harness Design Shapes Consumption as Much as the Model Does

The harness is everything around the model: the prompt templates, the tool call logic, the retry strategy, the memory management, the output parsing. In most enterprise deployments, the harness is where the most avoidable token spend lives. Verbose system prompts that repeat instructions the model does not need, tool call schemas that are larger than the task warrants, retry logic that re-sends full context on failure rather than a corrected partial prompt: these are architectural choices, and they accumulate.

Harness design also determines how gracefully the system degrades when a model call fails or returns low-confidence output. A harness that escalates to a more capable model only when necessary, rather than by default, keeps the routing strategy intact under load. The implication is that harness quality is not a product concern or a prompt engineering concern in isolation: it is a systems engineering concern that belongs in the architecture review.

Continual Learning Trade-offs Belong in the Cost Model

Fine-tuning a smaller model on domain-specific tasks can reduce the routing burden on frontier models and lower per-task cost over time. The trade-off is the upfront investment in data curation, training infrastructure, and evaluation, plus the ongoing cost of keeping the fine-tuned model current as the task distribution shifts.

Teams that fine-tune once and do not account for distribution drift will find that model performance degrades quietly while costs stay low, which is a worse outcome than the high-cost baseline they were trying to escape. The mechanism is that a fine-tuned model trained on last quarter's data will encounter this quarter's edge cases without the generalisation capability of the frontier model it replaced. Continual learning is not a one-time decision: it is a recurring operational cost that belongs in the total cost of ownership calculation from the start.

The correct framing for engineering leaders is not whether to fine-tune, but at what task volume and quality threshold fine-tuning becomes economically justified given the ongoing maintenance burden. That calculation requires the same value-per-token instrumentation described earlier, applied over time rather than at a single point.

Where Vector Labs Fits

We design and build production AI architectures where inference economics are treated as a first-class engineering constraint, not an afterthought. In our token governance analysis, we set out the organisational and technical controls that prevent AI coding agent costs from compounding before teams have the instrumentation to see it happening. If your token spend is accelerating past forecast and you want an architectural assessment of where it is coming from, contact us at vector-labs.ai/contacts.

FAQs

We are already using a cheaper model tier for some tasks. Is that enough?

Tier selection alone is not a routing strategy. Without a systematic classifier or heuristic that dispatches tasks based on complexity and quality thresholds, you are likely either over-routing simple tasks to capable models or under-routing complex tasks to cheaper ones. Both patterns cost you: the first wastes money, the second wastes money on retries and corrections. A proper routing layer requires benchmarks per task type so you know where the quality cliff is for each model tier.

How do we instrument cost-per-successful-outcome without overhauling our observability stack?

You do not need a full observability overhaul to start. The minimum viable approach is to attach a task identifier and a binary success flag to each inference call at the harness level, then join that to your billing data in whatever analytics tool you already use. That gives you a cost-per-resolved-task metric without requiring new infrastructure. Refinements, like confidence scoring, retry attribution, and model-tier breakdowns, can be layered on incrementally once the baseline is in place.

At what scale does fine-tuning a smaller model become worth the investment?

There is no universal threshold, but the calculation has three inputs: the volume of calls on the target task, the cost differential between the fine-tuned model and the frontier model it would replace, and the ongoing cost of keeping the fine-tuned model current as your data distribution shifts. For high-volume, stable tasks with well-defined output formats, fine-tuning typically becomes economically justified earlier than teams expect. For tasks with high variance or frequent schema changes, the maintenance burden often erodes the savings.

Our context retrieval is already using RAG. Why are our input token counts still high?

RAG reduces context volume relative to stuffing entire documents into the prompt, but it does not automatically produce tight, high-precision retrieval. Coarse chunking strategies, low-relevance scoring thresholds, and fixed top-k retrieval regardless of query complexity are the most common causes of bloated input tokens in RAG deployments. Auditing your average retrieved token count per query against the proportion of retrieved content that actually influences the output is a useful diagnostic step before assuming the retrieval architecture is performing well.

Should token cost architecture decisions sit with the AI engineering team or the platform team?

In practice, the decisions that drive the most cost, context layer design, routing logic, harness structure, span both teams and are often owned clearly by neither. The most effective organisational pattern we have seen is to assign explicit ownership of inference economics to a named role or working group that has authority over both the AI layer and the platform instrumentation underneath it. Without that accountability, cost optimisation work gets deprioritised against feature delivery on both sides.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert