Serving Multiple Models to Multiple Tenants Without Losing Control of GPU Costs

Most engineering teams build inference infrastructure the way they build everything else under deadline pressure: one model at a time, one integration at a time, with routing logic that lives in application code and rate limiting that lives nowhere. This works fine at low scale. At the point where you are running five or six LLM-powered products on shared GPU infrastructure, it stops working quietly and starts failing expensively. The argument here is that treating the inference layer as a first-class control plane, with declarative policies and per-tenant enforcement, is not an optimisation you do later. It is the architectural decision that determines whether your GPU unit economics improve with scale or deteriorate with it.

Why Ad-Hoc Routing Breaks GPU Unit Economics

The core problem with bolt-on inference infrastructure is that GPU memory is not fungible in the way that CPU compute is. A GPU holding a loaded model checkpoint cannot simultaneously serve a different model without eviction cost. When routing decisions are made at the application layer without visibility into backend state, you get unnecessary model reloads, fragmented KV cache utilisation, and GPU memory that is nominally allocated but producing no throughput.

The commercial implication compounds quickly. Each unnecessary eviction is a latency spike for the tenant that triggered it and a throughput reduction for every other tenant sharing that node. When you have no per-tenant accounting, those costs are invisible until they appear as a GPU utilisation plateau that looks like a capacity problem but is actually a scheduling problem.

Teams that respond by adding more GPUs are not solving the underlying issue. They are increasing the surface area over which the same inefficiencies operate.

Building a Declarative Policy Layer for Tenant Isolation

The first structural change is separating policy definition from routing execution. Rather than encoding tenant constraints in application logic, you define them in a central policy store: maximum concurrent requests per tenant, token budget windows, priority tiers, and which model variants each tenant is permitted to access. The routing layer reads from this store and enforces constraints before a request reaches any GPU worker.

This separation matters because it makes policy auditable. When a tenant's bill spikes or their SLO degrades, you can inspect the policy state at the time of the incident rather than reconstructing it from application logs. It also means policy changes do not require application deployments, which is the operational difference between a control plane and a configuration file.

Priority tiers deserve particular attention. Not all tenants have the same latency requirements, and not all requests within a single tenant's traffic are equally time-sensitive. A declarative policy layer lets you express this hierarchy explicitly and enforce it under load, rather than discovering implicitly that your highest-value tenant's batch job is competing with their real-time product feature on the same queue.

SLO-Aware Routing and What It Actually Requires

SLO enforcement at the inference layer is not the same as SLO monitoring. Monitoring tells you when you have missed a target. Enforcement means the routing layer has enough state to make decisions that prevent misses before they occur.

Time-to-First-Token vs. End-to-End Latency

These are distinct SLO dimensions that require different routing signals. Time-to-first-token is dominated by queue depth and prefill compute. End-to-end latency is dominated by sequence length and decode throughput. A router that optimises only for queue depth will systematically misroute long-context requests from latency-sensitive tenants onto nodes that are decode-bound.

Effective SLO-aware routing requires per-request metadata at decision time: estimated input and output token counts, tenant priority, and current backend decode load. Without this, routing degrades to round-robin with extra steps.

Admission Control as a First-Class Mechanism

Admission control is the mechanism that prevents SLO degradation from propagating. When a backend node is operating near its decode throughput ceiling, the correct response is to reject or queue new requests at the control plane rather than admit them and degrade latency for requests already in flight. This is straightforward in principle and consistently skipped in practice because it requires the control plane to maintain a real-time model of backend state, which is engineering work that does not ship features.

KV Cache-Aware Load Balancing

KV cache reuse is one of the most significant levers available for reducing inference compute cost, and it is almost entirely wasted by stateless load balancers. When a request with a long system prompt or shared prefix is routed to a backend that has not previously processed that prefix, the full prefill computation runs again. At scale, across thousands of requests per minute sharing common prefixes, this is a measurable fraction of total GPU compute that produces no marginal output.

Cache-aware routing requires the control plane to maintain a registry of which backends hold which prefix hashes in their KV cache. Incoming requests are matched against this registry and preferentially routed to backends with a cache hit, subject to load constraints. The routing decision becomes a joint optimisation over cache hit probability and backend utilisation rather than utilisation alone.

The implementation complexity is real. Cache state is dynamic, prefix hashes must be computed efficiently, and the registry introduces a consistency challenge when backends evict cache entries under memory pressure. These are solvable engineering problems, but they require the control plane to be a stateful service, not a stateless proxy.

Governing Token Economics Across Heterogeneous GPU Pools

Heterogeneous GPU pools, mixing older and newer generations to manage procurement cost, introduce a further complication: the same model running on different hardware has different throughput and memory characteristics. A token budget that is sustainable on an H100 node may exhaust a A100 node's memory bandwidth under the same request pattern.

The control plane needs hardware-aware token accounting. This means normalising token budgets against the throughput profile of the backend serving the request, not against a single fleet-wide constant. Without this normalisation, tenants routed to lower-throughput hardware will consume their budgets faster in wall-clock time, which creates billing anomalies and SLO violations that are difficult to attribute correctly.

Per-tenant token ledgers, updated in real time and visible to both the routing layer and the tenants themselves, are the mechanism that makes token economics governable. When tenants can see their consumption against their budget, they can make informed decisions about prompt length and request frequency. When the routing layer can see the same data, it can enforce limits before they are breached rather than after.