AI Gateway Architecture: The Infrastructure Decision Your Platform Team Is Probably Getting Wrong

Most engineering teams arrive at their AI gateway architecture the same way: a developer needs to call an LLM, picks a library, and the pattern spreads. Six months later, authentication logic lives in three different services, rate limiting is handled inconsistently across teams, and nobody can produce a reliable breakdown of inference spend by customer or product line. The gateway decision was never made. It accumulated.

This article makes the case that AI gateway architecture is a foundational platform engineering problem, not a procurement question. The trade-offs differ significantly depending on whether your team is consuming external model providers or serving your own models to downstream customers, and conflating the two leads to designs that fail quietly in production.

The Access Gateway Versus Serving Gateway Distinction

These two gateway types solve different problems, and treating them as interchangeable is where most architectural drift begins.

An access gateway sits between your application layer and external model providers. Its primary function is to abstract provider-specific APIs, enforce authentication, and route requests across vendors based on cost, latency, or availability. The canonical use case is a platform team that wants to call OpenAI, Anthropic, and a self-hosted model through a single internal endpoint without rewriting application code each time a provider changes their interface.

A serving gateway sits between your inference infrastructure and your downstream consumers, whether those are internal product teams or paying customers. Its primary function is to enforce tenancy boundaries, attribute usage, and apply per-tenant policy. The distinction matters because the threat model is different. An access gateway protects your credentials and controls your spend. A serving gateway protects your customers from each other and protects your business from unbounded consumption.

Identity and Tenancy Models

The most consequential decision in serving gateway design is how identity maps to tenancy. Get this wrong and you will find yourself retrofitting customer isolation into a system that was built assuming a single principal.

The three common tenancy models are shared infrastructure with logical isolation, dedicated infrastructure per tenant, and a hybrid model where high-volume tenants get dedicated capacity while smaller tenants share a pool. Logical isolation is cheaper to operate but requires the gateway to enforce tenancy boundaries at the request level on every call. Dedicated infrastructure removes that enforcement burden but introduces provisioning complexity and makes efficient utilisation harder.

For most SaaS businesses serving inference to customers, logical isolation with strong per-tenant API key scoping and request tagging is the practical starting point. The gateway must attach a tenant identifier to every request at ingress, propagate it through the inference pipeline, and use it as the primary key for usage accounting. Any design that relies on application code to supply the tenant identifier is one misconfigured service away from attribution failure.

Rate Limiting Strategies

Rate limiting in AI inference is more complex than in conventional APIs because the cost of a request is not uniform. A single long-context completion can consume as much compute as hundreds of short ones, which means request-count limits are a poor proxy for capacity protection.

Token-Based Limiting

Token-based rate limiting measures consumption in input and output tokens rather than request counts. This requires the gateway to either enforce limits based on declared request parameters before inference begins, or apply post-hoc accounting and reject requests once a tenant approaches their allocation. Pre-inference enforcement is more protective but requires accurate token estimation, which is model-specific. Post-hoc accounting is simpler to implement but allows burst overruns that must be reconciled after the fact.

Concurrency and Queue Depth

For serving gateways operating against self-hosted models, concurrency limits on active inference slots are often more operationally meaningful than token budgets. The gateway should enforce a maximum number of concurrent requests per tenant and queue or reject excess requests with appropriate backpressure signals. Combining concurrency limits with token budgets gives you protection against both request floods and individual high-cost requests.

Usage Metering and Attribution

Usage metering is where access gateways and serving gateways converge on a shared requirement: every inference call must produce a durable, attributable record. The failure mode is not usually a missing record. It is a record that cannot be joined to a customer, a product line, or a billing period because the tagging schema was never agreed across teams.

The gateway is the right place to enforce tagging discipline because it is the only component that sees every request. Application code will drift. Different teams will use different field names for the same concept. The gateway should validate that required attribution fields are present at ingress and reject or quarantine requests that arrive without them. This is operationally uncomfortable at first, but it is considerably less uncomfortable than discovering six months of usage data that cannot be attributed.

Metering records should be emitted as structured events to a durable stream, not written synchronously to a database during the request path. Synchronous writes couple your inference latency to your metering infrastructure availability, which is an unnecessary dependency.

Degradation Handling and Fallback Design

Production inference infrastructure fails in ways that are qualitatively different from conventional API failures. Model endpoints can return valid HTTP responses while producing degraded output quality. Latency can increase gradually rather than producing hard timeouts. Providers can throttle requests in ways that are not always clearly signalled.

The gateway should implement explicit degradation policies rather than leaving fallback behaviour to individual application teams. For access gateways consuming multiple providers, this means defining a priority-ordered fallback chain per model capability tier, with circuit breaker logic that removes a provider from rotation when error rates or latency exceed defined thresholds. For serving gateways, it means deciding in advance whether to queue requests during capacity constraints, return a degraded response from a smaller model, or reject with a retriable error code.

The critical point is that these decisions must be made once at the gateway layer and enforced consistently. If each application team implements its own fallback logic, you will have no reliable way to observe system-wide degradation behaviour, and incident response becomes a process of reconstructing what each service decided to do independently.

Companion piece to our broader work on model routing and infrastructure abstraction. See Router Architecture Pattern for Enterprise AI Pipelines for how model selection logic fits into a coherent infrastructure layer and where vendor lock-in risk enters the picture.