AI Strategy , Software development , Company Jul 22, 2026

Why Netflix Runs Its Own LLM Stack and What That Decision Reveals About the Real Cost of Hosted APIs

VECTOR Labs Team

Last updated on: Jul 22, 2026

When Netflix announced it was building and operating its own large language model serving infrastructure, most commentary treated it as a curiosity specific to hyperscale engineering culture. That reading misses the point. What Netflix actually did was force a rigorous cost-benefit analysis that most enterprises skip entirely, and the conclusions that analysis produces are uncomfortable for organisations that have quietly made hosted APIs a structural dependency.

This article works through the technical and commercial logic behind that decision: what it costs to own the stack, what you give up by not owning it, and how to evaluate which answer is correct for your organisation before the default choice becomes irreversible.

The Hosted API Default and Why It Holds

Hosted APIs from providers like OpenAI, Anthropic, and Google are the rational starting point for almost every organisation. The time-to-first-token for a proof of concept is measured in hours, not weeks. There is no infrastructure to provision, no model to serve, and no serving framework to configure. For exploration and early production workloads, that simplicity is genuinely valuable.

The problem is that the decision rarely gets revisited. Once a product team has shipped a feature on a hosted API, the integration becomes load-bearing. Revisiting it requires engineering effort, a migration plan, and organisational will, none of which are easy to find when the product is already running.

The result is that many organisations are not choosing hosted APIs so much as drifting into a permanent dependency on them. That is a different thing, and it carries a different risk profile.

What Owning the Serving Stack Actually Involves

Running your own LLM serving infrastructure is not simply a matter of downloading model weights and pointing a server at them. It involves decisions at several distinct layers, each of which carries its own operational surface.

Inference Engine Selection

The inference engine determines how efficiently the model uses GPU memory and compute. vLLM's PagedAttention mechanism, for example, significantly reduces memory fragmentation during batched inference, which directly affects throughput under concurrent load. TensorRT-LLM offers tighter integration with NVIDIA hardware but introduces a compilation step that extends the deployment cycle. The choice between them is not academic; it determines your cost per token at production scale.

Model Packaging and Versioning

Serving a model in production requires a reproducible packaging format, a versioning strategy, and a rollback mechanism. Organisations that underinvest here discover the problem when they need to roll back a fine-tuned model after a quality regression. Without a disciplined model registry and promotion workflow, that rollback becomes a manual, error-prone process under pressure.

API Surface Design

The API your internal teams or products call is not the same thing as the model interface. You need to design for rate limiting, request prioritisation, streaming response handling, and observability hooks. These are standard platform engineering concerns, but they require deliberate design rather than implicit inheritance from a provider's existing surface.

The Cost Arithmetic That Changes at Scale

The economics of self-hosting versus hosted APIs follow a predictable crossover curve. At low request volumes, hosted APIs are almost always cheaper when you account for the full cost of GPU infrastructure, engineering time, and operational overhead. The crossover point depends on your workload characteristics, but for most organisations running sustained inference workloads, it sits somewhere between several million and tens of millions of tokens per day.

Netflix's scale makes the arithmetic straightforward. At the volume of inference requests a platform serving 300 million subscribers generates, per-token pricing from a hosted provider would represent a significant recurring cost. Owning the serving layer converts that variable cost into a fixed infrastructure cost that amortises over time.

For mid-market organisations, the crossover is less obvious and more dependent on workload shape. Bursty, unpredictable workloads favour hosted APIs because you pay only for what you use. Sustained, predictable workloads with known concurrency patterns favour self-hosting because you can right-size GPU capacity and eliminate the per-token margin the provider charges.

Latency, Control, and the Production Surprises

The latency characteristics of hosted APIs are opaque by design. You observe the tail latency your application experiences, but you have no visibility into why it varies or how to reduce it. For user-facing features where response time directly affects perceived quality, that opacity is a meaningful constraint.

Self-hosted infrastructure gives you full observability into the inference pipeline. You can profile where latency is accumulating, whether in tokenisation, the attention computation, or the decode loop, and address it directly. You can also implement request batching strategies tuned to your specific traffic pattern rather than accepting whatever batching strategy the provider has optimised for its aggregate workload.

The production surprises that only emerge under real load are worth naming explicitly. KV cache eviction under sustained concurrency is one. GPU memory fragmentation under variable sequence lengths is another. Thermal throttling on dense GPU clusters is a third. None of these appear in benchmarks. All of them affect production throughput, and all of them require infrastructure expertise to diagnose and address.

Evaluating the Decision Without Defaulting to Either Answer

The correct answer for most organisations is not "build everything" or "use hosted APIs for everything." It is a deliberate allocation of workloads based on their characteristics.

Workloads that are latency-sensitive, high-volume, and involve proprietary data are the strongest candidates for self-hosting. The latency control matters for user experience, the volume makes the economics work, and the data sensitivity may make external API calls a compliance risk. Workloads that are low-volume, experimental, or require access to frontier model capabilities that open-weight models cannot match are reasonable candidates for hosted APIs.

The structural risk to avoid is treating hosted APIs as the permanent answer for workloads that have outgrown them, simply because migrating away requires effort. That effort grows over time as the integration deepens. Evaluating the question deliberately at each stage of scale, rather than once at the beginning, is the practice that keeps the decision reversible.

Netflix's infrastructure choices are not a template to copy. They are a demonstration that the question deserves a rigorous answer, and that the default is not always the right one.

FAQs

At what scale does self-hosting LLM inference become economically justified?

The crossover depends on workload shape, GPU type, and the specific hosted provider's pricing, but as a rough frame, sustained workloads in the range of tens of millions of tokens per day typically reach the point where infrastructure costs plus engineering overhead are lower than per-token API fees. Bursty or unpredictable workloads shift that crossover significantly higher, because you cannot right-size GPU capacity without over-provisioning. The honest answer is that the calculation requires your actual traffic data, not a rule of thumb.

What engineering capabilities does an organisation need before self-hosting is viable?

At minimum, you need engineers who can operate GPU infrastructure, configure and tune an inference serving framework, build a model registry with versioning and rollback, and instrument the serving pipeline for observability. That is typically a team of three to five senior engineers with ML infrastructure experience, not a single hire. Organisations that attempt self-hosting without this foundation tend to encounter the production failure modes, KV cache eviction, memory fragmentation, thermal throttling, without the expertise to diagnose them quickly.

How should we think about data privacy when choosing between hosted APIs and self-hosting?

Hosted API calls send request data to a third-party provider's infrastructure. For many workloads that is acceptable, but for workloads involving sensitive customer data, regulated information, or proprietary business logic, the compliance and contractual implications require careful review. Self-hosting eliminates the third-party data transfer entirely, which simplifies the compliance position for regulated industries. This is often the deciding factor for financial services and healthcare organisations before the cost arithmetic even becomes relevant.

Which inference serving frameworks are worth evaluating, and how do they differ?

vLLM is the most widely adopted open-source option and handles memory management well through its PagedAttention implementation, which reduces fragmentation under concurrent batched requests. TensorRT-LLM delivers higher throughput on NVIDIA hardware but requires a model compilation step that adds deployment complexity. SGLang is gaining traction for workloads with structured output requirements. The right choice depends on your hardware, your model family, and whether compilation overhead is acceptable in your deployment workflow. Benchmarking against your actual traffic distribution matters more than vendor benchmark numbers.

What are the most common production failure modes that organisations encounter after moving to self-hosted inference?

The failures that appear most consistently in production are KV cache eviction under sustained concurrency, which causes latency spikes that are difficult to attribute without deep observability; GPU memory fragmentation under variable sequence length distributions, which reduces effective throughput below benchmarked levels; and thermal throttling on dense GPU clusters that have not been provisioned with adequate cooling headroom. A fourth category is model versioning failures, where a fine-tuned model is promoted to production without a tested rollback path, and a quality regression requires a manual recovery process. None of these appear in pre-production testing at realistic scale.

Is a hybrid approach, using hosted APIs for some workloads and self-hosting for others, operationally practical?

Yes, and for most organisations at mid-scale it is the most defensible position. The practical requirement is an internal abstraction layer that routes requests to the appropriate backend based on workload characteristics, without product teams needing to manage that routing directly. That abstraction also preserves the option to migrate workloads between backends as economics or compliance requirements change. The risk to manage is that the abstraction layer itself becomes a maintenance burden if it is not designed with sufficient care from the outset.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert