The Open Model Cost Arbitrage: A Framework for Engineering Leaders Evaluating the Switch from Proprietary AI

Enterprises running meaningful proprietary AI spend are increasingly asking whether open-weight models represent a genuine cost reduction or simply a different cost structure. The honest answer is that it depends on where your inference volume lives, what your engineering team can absorb, and whether the performance delta between open and proprietary models actually matters for your specific workloads. This article builds a decision framework around those three variables, drawing on production infrastructure evidence and real-world deployment trade-offs.

Companion piece to our broader work on open-weight model economics. See Open-Weight Models in Production: What the Performance Gap Actually Costs and When It Stops Mattering for a detailed analysis of benchmark performance, self-hosting costs, and the business case for switching.

Where the 50% Cost Reduction Figure Actually Comes From

The headline number circulating in engineering leadership discussions, that enterprises can cut AI spend by half through open model adoption, is real but context-dependent. It applies most reliably to high-volume, low-complexity inference workloads where a well-quantised open model running on owned or reserved compute can match proprietary output quality at a fraction of the per-token cost.

The mechanism is straightforward: proprietary API pricing bundles inference compute, model development amortisation, and margin into a single per-token rate. When you self-host, you pay only for compute, and at sufficient volume that arithmetic tips decisively in your favour. The inflection point for most enterprise workloads sits somewhere between 50 million and 200 million tokens per month, depending on the GPU tier you can access and the model size required for your task.

What the comparison decks rarely surface is that this calculation assumes stable, predictable throughput. Spiky or seasonal workloads reduce the advantage because idle reserved compute still costs money, while API pricing scales to zero when you are not using it.

The Infrastructure Overhead That Erodes the Savings

Self-hosting an open-weight model is not a one-time engineering task. It is an ongoing operational commitment covering model versioning, inference optimisation, hardware procurement cycles, and failure recovery. Each of these carries an engineering cost that does not appear in the initial TCO model.

Inference serving is the first place this overhead becomes visible. Production deployments require batching strategies, KV cache management, and quantisation decisions that meaningfully affect both latency and throughput. Getting these right for a specific workload requires engineers who understand the model architecture and the serving framework simultaneously, which is a narrower skill set than most teams carry.

The post-training question compounds this further. If your use case requires domain adaptation or instruction tuning, you are now running a training infrastructure alongside your inference infrastructure. Systems like Miles v0.1, a full-stack reinforcement learning training framework from RadixArk built around verifiability, efficiency, and customisability as first-class design principles, demonstrate that production-grade open training infrastructure does exist (RadixArk et al., arXiv 2026). The capability is real, but operating it requires engineering capacity that proprietary APIs simply do not demand.

Edge Deployment as a Distinct Cost Case

Edge inference follows a different economic logic than cloud self-hosting. When the model runs on-device or at the network edge, there is no per-query compute cost at all after the initial hardware and deployment investment. For high-frequency, latency-sensitive applications, this can produce cost structures that no cloud API arrangement can match.

The trade-off is model capability. Edge-viable models are necessarily smaller, and smaller models carry a larger performance gap against frontier proprietary alternatives. Whether that gap is acceptable depends entirely on task complexity. Structured extraction, classification, and short-form generation often tolerate it well. Open-ended reasoning and multi-step agentic tasks typically do not.

The architectural decision here is not binary. Several production deployments use a routing layer that sends simple queries to an on-device model and escalates complex queries to a cloud endpoint, either open or proprietary. This hybrid approach captures edge economics for the majority of traffic while preserving quality on the queries that require it.

Building the Decision Framework

The evaluation reduces to four questions that engineering leaders should answer before committing infrastructure budget.

First, what is your monthly token volume and how predictable is it? Below the self-hosting inflection point, or with high variance, API pricing is likely more efficient in total cost terms.

Second, what is the performance gap on your actual workloads, not benchmarks? Run the candidate open model against a representative sample of your production queries before making any infrastructure commitment.

Third, what is your team's realistic capacity to operate inference and, if needed, training infrastructure? The engineering overhead is not theoretical. It requires sustained headcount with specific skills.

Fourth, does your deployment context include edge or air-gapped environments? If it does, open models may be the only viable path regardless of cost, which changes the framing from cost arbitrage to capability access.

Where the Framework Breaks Down

The cost arbitrage argument weakens in two specific situations. The first is when the workload genuinely requires frontier-level reasoning capability that current open models do not match. Forcing a smaller open model onto a task it cannot reliably complete does not save money; it creates downstream quality costs that are harder to measure but very real.

The second is when your organisation lacks the engineering depth to operate the infrastructure competently. A poorly optimised self-hosted deployment can cost more than the API it replaced while delivering worse latency and reliability. The framework assumes a minimum level of ML infrastructure competence that not every engineering team has, and being honest about that gap upfront is considerably less expensive than discovering it after six months of migration work.