From Cloud Dependency to On-Premise Control: What the New AI Hardware Stack Means for Enterprise Infrastructure Decisions

Enterprise AI spending has matured past the point where cloud-first is automatically the right answer. As inference volumes grow and model complexity increases, the economics of renting compute begin to shift against organisations that run sustained, high-throughput workloads. The decision to own inference infrastructure is no longer a niche concern for defence contractors and regulated financial institutions. It is a live commercial question for any enterprise that has moved beyond pilot deployments and is now managing AI at operational scale. This article works through the criteria that actually determine whether on-premise hardware makes sense, and where the transition from sovereignty argument to cost argument genuinely occurs.

The VRAM Ceiling Is the First Constraint to Model

Before any procurement conversation begins, the parameter count of the models you intend to run determines your minimum viable hardware configuration. A 70-billion-parameter model quantised to 4-bit precision requires roughly 35GB of VRAM to load, with additional headroom needed for the key-value cache during inference. That number immediately rules out single-GPU consumer hardware and places you in multi-GPU territory.

The practical implication is that VRAM capacity is not a performance variable, it is a binary gate. A GPU with insufficient VRAM cannot run the model at all, regardless of compute throughput. Organisations that size hardware around benchmark scores rather than memory requirements routinely discover this constraint only after procurement.

The secondary issue is memory bandwidth, not just capacity. Autoregressive generation in large language models is memory-bandwidth-bound during the decode phase. A configuration that technically fits the model in VRAM but runs on older memory interconnects will produce latency that makes synchronous user-facing applications impractical.

Rack-Scale Power and Cooling Are Not Afterthoughts

A single high-density GPU server drawing 10 to 14 kilowatts places demands on facility infrastructure that most enterprise data centres were not designed to meet. Standard rack power delivery is typically rated at 5 to 7 kilowatts per rack. Running modern AI accelerators at density requires either upgraded power distribution units, dedicated circuits, or purpose-built colocation space.

Cooling is the constraint that most frequently delays on-premise deployments in practice. Air-cooled racks struggle above certain thermal densities, and the transition to direct liquid cooling or rear-door heat exchangers introduces capital expenditure and lead times that procurement teams consistently underestimate. The hardware itself may arrive in weeks. The facility modifications to support it safely can take months.

This means the total cost of ownership calculation for on-premise AI hardware must include facility readiness as a first-order variable. Organisations that model hardware cost in isolation and treat facility upgrades as a separate budget line often find the business case looks materially different once the full picture is assembled.

Where Sovereignty Ends and Cost Advantage Begins

The initial case for on-premise AI infrastructure in regulated industries is almost always framed around data sovereignty. Healthcare, financial services, and defence-adjacent organisations face genuine constraints on where inference can occur and what data can transit external networks. For these organisations, the build decision is partly non-negotiable regardless of unit economics.

For organisations outside those regulatory categories, the cost argument takes longer to materialise. Cloud inference pricing for frontier models is structured to be competitive at low to moderate volumes. The crossover point, where owned hardware produces lower total cost per inference than cloud API consumption, typically requires sustained high-throughput usage over a multi-year depreciation horizon. Below that threshold, the operational overhead of managing owned infrastructure erodes the apparent saving.

The calculation shifts when organisations move from proprietary API-dependent models to open-weight alternatives. Running a capable open-weight model on owned hardware removes per-token pricing entirely. We have covered the performance and cost dynamics of that transition in more depth in our analysis of open-weight models in production, which is directly relevant to teams working through this specific decision.

Companion piece to our broader work on open-weight model deployment economics. See Open-Weight Models in Production: What the Performance Gap Actually Costs and When It Stops Mattering for a detailed breakdown of benchmark performance gaps, self-hosting costs, and the business case for switching.

Building a Procurement Framework That Matches Model Ambition to Infrastructure Reality

A hardware procurement decision made without a clear model roadmap is almost always wrong within eighteen months. GPU generations advance quickly, and a configuration sized for today's frontier model may be inadequate for the next generation or unnecessarily over-specified for a quantised mid-tier model that delivers equivalent task performance at a fraction of the memory requirement.

The framework we recommend to enterprise clients starts with three inputs: the maximum parameter count you expect to run in the next two years, the peak concurrent inference load that represents your operational ceiling, and the latency requirements of the workloads you are targeting. Those three variables constrain the hardware configuration more precisely than any benchmark comparison.

From those inputs, the procurement decision branches into two distinct paths. Workloads with hard latency requirements and high concurrency demand dedicated accelerator clusters with high-bandwidth interconnects. Workloads that are batch-oriented or asynchronous can tolerate shared infrastructure and lower-cost hardware configurations, which substantially changes the capital commitment required.

The Messy Middle Ground Between Consumer and Enterprise Hardware

There is a class of hardware that sits between a $2,000 workstation GPU and a $40,000-plus rack system that rarely appears in enterprise procurement discussions. Workstation-class multi-GPU configurations using professional accelerators can run 30 to 70 billion parameter models at practical inference speeds for internal tooling, developer access, and moderate-volume production workloads.

These configurations are not appropriate for high-concurrency customer-facing applications. They are, however, a legitimate entry point for organisations that want to validate on-premise deployment operationally before committing to rack-scale capital expenditure. The operational learning that comes from running owned inference infrastructure at this tier is commercially valuable regardless of whether the organisation eventually scales up.

The risk of this approach is treating it as a permanent solution rather than a validation phase. Hardware at this tier lacks the redundancy, serviceability, and thermal management of purpose-built inference servers. Organisations that deploy it without a defined upgrade path often find themselves managing fragile production dependencies on hardware that was never designed for that role.