The Hidden Hardware Cost Hitting Enterprise AI Budgets: Memory Prices, Compute Scarcity, and What to Do Before 2028

The infrastructure assumptions baked into most enterprise AI roadmaps were written during a period of relative hardware abundance. That period is ending. GPU rental rates on major cloud platforms have more than doubled in some configurations, HBM memory costs have reached multi-year highs, and credible analysis of US compute capacity trajectories suggests that domestic supply will struggle to keep pace with demand through the late 2020s. For CTOs and VP-level infrastructure leaders making multi-year commitment decisions today, the relevant planning assumption is no longer "how do we optimise cost in a buyer's market" but "how do we secure capacity in a constrained one."

Companion piece to our broader work on AI infrastructure economics. See Cloud GPU Strategy: Cut Hidden AI Costs for a detailed breakdown of capacity gaps, egress traps, and commitment mismatches driving hidden cloud spend.

The Supply Signals That Most Procurement Teams Are Missing

GPU rental pricing is the most visible symptom of a deeper structural shift. Spot and on-demand rates for high-memory GPU instances have moved from roughly $5 per hour to over $13 per hour on some configurations over the past 18 months. That repricing is not a temporary market anomaly. It reflects a structural imbalance between accelerator supply, which is constrained by TSMC wafer allocation and advanced packaging capacity, and demand, which is compounding as enterprise inference workloads scale.

HBM memory is a second pressure point that receives less attention than GPU silicon itself. High-bandwidth memory is manufactured by a small number of fabs, and capacity expansion timelines are measured in years, not quarters. When memory costs rise, they affect not only new hardware purchases but also the economics of running existing infrastructure, since memory-intensive workloads consume more per token of useful output.

The third signal is geopolitical. US export controls on advanced semiconductors have reshaped global supply chains, and there is a credible near-term scenario in which the US becomes a net importer of compute tokens, meaning that inference capacity serving US enterprises is physically located offshore and subject to cross-border latency, data sovereignty risk, and foreign policy uncertainty. This is not a distant risk to model in a sensitivity analysis. It is a procurement consideration for decisions being made now.

Why Build-Versus-Buy Calculus Has Changed

The standard enterprise build-versus-buy framework for AI infrastructure assumes that cloud is flexible and on-premise is rigid. That framing made sense when cloud capacity was effectively unlimited and on-premise procurement lead times were the primary friction. Neither assumption holds cleanly today.

Cloud GPU reservations on one- and three-year committed use contracts are increasingly difficult to secure at scale for high-memory configurations. When capacity is available, the commitment terms are less favourable than they were two years ago. This shifts the calculus: on-premise ownership, despite its capital intensity and operational overhead, now offers something cloud cannot always guarantee, which is certainty of availability.

The practical implication is that enterprises should model their AI workload portfolio in terms of latency sensitivity and utilisation predictability. Workloads with high, predictable utilisation and no hard requirement for burst elasticity are now stronger candidates for on-premise deployment than they were 18 months ago. Workloads that are bursty, experimental, or geographically distributed remain better suited to cloud, but only where capacity can actually be committed.

Refurbished Hardware as a Legitimate Strategic Option

Refurbished enterprise GPU hardware has historically been treated as a fallback for budget-constrained teams without the procurement standing to negotiate new equipment. That framing deserves revision. In a supply-constrained market, refurbished kit from prior-generation accelerators represents a way to deploy inference capacity at lower cost and shorter lead times than new hardware, without depending on allocation from a fab running at capacity.

The trade-offs are real and should be modelled explicitly. Refurbished hardware carries higher failure rates, shorter remaining useful life, and reduced vendor support. For inference workloads running established model architectures, these trade-offs are often acceptable. For training runs that depend on weeks of uninterrupted compute, they are not.

The right approach is to treat refurbished hardware as a distinct tier within your infrastructure portfolio, appropriate for specific workload classes and managed with tighter monitoring and redundancy assumptions than new kit. Procurement teams that have never evaluated this market should begin building supplier relationships now, before scarcity drives further price convergence between refurbished and new.

Stress-Testing Your Roadmap Against a 2028 Compute Import Scenario

The 2028 planning horizon matters because it is when several converging pressures are likely to peak simultaneously. New fab capacity announced today will not reach volume production until 2026 or 2027. Demand from hyperscalers, sovereign AI programmes, and enterprise inference is compounding. If domestic compute supply falls short of demand at that point, enterprises without secured capacity will face rationing, not just repricing.

Stress-testing against this scenario means asking a specific set of questions about your current roadmap. What proportion of your inference capacity is on spot or short-term cloud contracts that could be repriced or withdrawn? What is your fallback if your primary cloud provider cannot fulfil a committed reservation? Do your model architecture choices assume access to specific memory configurations that may become scarce?

The goal of this exercise is not to produce a worst-case forecast but to identify the decisions that are reversible and the ones that are not. Long-term cloud commitments, on-premise capital expenditure, and model architecture choices all have different lead times and different costs to unwind. Understanding which commitments you are making implicitly, through inaction, is as important as the ones you are making explicitly through procurement.

Building Supply Resilience Into Your AI Stack

Supply resilience in AI infrastructure is not primarily a technical problem. It is a procurement and portfolio construction problem. The technical choices that matter most are the ones that preserve optionality: model architectures that can run across multiple hardware generations, inference pipelines that can shift between cloud providers without re-engineering, and monitoring instrumentation that gives you genuine visibility into utilisation and cost per workload.

On the procurement side, the practical steps are to diversify cloud provider relationships before you need to, to evaluate on-premise capacity for your highest-utilisation workloads, and to build refurbished hardware into your evaluation process as a legitimate option rather than a last resort. None of these steps requires a dramatic change in strategy. They require treating hardware supply as a risk variable that needs active management, in the same way that your team manages model performance or data quality.

The enterprises that will be best positioned in 2028 are not necessarily the ones with the largest budgets. They are the ones that made procurement decisions in 2025 and 2026 with an accurate model of the supply environment they were entering.