TPU vs GPU: What the Inference Economics War Means for Your Cloud AI Spend

The default infrastructure stack for enterprise AI has been GPU-centric for long enough that many teams treat it as a settled question. It is not. The emergence of Google's TPUv7 Ironwood alongside NVIDIA's Blackwell architecture has created a genuine like-for-like comparison at the frontier of inference hardware, and the cost implications are significant enough to warrant a structured audit of where your workloads actually sit. The problem is that most published comparisons flatten the architectural differences in ways that make the numbers look cleaner than they are in practice.

Companion piece to our broader work on AI infrastructure economics. See The Memory Wall Is Now Your Inference Problem for how chip architecture, KV cache efficiency, and high-bandwidth memory shape inference costs at the hardware level.

Why the Ironwood vs Blackwell Comparison Is Harder Than It Looks

Topology and Memory Layout

TPUv7 Ironwood is designed around a tightly coupled pod topology where inter-chip communication is handled through Google's custom interconnect fabric rather than NVLink or InfiniBand. This matters because the effective memory bandwidth available during inference depends heavily on how the model is sharded across chips, and the sharding strategies that work well on TPU pods do not map cleanly onto Blackwell's NVLink-connected GPU clusters.

Blackwell's architecture centres on a high-bandwidth memory configuration that gives individual GPUs substantial local memory capacity. This suits workloads where the model fits within a small number of devices and latency-sensitive serving benefits from minimising cross-device communication. Ironwood's design trades some of that per-chip flexibility for higher aggregate throughput across the pod, which changes the cost calculus depending on whether your serving pattern is throughput-bound or latency-bound.

Disaggregated vs Aggregated Serving

The architectural distinction that most enterprise teams underestimate is the difference between disaggregated and aggregated inference serving. Disaggregated serving separates the prefill and decode phases of autoregressive generation onto different hardware pools, allowing each phase to be scaled independently. TPU pods are well-suited to disaggregated architectures because the interconnect fabric can route prefill requests to a dedicated set of chips without the overhead that cross-node GPU communication introduces.

Blackwell clusters running aggregated serving keep both phases on the same device, which simplifies orchestration but creates resource contention under mixed traffic patterns. The cost difference between these two approaches only becomes visible when you model your actual request distribution, not a synthetic benchmark.

The Workload Conditions Where TPU Economics Win

Ironwood's cost advantage is most pronounced under sustained, high-throughput batch inference with predictable request shapes. When your serving pattern looks like a continuous stream of similar-length prompts, the pod topology can sustain high chip utilisation because the interconnect is not being stressed by variable-length sequences that cause load imbalance.

The economics shift when you introduce long-context requests or highly variable output lengths. These patterns create prefill-decode imbalance that disaggregated TPU serving handles well in theory, but requires careful capacity planning to avoid underutilising the decode pool during low-traffic periods. If your traffic is spiky or your context lengths vary widely, the utilisation assumptions embedded in TPU pricing need to be stress-tested against your actual request logs, not averaged traffic figures.

Where GPU Infrastructure Retains the Advantage

Blackwell retains a meaningful advantage in two scenarios that are common in enterprise deployments. The first is multi-model serving, where a single cluster needs to host several models simultaneously and shift capacity between them based on demand. GPU clusters support this through more granular resource partitioning than current TPU pod configurations allow.

The second is fine-tuned model iteration. Teams that run frequent fine-tuning cycles and then immediately serve the updated model benefit from GPU infrastructure because the same hardware handles both workloads without requiring a separate training cluster. TPU economics assume a cleaner separation between training and serving infrastructure, which not all organisations can operationalise cleanly given their deployment cadences.

Building a TCO Model That Accounts for Architecture

Total cost of ownership calculations for inference infrastructure need to account for at least four variables that vendor pricing sheets do not surface directly: chip utilisation at your actual traffic distribution, memory bandwidth saturation under your specific model architecture, inter-chip communication overhead at your sequence length distribution, and the engineering cost of serving framework compatibility.

The last point is frequently underweighted. JAX-based serving on TPUs requires different engineering investment than CUDA-based serving on GPUs, and teams with existing CUDA expertise face a real productivity cost during transition that belongs in the TCO model. The hardware cost per token is only one line in that calculation.

What a Workload Audit Should Cover

Before committing infrastructure spend based on vendor benchmarks, an audit of your inference workloads should establish the following:

• Request shape distribution: median and 95th percentile prompt length, output length, and requests per second by time of day
• Prefill-to-decode ratio: the proportion of compute time spent in each phase across your production traffic
• Model serving pattern: whether you run one large model, multiple smaller models, or a mixture that shifts across the day
• Fine-tuning cadence: how frequently updated model weights need to be served and whether training and serving infrastructure is shared

These inputs determine which architectural assumptions hold for your workloads. Without them, a cost comparison between Ironwood and Blackwell is not a TCO analysis. It is a benchmark applied to someone else's workload.