Engineering leaders procuring AI infrastructure in 2026 are still making decisions that will shape their operational costs for the next three to five years, yet most procurement conversations anchor almost entirely on compute specifications. Peak FLOPS, tensor core counts, and GPU generation are the numbers that dominate the slide decks. The actual utilisation ceiling in production clusters, however, is almost never set by raw compute capacity. It is set by how quickly data can move between compute elements, and how much memory bandwidth is available to feed them. Until infrastructure teams treat memory and interconnect as a unified system rather than separate line items, they will continue paying for compute they cannot fully use.
Why Compute Utilisation Numbers Lie
Headline GPU utilisation figures reported by cluster monitoring tools are frequently misleading. A GPU can show 90% utilisation while the underlying compute elements are stalled waiting for memory transfers, which means the metric is capturing activity, not productive throughput.
The distinction matters commercially. An infrastructure team that interprets high utilisation as a signal that the cluster is well-provisioned will resist adding memory bandwidth or rethinking interconnect topology. The result is continued underperformance from hardware that was expensive to procure and is expensive to operate.
The correct diagnostic question is not "how busy are the GPUs?" but "what fraction of peak theoretical throughput are workloads actually achieving, and where is the gap?" Answering that question requires instrumenting memory bandwidth saturation and interconnect contention alongside compute utilisation, and treating all three as co-equal signals.
The CXL Memory Expansion Trade-off
CXL (Compute Express Link) memory expansion has attracted significant attention as a way to increase the effective memory capacity available to AI accelerators without replacing the accelerators themselves. The appeal is straightforward: attach additional DRAM over a standardised coherent interconnect and defer the need for a hardware refresh.
The trade-off is latency. CXL-attached memory introduces additional hop latency compared to on-package HBM or directly attached DRAM, and for memory-bandwidth-bound workloads, that latency difference is not negligible. Inference serving for large language models, where KV-cache access patterns are irregular and latency-sensitive, is particularly exposed to this penalty.
The practical implication is that CXL memory expansion is a reasonable fit for capacity-constrained workloads where access patterns are relatively predictable and latency tolerance is higher, such as batch inference or certain training data pipelines. It is a poor fit for latency-critical serving paths. Infrastructure teams should model their specific access patterns before treating CXL as a general-purpose capacity solution.
Scale-Up Versus Scale-Out Interconnect Architecture
Scale-up interconnect, typified by NVLink and its successors, increases bandwidth between a small number of tightly coupled accelerators within a node. Scale-out interconnect, typified by InfiniBand or RoCE fabrics, connects larger numbers of nodes at lower per-link bandwidth. The two approaches are not substitutes for each other, but the boundary between them is shifting.
As model sizes have grown, the communication volume associated with tensor parallelism has increased to the point where scale-out fabrics become a bottleneck before compute is saturated. All-reduce operations across hundreds of nodes at 400Gb/s InfiniBand links can leave GPU compute idle for a significant fraction of each training step. The interconnect is not a supporting component in this architecture; it is a primary determinant of effective throughput.
The architectural implication is that clusters designed for very large model training need to be sized with interconnect bandwidth as a first-order constraint, not an afterthought. Teams that have already committed to a scale-out topology should audit whether their collective communication patterns can be restructured, through pipeline parallelism or gradient compression, to reduce the volume of data crossing the fabric.
Wafer-Scale Mapping Inefficiencies and What They Reveal
Wafer-scale accelerators represent the most aggressive available approach to reducing the cost of on-chip data movement by eliminating the package boundaries that constrain conventional multi-chip systems. The Cerebras WSE-3 is the most prominent production example. The theoretical advantage is significant: hundreds of thousands of processing elements sharing a single on-wafer interconnect fabric, with memory distributed across the die.
The practical challenge is mapping. Existing template-based mapping approaches impose fixed geometric layouts over the processing element grid, which creates spatial fragmentation and leaves both compute and memory resources underutilised. Research from the University of Edinburgh demonstrates this concretely: WaferTetris, a freeform mapping system that removes geometric layout constraints through a Tetris-like packing strategy, achieves up to 4.42 times the system throughput of a template-based automatic mapping baseline on the same fabric, and supports 1.8 times the context length for LLaMA 3.1-8B compared to manually optimised approaches (Du et al., arXiv 2026).
The broader lesson for infrastructure teams who are not yet operating wafer-scale hardware is that mapping efficiency is a real and measurable variable, not an implementation detail. The same principle applies at smaller scales: how workloads are partitioned and placed across a multi-GPU cluster affects memory locality and interconnect utilisation in ways that are not captured by compute specifications alone.
A Procurement Audit Framework for Infrastructure Leaders
Before committing to a hardware refresh or a new cluster build, infrastructure teams should work through four diagnostic questions.
First, what is the memory bandwidth utilisation profile of current workloads, separated by training and inference? If bandwidth saturation is occurring before compute saturation, additional GPUs will not solve the problem.
Second, what is the ratio of inter-node communication volume to compute volume for the largest models being trained? If that ratio is high, the interconnect fabric deserves more budget weight than the procurement model currently assigns it.
Third, for inference specifically, what is the KV-cache access pattern, and does the memory hierarchy match it? Mismatches between access patterns and memory tier latency are a common source of unexplained latency in production serving.
Fourth, for teams evaluating wafer-scale or novel accelerator architectures, what tooling exists for mapping workloads onto the hardware, and how mature is it? Hardware that requires manual mapping optimisation to approach its theoretical throughput is hardware that carries a hidden engineering cost.
Answering these questions before procurement, rather than after, is the difference between infrastructure that performs as specified and infrastructure that underperforms for reasons that are difficult to diagnose once the hardware is installed.
Where Vector Labs Fits
We help engineering teams diagnose and resolve the gap between theoretical hardware capacity and production AI performance. In our memory-wall analysis, we examined how chip architecture, KV-cache efficiency, and high-bandwidth memory interact to determine real inference economics across deployment configurations. If you are working through a cluster procurement decision or an infrastructure audit, contact us at vector-labs.ai/contacts.
FAQs
The most direct method is to instrument your workloads with hardware performance counters that report memory bandwidth utilisation alongside compute utilisation. If memory bandwidth is saturating while GPU compute utilisation remains below 80–85%, the cluster is bandwidth-bound. Profiling tools such as NVIDIA Nsight Systems expose this at the kernel level, which allows you to identify specific operations rather than drawing conclusions from aggregate metrics alone.
It depends on the latency tolerance of the specific serving path. CXL-attached memory increases capacity at lower cost than HBM, but it introduces additional access latency relative to on-package memory. For batch inference where throughput matters more than individual request latency, CXL can be a reasonable fit. For real-time serving with tight SLA requirements, the latency penalty is likely to create problems that offset the capacity benefit.
Scale-up interconnect is the better choice when workloads require tight tensor parallelism across a small number of accelerators, because the higher per-link bandwidth reduces the communication overhead that otherwise stalls compute. Scale-out becomes the primary architecture when the cluster needs to support many concurrent jobs or pipeline-parallel training across large node counts. Most production clusters need both, and the error is treating them as alternatives rather than complementary layers with different roles.
The most important questions concern mapping tooling maturity and the engineering effort required to achieve near-peak utilisation. Ask for documented utilisation figures from production workloads that are similar to yours, not benchmark figures from idealised configurations. Ask whether automated mapping tools are available or whether achieving high utilisation requires manual placement work. The gap between theoretical peak throughput and what your team can realistically extract in a production environment is a real cost that belongs in the procurement model.
A reasonable starting point is to treat interconnect bandwidth as a constraint that must be satisfied before compute capacity is expanded. Calculate the all-reduce communication volume your largest planned workload requires per training step, then verify that your fabric can sustain that volume without leaving GPU compute idle. If the numbers do not balance, additional compute spend will not improve end-to-end throughput, and the budget is better directed at the interconnect layer first.

