AI Strategy , Data science & AI , Software development Sep 25, 2026

Topology-Aware Scheduling Is the MLOps Gap Costing You GPU Utilisation at Scale

VECTOR Labs Team

Last updated on: Sep 25, 2026

Most engineering teams optimising GPU clusters spend their effort where the work is visible: model architecture, data pipeline throughput, checkpoint frequency. The scheduling layer sits beneath all of that, largely invisible until something goes wrong. When it does go wrong, the symptom is rarely a hard failure. It is a cluster running at 60 percent effective utilisation while the billing dashboard shows 100 percent allocation, with training jobs that should complete in hours taking days because collective communication is bottlenecked by inter-node bandwidth that the scheduler never accounted for.

Topology-aware scheduling is the practice of placing distributed workloads onto physical infrastructure in a way that reflects the actual communication costs between compute nodes. At scale, ignoring it is not a minor inefficiency. It is a structural drag on every distributed training run and multi-node inference deployment you operate.

What Topology Actually Means in a GPU Cluster

A modern GPU cluster is not a flat network of interchangeable nodes. Within a single node, GPUs are connected via NVLink or NVSwitch, which delivers bandwidth measured in terabytes per second. Across nodes within a rack, the interconnect drops to InfiniBand or high-speed Ethernet. Across racks or availability zones, it drops further. Each boundary introduces latency and bandwidth constraints that directly affect the all-reduce operations that distributed training depends on.

The implication is that two jobs with identical GPU counts can have dramatically different runtime characteristics depending on whether their processes are co-located on the same node, spread across a rack, or fragmented across the cluster. A scheduler that treats all GPUs as equivalent will routinely produce the worst-case placement for communication-intensive workloads.

This is not a theoretical concern. At the scale of hundreds or thousands of GPUs, the difference between topology-aware and topology-naive placement can determine whether a training run completes within a cost envelope or blows past it.

How Topology Discovery Works: NVIDIA Topograph and Related Tooling

Before a scheduler can make topology-aware decisions, it needs a structured representation of the cluster's physical layout. NVIDIA Topograph is a topology discovery tool designed to expose GPU interconnect hierarchies to orchestration layers. It interrogates the hardware to produce a graph representation of which GPUs share NVLink domains, which nodes share high-bandwidth fabric, and where the bandwidth boundaries sit.

Integration with Kubernetes

In Kubernetes environments, topology information surfaces through device plugins and node labels. The NVIDIA GPU Operator and associated device plugin expose GPU topology metadata that scheduling extensions can consume. The Kubernetes scheduler itself is extensible via the scheduling framework, which allows custom plugins to score node candidates based on topology affinity. Without this, the default scheduler has no concept of GPU locality and will place pods based on generic resource availability.

Integration with Slurm

Slurm has supported topology-aware scheduling for longer than Kubernetes, through its topology plugin architecture. The topology/tree plugin allows administrators to define a switch hierarchy that mirrors the physical network, and Slurm uses this to prefer job placements that minimise inter-switch communication. The critical dependency is that the topology configuration must accurately reflect the current physical state of the cluster, which requires operational discipline to maintain as hardware changes.

Gang Scheduling and Its Trade-offs

Distributed training jobs require all participating processes to start simultaneously. This is the gang scheduling problem: you cannot begin a 64-GPU job until all 64 GPUs are available and allocated together. Partial allocations that wait for the remaining resources create head-of-line blocking, where a large job holds partially allocated resources while smaller jobs that could run immediately are queued behind it.

The trade-off is between throughput and fairness. Strict gang scheduling maximises the likelihood that large jobs run at full efficiency once they start, but it can reduce overall cluster throughput if the scheduler cannot fill the gaps left while waiting for a full allocation. Backfill scheduling, where smaller jobs are inserted into idle slots without delaying higher-priority jobs, is the standard mitigation, but it requires the scheduler to reason about both topology and time simultaneously.

Volcano, an open-source batch scheduling system for Kubernetes, addresses this through job-level scheduling primitives that treat a distributed training job as a single schedulable unit. It supports gang scheduling with configurable minimum viable gang sizes, which allows a job to proceed with fewer than its maximum requested resources if the topology is still acceptable. This is a meaningful operational lever for teams running clusters with heterogeneous utilisation patterns.

What to Evaluate Before Topology Problems Become Production Bottlenecks

The first question to answer is whether your current scheduler has any awareness of GPU topology at all. If you are running vanilla Kubernetes without custom scheduling plugins, the answer is almost certainly no. Auditing your scheduler configuration and node labelling strategy is the necessary starting point, not an optimisation step.

The second question is whether your topology metadata is accurate and current. A topology graph that does not reflect recent hardware additions, node replacements, or network reconfigurations will produce placements that are worse than random because the scheduler will make confident wrong decisions. Topology discovery tooling needs to be integrated into your infrastructure provisioning pipeline, not run as a one-time setup step.

The third question is whether your job submission tooling exposes topology preferences to the scheduler. Even with a topology-aware scheduler in place, jobs that do not declare their communication patterns or resource group preferences will not benefit from topology affinity. This is an interface design problem between your ML platform and your infrastructure layer, and it is frequently the gap that renders topology-aware schedulers ineffective in practice.

Where Vector Labs Fits

We design and implement ML infrastructure systems where resource allocation decisions have direct cost and performance consequences. In our MLOps pipeline cost analysis, we identified how synchronous orchestration patterns and idle worker accumulation translate directly into wasted cloud spend, with practical remediation patterns that improved worker saturation without sacrificing pipeline reliability. If you are evaluating your cluster scheduling architecture and want a structured assessment of where topology gaps are affecting your utilisation economics, contact us at vector-labs.ai/contacts.

FAQs

How do I know if topology-naive scheduling is actually affecting my training job performance?

The clearest signal is a gap between theoretical and observed all-reduce bandwidth. Profile your collective communication operations using tools like NCCL's built-in logging or NVIDIA Nsight Systems, and compare the bandwidth you observe against the rated bandwidth of your intra-node interconnect. If jobs placed across multiple nodes are achieving bandwidth closer to your inter-node network ceiling than your NVLink ceiling, the scheduler is likely fragmenting placements that should be co-located. A secondary signal is high variance in job completion times for identically configured runs submitted at different cluster load levels.

Is Volcano the right choice for topology-aware scheduling on Kubernetes, or are there alternatives worth evaluating?

Volcano is the most mature open-source option for batch and distributed training workloads on Kubernetes, and its gang scheduling primitives are well-suited to the problem. The main alternatives are Yunikorn, which has strong multi-tenant fairness capabilities and topology plugin support, and the upstream Kubernetes scheduler extended with the Topology Manager and custom scoring plugins. The right choice depends on whether your primary constraint is topology affinity, multi-tenant fairness, or integration with existing Kubernetes tooling. Volcano is a reasonable default for teams whose primary workload is distributed training. Teams running mixed inference and training workloads may find Yunikorn's queue management model a better fit.

How does topology-aware scheduling interact with spot or preemptible instance strategies?

This is where the trade-offs become genuinely difficult. Spot instances introduce topology uncertainty because the available inventory at any given moment may not align with your preferred placement topology. A scheduler optimising for topology affinity on spot capacity may wait longer for a topologically ideal allocation that never materialises, effectively reducing throughput. The practical mitigation is to define minimum acceptable topology constraints rather than optimal ones, so the scheduler can proceed with a good-enough placement when ideal inventory is unavailable. Checkpoint frequency and elastic training support also become more important in this regime, because preemption events will disrupt gang allocations regardless of how well the initial placement was optimised.

Does topology-aware scheduling matter for inference workloads, or is it primarily a training concern?

It matters for inference at the scale where multi-node tensor parallelism is required, which is increasingly common for large language model serving. When a single model is sharded across multiple GPUs and nodes, the inter-GPU communication for attention and feed-forward layers during each forward pass is subject to the same bandwidth constraints as training collectives. Topology-naive placement of a multi-node inference deployment will produce higher per-token latency and lower throughput than a topology-aware placement. For single-node inference, the concern is intra-node GPU affinity, which is typically handled by the device plugin rather than the cluster scheduler.

What is the operational overhead of maintaining accurate topology metadata as the cluster evolves?

This is underestimated by most teams when they first implement topology-aware scheduling. The topology graph is only as useful as its accuracy, and clusters change: nodes are replaced, NICs are swapped, network switches are reconfigured. The sustainable approach is to integrate topology discovery into your infrastructure-as-code pipeline so that any hardware change triggers a topology re-discovery and metadata update automatically. Manual topology configuration that is updated on an ad-hoc basis will drift from physical reality within months on an actively managed cluster, and a stale topology graph can produce placements that are confidently wrong. The operational investment in automating this is modest relative to the cost of running on incorrect topology data.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert