AI Factory Observability: Why Your GPU Cluster Needs a Knowledge Graph, Not Just Dashboards

Most engineering teams inherit their GPU monitoring philosophy from the web services world: stand up a metrics exporter, wire it into Grafana, and declare the infrastructure observable. That approach works reasonably well when your failure modes are stateless and your components are loosely coupled. GPU training clusters are neither. When a single degraded NVLink connection or a thermally throttled accelerator quietly reduces effective compute throughput across a multi-node job, no individual dashboard catches it because no individual dashboard has the context to connect the symptom to its cause. Observability architecture for AI factories is now an engineering decision that directly determines operational yield, and teams that treat it as an afterthought are paying for that decision in GPU-hours they cannot account for.

Companion piece to our broader work on AI infrastructure visibility. See Full-Stack Observability for AI Systems for what engineering teams consistently get wrong before production breaks.

The Fragmentation Problem at Scale

The standard monitoring stack for a mid-sized GPU cluster typically looks like this: a DCGM exporter feeding GPU utilisation and memory bandwidth metrics, a separate networking dashboard tracking InfiniBand or RoCE counters, and a storage layer monitored in isolation through filesystem or NVMe telemetry. Each layer is instrumented. None of them are correlated.

The problem is not that these tools are bad individually. The problem is that degradation in AI training workloads is almost always a cross-layer phenomenon. A job running at 60% of its expected MFU (Model FLOPS Utilisation) may be suffering from PCIe contention, a single slow NVMe drive in the checkpoint path, or a fabric congestion event that only manifests during the all-reduce phase of a distributed step. Diagnosing that requires correlating GPU kernel execution timelines, network flow telemetry, and storage I/O latency simultaneously, with enough context to know which components were involved in which training job at which point in time.

Siloed dashboards cannot provide that context. They can tell you that something is wrong in their domain. They cannot tell you why throughput degraded at the workload level, which is the question that actually matters operationally.

AI Factory Yield as an Operational Metric

The concept of yield, borrowed from semiconductor manufacturing, translates directly to GPU cluster operations. In a fab, yield measures the fraction of wafers that produce functional chips. In an AI factory, the equivalent is the fraction of reserved GPU-hours that produce useful gradient steps at target throughput.

Yield captures what utilisation metrics miss. A cluster can report 95% GPU utilisation while delivering 60% of its theoretical training throughput, because utilisation measures whether the GPU is busy, not whether it is doing productive work. Stragglers in data pipelines, synchronisation overhead in collective operations, and checkpoint I/O blocking all consume GPU time without advancing training. None of these show up as underutilisation.

Tracking yield requires instrumenting the full compute path: training step duration, time spent in collective communication versus compute, checkpoint write latency, and data loader idle time. When yield drops, the diagnostic question is which phase of the training loop degraded and on which nodes, which is precisely the question that requires cross-layer telemetry to answer.

The Knowledge Graph Architecture

A knowledge graph approach to infrastructure telemetry replaces the parallel-dashboard model with a unified entity-relationship store. Every physical and logical component in the cluster becomes a node: GPUs, CPUs, NICs, NVMe devices, switches, training jobs, model checkpoints, and NCCL process groups. Relationships between them are edges: which GPU belongs to which node, which node is connected to which switch port, which training job is allocated to which set of GPUs.

Entity Modelling

The graph schema needs to capture both static topology and dynamic state. Static topology changes infrequently and describes the physical wiring of the cluster. Dynamic state includes current thermal readings, error counters, bandwidth utilisation, and job assignments. Separating these two layers matters because topology queries and time-series queries have different access patterns and different retention requirements.

Causal Traversal

The operational value of the graph is in traversal. When a training job reports degraded step time, the query is not "which metrics are elevated?" but rather "which components in the subgraph occupied by this job have anomalous state?" That traversal can cross layer boundaries in a single query: start from the job node, walk to its GPU assignments, check thermal and error state, walk to the NIC attached to each GPU host, check fabric error counters, walk to the NVMe devices in the checkpoint path, check write latency percentiles.

This is the query that siloed dashboards cannot execute. The graph makes it a single traversal with a sub-second response time.

Instrumenting for Degradation Tracing

Building this system requires instrumentation decisions at each layer that prioritise relationship fidelity over raw metric volume. The most common mistake is collecting high-cardinality time-series data without tagging it in a way that preserves its relationship to specific jobs and hardware components.

GPU and Interconnect Telemetry

DCGM provides a rich set of per-GPU counters, but the ones that matter most for degradation tracing are not always the ones teams enable by default. NVLink replay errors, PCIe correctable error rates, and SM throttle reasons are more diagnostically useful than aggregate utilisation when the goal is identifying which GPU is the straggler in a multi-node job. Every metric emitted should carry both the physical device identifier and the job identifier of any workload currently assigned to that device.

Collective Communication Profiling

NCCL and similar collective communication libraries are often the least-instrumented layer in the stack, despite being the most common source of throughput degradation in large distributed training runs. Profiling collective operations at the level of per-rank timing distributions reveals whether slow steps are caused by a specific rank, which maps directly back to a specific host and NIC. This data should flow into the knowledge graph as edge attributes on the relationship between training jobs and network components.

Incident Response at Cluster Scale

The operational payoff of a unified knowledge graph becomes clearest during incident response. When a training job degrades, the mean time to diagnosis in a siloed monitoring environment is typically measured in hours: engineers manually correlating timestamps across separate dashboards, forming hypotheses, and checking each system in sequence.

With a graph-backed observability system, the diagnostic workflow inverts. The starting point is the affected job node. The first query returns all components in its subgraph with anomalous state, ranked by the magnitude of the anomaly and the directness of the relationship to the job. In most cases, the root cause component is in the top three results.

This matters commercially because training jobs at scale are expensive. A four-hour degradation event on a 512-GPU job represents a material cost even before accounting for the delayed model delivery. Reducing mean time to diagnosis from hours to minutes is not an operational nicety. It is a direct recovery of yield that the cluster was already paying for.