Gang scheduling and topology-aware placement for AI training in Kubernetes | CNCF

Enroll your company as a CNCF End User and save more than $10K in training and conference costs
Posted on November 12, 2026 by David Lentz, DataDog
CNCF projects highlighted in this post
Distributed AI training workloads impose complex scheduling requirements that Kubernetes’s built-in scheduler can’t meet. Kubernetes schedules pods individually and independently, but distributed training introduces two requirements that break this model: Pods must land on hardware with the right inter-GPU bandwidth, and all pods must be scheduled simultaneously. If either requirement goes unmet, training stalls or runs far below the hardware’s potential.
Topology-aware scheduling (TAS) and gang scheduling address these two requirements and help organizations extract full value from GPU capacity, which represents a growing share of infrastructure spend. TAS shapes the relative placement of pods to enable fast communication between GPUs. Gang scheduling helps ensure that all pods in a distributed job start running simultaneously, preventing the partial starts that leave GPUs allocated but idle. Together, they give Kubernetes the placement precision and coordination that distributed workloads require.
In this post, we’ll look at the scheduling problems that AI training presents for Kubernetes and how TAS and gang scheduling, implemented by Kueue and the Coscheduling plugin, solve them. We’ll explore:
We’ll also cover the evolving capabilities of Kubernetes native scheduler to support AI training workloads.
In most batch workloads, pods operate independently. A pod can fail, be rescheduled, or be replaced with no effect on the rest of the workload. The Kubernetes scheduler handles this model well by placing pods one at a time on any eligible node without coordinating across pods.
But the pods in an AI training workload are interdependent. They need to be placed on hardware with appropriate inter-GPU bandwidth, and they all need to start simultaneously. The following sections explain these requirements and how Kueue and Coscheduling address them by augmenting Kubernetes’ scheduling capabilities.
The physical interconnections between nodes are a key factor in the performance of AI training workloads. In each step of the training process, workers exchange data through collective communication operations, and the type and frequency of those operations depends on the job’s parallelization strategy. In pure data-parallel (DDP) training, one AllReduce operation per step aggregates, averages, and distributes gradients—the computed signals that indicate how the model’s parameters should be adjusted to improve its predictions. In large-scale training strategies that shard the model across workers, those workers instead exchange data continuously throughout a step, by using operations like gather, reduce, and all-to-all. These large-scale strategies include tensor, pipeline, and expert parallelism, which are often combined in LLM pretraining. Whichever pattern a job uses, the underlying constraint is the same: This communication is latency-sensitive, and where those workers land in the cluster determines how much of that latency the job absorbs.
The bandwidth available between workers depends on where they sit relative to each other in the cluster. The following topology domains are common in GPU clusters:
Different workloads have different tolerances for latency. In a pretraining job that comprises a large number of workers, each step blocks until the slowest worker completes its share of that step’s collective communication. Any cross-rack latency here compounds quickly, reducing effective GPU utilization. A small fine-tuning job involves fewer workers exchanging smaller gradients, so the same cross-rack penalty represents a much smaller fraction of each step’s total time.
Without topology awareness, the Kubernetes scheduler places workers on any eligible nodes, regardless of their physical relationships in the cluster. The gang assembles and the training job runs, but communication overhead—rather than computation—consumes a substantial portion of each step. The job completes more slowly and at greater cost than the hardware should require. Crucially, infrastructure metrics don’t reflect this inefficiency because they measure activity, not productivity. Nothing looks wrong from an infrastructure perspective, but training throughput remains low relative to what the hardware should deliver.
Kueue closes part of this gap through topology-aware placement. It groups nodes into domains—block, rack, and host—based on topology labels applied to those nodes (for example, topology.kubernetes.io/rack). Kueue places a job’s pods within a single domain that satisfies its bandwidth requirements. But Kueue doesn’t discover cluster topology on its own. The labels that define each domain have to be applied to nodes first, typically by a cluster administrator or a cloud provider’s topology labeler.
Kueue enforces this placement decision at admission time. It’s an open source Kubernetes-native job queuing system that sits above the kube-scheduler, controlling which jobs reach the scheduler and when. During that admission check, it also verifies that a job’s pods fit within a topology domain. It evaluates each job against quota, priority, and preemption criteria and decides whether to allow the job to proceed. It tracks that decision in a Kueue Workload object, which represents the resource requirements, admission status, and pod sets for a job. Kueue also provides optional life cycle management for admitted jobs, including evicting and requeueing workloads whose pods don’t all reach a ready state within a configured timeout.
When Kueue admits a job to be scheduled (by setting spec.suspend: false, allowing the Job controller to create pods), it writes scheduling constraints into the training job’s pod templates based on the placement mode and topology assignment it selected. Those constraints restrict which nodes the Kubernetes schedu