Kubernetes was designed to schedule containers on CPU and memory. The model was: you have a cluster of nodes, you have workloads that need compute, Kubernetes figures out placement and keeps things running.
AI inference changes almost every assumption in that model.
GPU scheduling is fundamentally different from CPU scheduling. A model serving pod does not just need a GPU. It needs a specific amount of VRAM, it may need multiple GPUs with NVLink connectivity, it needs to know whether it is sharing a GPU with another workload or getting exclusive access, and the latency of cold-starting a model is measured in minutes, not milliseconds.
None of this fits cleanly into Kubernetes resource primitives.
The Specific Gaps
GPU fractioning. A single H100 has enough memory to run several small models simultaneously. Kubernetes does not natively support fractional GPU allocation. You can request nvidia.com/gpu: 1 and get a whole GPU, or not get scheduled at all. Tools like NVIDIA's MIG and the Time-Slicing scheduler exist, but they require significant configuration and add operational complexity.
Model loading latency. When a Kubernetes node starts a new inference pod, the model weights have to be loaded into GPU memory. For large models, this takes minutes. Kubernetes health checks will kill the pod if it does not pass readiness checks quickly enough. Getting startup probe timing right for inference workloads requires knowing model sizes and load times for every model in your fleet.
VRAM as a first-class resource. The scheduler does not understand VRAM. It understands CPU millicores and RAM bytes. Scheduling an inference workload requires either pre-calculating VRAM requirements and encoding them as custom resource requests, or accepting that pods will get scheduled onto nodes where they will OOM-kill at startup.
Topology awareness. A distributed inference job that spans multiple GPUs needs those GPUs to be on the same node or connected with high-bandwidth interconnects. Standard Kubernetes scheduling does not understand NVLink or InfiniBand topology.
What the Community Is Building
The Kubernetes AI Working Group is actively developing extensions. KWOK provides simulation environments for testing AI scheduling policies without burning real GPU budget. The Device Plugin API is being extended to surface more hardware topology information to the scheduler.
Projects like Kueue (batch job queuing), LWS (LeaderWorkerSet for multi-GPU distributed jobs), and KEDA (event-driven autoscaling based on inference queue depth) are filling gaps that core Kubernetes does not cover.
Should You Use Kubernetes for AI Workloads Today?
If you are already a Kubernetes shop, yes, with caveats. The ecosystem tooling is mature enough to make it work, but expect to invest in configuration and operational work that CPU workloads do not require.
If you are starting fresh and your primary workload is inference serving, purpose-built inference platforms like Ray Serve, BentoML, or the managed options from your cloud provider may have a lower operational burden.
The honest answer is that Kubernetes will adapt. The question is whether it adapts fast enough to be the obvious choice for AI infrastructure by the time your organization needs to make that decision.