🛠️Netflix Hits 99.95% GPU Node Uptime With Kubernetes-Based Reliability Platform

Your GPU training jobs can stop surviving hardware failures

Netflix just showed how to keep giant GPU training jobs alive when hardware dies. Its platform detects failures, pulls bad capacity out automatically, and lets jobs recover from machine loss instead of starting over. The problem is brutal for big AI jobs. A training run can span hundreds of GPUs for weeks, and one failed GPU, flaky network adapter, or slow machine can stall the whole thing. A normal cloud health check only tells you the VM is up and reachable. It won't catch a dying GPU. If you run distributed training on Kubernetes, this is the playbook for closing that gap. The numbers: Netflix reports more than 99.95% GPU-node availability and job success climbing from roughly 98% to above 99.5%. The stack uses NVIDIA DCGM for GPU errors, memory, interconnect, and thermal checks, with kernel log fallbacks when DCGM fails. Checks are built to fail closed, integrated with Karpenter node auto-repair, and governed by policies on failure tolerance and maximum cluster disruption. Netflix is also using MIG to share GPUs for smaller inference workloads and moving toward Kubernetes DRA for topology-aware placement.