Post

Netflix Treats GPU Reliability as a Distributed Systems Problem

Netflix uses layered GPU and network health checks, automated remediation and elastic checkpoint recovery to keep distributed training jobs running through hardware failures.

Its Kubernetes node agents fail closed on diagnostic timeouts, and controllers escalate from blocking new work to draining, resetting, rebooting or replacing unhealthy capacity. Netflix reports over 99.95% GPU-node availability and job success improving from roughly 98% to over 99.5%; these are its own operational figures, not a controlled benchmark.