LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn
LinkedIn reports that moving Spark and Flink shuffle storage to dedicated Apache Celeborn workers cut daily fetch failures from about 750 to 20 and improved application runtime by 14% compared with its previous setup.
The deployment separates intermediate storage from executors, uses a five-node Raft control plane and network-aware replication, and added health-aware placement. LinkedIn attributes further savings to NVMe and Zstandard; these are its production measurements, not general benchmarks.