Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs
Google has open-sourced TPU Sync, a C++ library for moving LLM KV caches and model weights among TPU memory, host memory, storage, and the network.
It targets disaggregated prefill/decode serving, prefix caching, and synchronization between reinforcement-learning trainers and samplers. Google reports large time-to-first-token improvements in selected workloads; those figures are workload-specific. Python integrations target JAX and PyTorch, including vLLM and SGLang backends.
The project warns that its APIs are changing rapidly, PyPI packages are development builds, and its RL weight synchronization is experimental.