Spotify’s Random Access Parquet adds point queries to the data lake
Spotify’s Random Access Parquet (RAP) adds an external index over immutable Parquet/Iceberg data, letting online services and AI applications resolve a key to a file and row location, then issue a small ranged read from object storage instead of copying the dataset into an operational database.
The design keeps one lake for analytics, ML, notebooks, agents, and serving while using append-only index fragments, sorted/hash/secondary/covering indexes, and layout locality to reduce metadata and storage operations. Spotify’s scale makes the motivation concrete: petabytes in Bigtable and exabytes in its GCS lake. The trade-off is another serving/index layer and careful data-layout management, but it avoids duplicating every dataset just to support point lookups.