One Pass to Parse Them All: Fused Parallel CSV Processing
The Rust csveee parser fuses parallel CSV parsing with record processing: workers speculatively parse independent chunks using the expected record shape, then a merge step validates boundaries. Its authors report up to 181 GB/s on a 128-core AMD EPYC 9745 server, close to their measured 240 GB/s memory bandwidth. Those headline results count records; complex type conversion cut throughput substantially, and the results are hardware- and workload-specific.
The design also uses SIMD, a ring-buffered reader, and zero-copy record construction. Reddit commenters asked for comparisons with Polars and DataFusion; the author posted a follow-up table, while noting DataFusion’s optimized record-count path complicates a fair comparison. Another thread asked whether stable portable SIMD would suffice; the author said parallelization drives the main gains and is looking into stable SIMD crates.