The Missing Cores: A 24-core machine is idle inside Polars?!

A hands-on Polars profiling report finds several different scaling bottlenecks: shared regex scratch-buffer contention, too few large groups to keep workers busy, and per-group object overhead for tiny groups. On the author’s 24-core test machine, the proposed fixes cut regex-heavy queries by roughly 57–63%, a three-group sort by 48%, and row numbering across 45 million groups by 74%. The post uses paired runs, control queries, and output checks; results are specific to its machine and workloads. Two of the cited changes were still under review when the article was written.

The r/rust discussion (38 comments) focused heavily on whether the writing was AI-assisted; the author said he used an LLM to group his profiling notes, then fact-checked and rewrote them. On the technical side, commenters raised high-core-count and NUMA scaling concerns, and an early regex-change concern was addressed in a follow-up PR that Polars merged on September 22.

• r/rust discussion
• Merged regex follow-up