xz’s liblzma supports multi-threaded compression since 5.2.0 (2014-12-21). Making something as CPU-intensive as compression parallel is quite useful, so Python should have it too.
The MT mode compresses the data in individual chunks, such that the only additional barrier for reproducible output is the chunk size (defaulting to be derived from the preset’s dict size), and that any thread count would compress the same. (xz itself can still compress differently from version to version.)
The entry point to this feature seems to be lzma_stream_encoder_mt. src/xz/coder.c, as in the reference xz program, provides an example for using this interface – complete with sensible values in mt_options. CPU count detection with lzma_cputhreads() is found in src/xz/hardware.c.
Hey I opened an issue and a PR that adds support by adding an optional threads parameter to the lzma module compression APIs.
Feel free to comment and review
