It is possible to write a gzip file with multiple thread. The pigz implementation that Mark Adler made contains comments on how to do so efficiently.
gzip reading is limited to single-thread (although the crc could be checked in a separate thread).
Since CPython’s zlib module allows escaping the GIL when multiple threads are used this could be used to allow the use of multiple cores when compressing gzip files. For decompression, it should allow to escape the main python thread, and do the decompression on a separate core. Both applications will free the main thread to get actual work done in python.
Currently this is possible using the xopen library. But that escapes the GIL by calling a subprocess that calls pigz (or igzip) for reading and writing. It would be much more elegant to have a threads keyword-only argument to gzip.open that allows this behaviour.
It sounds to me that this would be an excellent feature for a 3rd party extension module, but it doesn’t feel like it belongs in the stdlib – it feels too much like an extreme corner case, and likely won’t work on all platforms.
Also keep in mind that even if it is an excellent idea, that doesn’t mean it’s going to be implemented by the core development team – this looks like a big coding project that requires someone really dedicated to making it work.
An analogue for JSON would be a simple stdlib json module and extremely fast, multithreaded, complex, C++ -based third-party packages like pysimdjson. Great idea, but probably not for stdlib.
It sounds to me that this would be an excellent feature for a 3rd party extension module, but it doesn’t feel like it belongs in the stdlib – it feels too much like an extreme corner case,
An analogue for JSON would be a simple stdlib json module and extremely fast, multithreaded, complex, C++ -based third-party packages like pysimdjson. Great idea, but probably not for stdlib.
Thanks for this feedback. In my field (bioinformatics) zlib compressed formats with multiple larger than GB files are pretty common, but that indeed distorts my view on “normal” workloads.
Also keep in mind that even if it is an excellent idea, that doesn’t mean it’s going to be implemented by the core development team – this looks like a big coding project that requires someone really dedicated to making it work.
That would be me. I already maintain python bindings for ISA-L and zlib-ng to allow for faster compression and decompression. My plan of attack would be to implement it in those libraries first and then backport the code to the python stdlib. (All performance improvements I found so far I have backported to CPython.)
I am glad that I pitched it here first though, because not having to worry about stdlib compatibility first gives me more freedom in how I implement it in the extension modules. Thank you both!