PEP 784: Adding Zstandard to the standard library

[dholth (Daniel Holth)]

Interesting, flag not documented here compression.zstd — Compression compatible with the Zstandard format — Python 3.14.0b2 documentation

In python-zstandard, compression fails if the size is wrong. Or at least if the input size is too big. So decompression problems are avoided.

[emmatyping (Emma Smith)]

@dholth

I’m thrilled about zstd in the standard library!

Yay! Glad to hear it

@dholth

Can you speak towards compression.zstd and setting the decompressed size header in the zstd data? A few years ago tests with python-zstandard showed this header was important to limit memory usage on decompression at higher levels.

That’s an interesting point. Several items in pyzstd were dropped to simplify the implementation. Like other compression modules, the goal of compression.zstd is to provide solutions for common use cases of compression "generate a .tar.zst, “read a format that is zstd compressed”, etc.

For the initial API I tried to be conservative in what was included, both due to time pressure, but also maintainability.

All that being said, I think allowing for setting a pledged size is a reasonable feature request (which unfortunately can only go into 3.15 now).

@dholth

Interesting, flag not documented here compression.zstd — Compression compatible with the Zstandard format — Python 3.14.0b2 documentation

This was excluded from the first documentation pass as I wanted to verify it worked properly with compress(). I’ll be opening a follow up PR to add it today.

Edit: PR is here gh-134906: Document CompressionParameter.content_size_flag by emmatyping · Pull Request #134907 · python/cpython · GitHub

[steve.dower]

@emmatyping

I think allowing for setting a pledged size is a reasonable feature request (which unfortunately can only go into 3.15 now).

I’d vote for this to go into 3.14. It’s far less inconvenient to have a difference between 3.14.0b2->b3 than it is to have a difference permanently between 3.14->3.15.

@hugovk can give official permission.

[emmatyping (Emma Smith)]

@steve.dower

I’d vote for this to go into 3.14. It’s far less inconvenient to have a difference between 3.14.0b2->b3 than it is to have a difference permanently between 3.14->3.15.

@hugovk can give official permission.

I agree, and I’m not opposed to adding it. Since this will eventually be a change regardless of version I propose we take this to a CPython issue: Add set_pledged_input_size to ZstdCompressor · Issue #134938 · python/cpython · GitHub

[hugovk (Hugo van Kemenade)]

@steve.dower

I’d vote for this to go into 3.14. It’s far less inconvenient to have a difference between 3.14.0b2->b3 than it is to have a difference permanently between 3.14->3.15.

@hugovk can give official permission.

This makes sense to include in 3.14, please aim to merge in time for beta 3 (2025-06-17).

[Rogdham]

To increase the adoption of the implementation of Zstandard in the standard library, I created the backports.zstd library (with some help from Emma ). It is a direct backport of the code from CPython for versions 3.9 to 3.13.

So if you want to try out the features from the PEP without installing 3.14, you can! Feel free to contact me directly or open an issue on GitHub if you have any questions.