# PEP 784: Adding Zstandard to the standard library

**URL:** <https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377>\
**Category:** PEPs\
**Created:** [April 7, 2025, 7:47am UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377 "2025-04-07T07:47:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 7:47am UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/1 "2025-04-07T07:47:28Z")

</div>

Hi!

After spending a number of weekends on the reference implementation, I have finally had time to write up a PEP about adding support for Zstandard compression to the standard library.

# Abstract

[Zstandard](https://facebook.github.io/zstd/) is a widely adopted, mature, and highly efficient compression standard. This PEP proposes adding a new module to the Python standard library containing a Python wrapper around Meta’s `zstd` library, the default implementation. Additionally, to avoid name collisions with packages on PyPI and to present a unified interface to Python users, compression modules in the standard library will be moved under a `compression.*` namespace package.

* * *

The fully rendered PEP [can be read here](https://peps.python.org/pep-0784/).

While this PEP currently targets implementation in 3.14, if the window between acceptance and the beta cutoff is too close, the PEP implementation can be pushed out to 3.15.

---

<div class="post-metadata">

**Author:** ![thomas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/thomas/32/5492_2.png) [@thomas](https://discuss.python.org/u/thomas)\
**Post date:** [April 7, 2025, 10:30am UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/2 "2025-04-07T10:30:14Z")

</div>

Thanks for the PEP, looking forward to zstd in the zstdlib!

! have one small question: the PEP mentions compression being a “namespace package” twice, but doesn’t explicitly say whether that’s an actual PEP 420 namespace package, and if so, why. The comments about “compression” already being a reserved PyPI package name suggests namespace collisions with existing packages is not a driving concern for this.

If the intent _is_ to make it a PEP 420 namespace package, I think we need to be very careful and deliberate with uses of `compression.zstd` in the stdlib, and not move any existing modules to the `compression` (providing shims in `compression` is fine, as long as nothing in the stdlib uses them for existing compression modules). Any module named `compression` would shadow the stdlib namespace package, and any imports of `compression` in the stdlib would fail in confusing and unpredictable ways. The PEP could use some mention of the risks there.

If the intent is _not_ to make `compression` a PEP 420 namespace package, yay! Let’s not! But maybe don’t use the exact wording “namespace package”(perhaps make it “top-level namespace” instead), and maybe mention making it an actual PEP 420 namespace package under rejected ideas.

---

<div class="post-metadata">

**Author:** ![malemburg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/malemburg/32/50_2.png) [@malemburg](https://discuss.python.org/u/malemburg)\
**Post date:** [April 7, 2025, 11:17am UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/3 "2025-04-07T11:17:40Z")

</div>

+1 on adding a new `zstd` module.

> [@emmatyping](#):
>
> Additionally, to avoid name collisions with packages on PyPI and to present a unified interface to Python users, compression modules in the standard library will be moved under a `compression.*` namespace package.

I don’t think these two things should go into the same PEP.

Adding a `zstd` module will cause a lot fewer headaches than introducing a generic `compression` namespace, since the latter is most likely going to conflict with existing modules and packages used in applications out there (not just packages and modules on PyPI).

---

<div class="post-metadata">

**Author:** ![merwok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/merwok/32/29_2.png) [@merwok](https://discuss.python.org/u/merwok)\
**Post date:** [April 7, 2025, 3:19pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/4 "2025-04-07T15:19:57Z")

</div>

I agree! I think the experience of `concurrent.futures` tells us that the stdlib is better as a flat namespace.  
futures are great, but `concurrent` did not work out. threading, asyncio, contextvars, etc happily live as top-level modules. I don’t think much would have been gained by moving or creating them under `concurrent`.  
So I don’t think introducing `compression` (or a better, less generic name) would be good.

---

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 3:34pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/5 "2025-04-07T15:34:24Z")

</div>

> [@thomas](#):
>
> If the intent _is_ to make it a PEP 420 namespace package, I think we need to be very careful and deliberate with uses of `compression.zstd` in the stdlib, and not move any existing modules to the `compression` (providing shims in `compression` is fine, as long as nothing in the stdlib uses them for existing compression modules). Any module named `compression` would shadow the stdlib namespace package, and any imports of `compression` in the stdlib would fail in confusing and unpredictable ways. The PEP could use some mention of the risks there.
> 
> If the intent is _not_ to make `compression` a PEP 420 namespace package, yay! Let’s not! But maybe don’t use the exact wording “namespace package”(perhaps make it “top-level namespace” instead), and maybe mention making it an actual PEP 420 namespace package under rejected ideas.

Thank you for pointing out that confusion. I think `compression` should _not_ be a namespace package, considering that would restrict the flexibility of the stdlib (which is part of the reason of this PEP). I don’t know of any packages that distribute `compression.foo`, so I don’t think that is a consideration either.

---

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 5:02pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/6 "2025-04-07T17:02:42Z")

</div>

> [@malemburg](#):
>
> I don’t think these two things should go into the same PEP.

I think the introduction of Zstandard motivates the creation of the `compression` package, so to me it seems reasonable to keep them in the same PEP. They could be two PEPs that reference each other, but I think that would make the `compression` package PEP hard to understand (and logistically more complicated ). I think as long as the `compression.zstd` module stays the proposed interface, it makes sense to have them stay in the same PEP.

> [@malemburg](#):
>
> Adding a `zstd` module will cause a lot fewer headaches than introducing a generic `compression` namespace, since the latter is most likely going to conflict with existing modules and packages used in applications out there (not just packages and modules on PyPI).

My concern with choosing a name that matches an existing package on PyPI is that it would make migration much more difficult for users of that package. If you are using advanced features of the `zstd` package, you lose access to those features and that package _entirely_ once there is a module in the standard library taking precedence. They could do some importlib hacks I’m sure, but given `zstd` has over 2.5M monthly downloads, I don’t think it makes sense to break them without obvious recourse. If `lz4` is going to be added to the stdlib in the future, is it acceptable to break the users who download it a total of 40 million times a month? For reference, that is twice as popular as Django.

When introducing a `compression` import name, I see one of a few cases where there would be a top level `compression` package or module in existing code (as to my understanding, but if I miss any cases or the below is wrong, someone please correct me):

1. There is a `compression.py` in the same directory as Python is run (implicit front of `sys.path`). This will override the standard library module, but the user can rename this module if that causes issues with other packages.
2. A `compression` module is introduced in `sys.path` some other, dynamic, way, e.g. `sys.path.append`. This is another case where the user has control over which module is imported, and can make any hard decisions.
3. The user ships a private package which has a top level `import compression`. In this case, things break and they _must_ rename.

Furthermore, by introducing a `compression` package, this change/break will only happen once. If we decide to not namespace and introduce `zstd`, and in the future `lz4` and for any future compression library, those will each break users.

I would rather choose to break some smaller subset of unknown users once (introducing `compression`) over breaking a large number of known users multiple times in the future (introducing `zstd`, `lz4`, etc.).

---

<div class="post-metadata">

**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\
**Post date:** [April 7, 2025, 5:41pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/7 "2025-04-07T17:41:30Z")

</div>

> [@emmatyping](#):
>
> My concern with choosing a name that matches an existing package on PyPI is that it would make migration much more difficult for users of that package. If you are using advanced features of the `zstd` package, you lose access to those features and that package _entirely_ once there is a module in the standard library taking precedence.

Only if the `zstd` maintainers are unwilling to migrate _their_ package name, right? If the introduction of the built-in module coincided with a rename, then users _could_ migrate\[1\] but most would be able to switch to the stdlib package with no changes at all.

The trade-off here depends on how many users would want to keep using the third-party package, versus how many would start using the stdlib version. If the former is large enough that you don’t want to disrupt them, that seems like an argument that this shouldn’t be added to stdlib in the first place (because it’s insufficient for most users).

* * *

1. if they care

---

<div class="post-metadata">

**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\
**Post date:** [April 7, 2025, 6:11pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/8 "2025-04-07T18:11:52Z")

</div>

> [@jamestwebber](#):
>
> Only if the `zstd` maintainers are unwilling to migrate _their_ package name, right?

No, because there are cross-version compatibility issues with doing that. Everything is just massively simpler for everyone if existing third party project names are left alone.

> [@jamestwebber](#):
>
> If the former is large enough that you don’t want to disrupt them, that seems like an argument that this shouldn’t be added to stdlib in the first place (because it’s insufficient for most users).

That doesn’t follow, as one rationale for adding things to the standard library is so that they can eventually become an assumed default feature (where different projects will assess when “eventually” has arrived based on the rate at which their userbase migrates to newer versions of Python).

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [April 7, 2025, 6:23pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/9 "2025-04-07T18:23:47Z")

</div>

> [@emmatyping](#):
>
> I think the introduction of Zstandard motivates the creation of the `compression` package, so to me it seems reasonable to keep them in the same PEP.

I have no opinion on whether `zstd` should be under `compression` or kept at the top level, but I’m definitely concerned about renaming the existing compression libraries. That’s the sort of change, being done “to keep things tidy”, that is typically discouraged because it introduces real breakages for little or no practical benefit.

Many 3rd party projects will rely on the existing compression libraries. And they will often support a much wider range of Python versions than the core team does. So for those projects, there’s no point where a “clean” switch is possible - instead, they will need to carry compatibility code for at least some period of time. That code might be as simple as

```python
try:
    from compression import lzma
except ImportError:
    import lzma

```

but that’s still code that is only needed because we’re moving things around. And I think we should acknowledge that imposing that sort of cost on 3rd party projects needs a better justification than simply “it’s tidier”.

> [@emmatyping](#):
>
> I would rather choose to break some smaller subset of unknown users once (introducing `compression`) over breaking a large number of known users multiple times in the future (introducing `zstd`, `lz4`, etc.).

That’s a false dichotomy. It’s not the introduction of `compression` that breaks users. It’s moving existing libraries under it. And conversely, introducing `compression` _without_ moving existing libraries still avoids the future breakage you’re concerned about\[1\] while not breaking users now.

* * *

1. I also feel that you overstate the future breakage, but that’s not relevant to my point

---

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 6:25pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/10 "2025-04-07T18:25:16Z")

</div>

> [@ncoghlan](#):
>
> > [@jamestwebber](#):
> >
> > Only if the `zstd` maintainers are unwilling to migrate _their_ package name, right?
> 
> No, because there are cross-version compatibility issues with doing that. Everything is just massively simpler for everyone if existing third party project names are left alone.

I agree with Alyssa on this. Additionally, package distribution renaming is very difficult to properly communicate and execute without breaking people and having users miss the change. I would expect a number of users simply staying on the last release running into errors unaware of why.

> [@jamestwebber](#):
>
> The trade-off here depends on how many users would want to keep using the third-party package, versus how many would start using the stdlib version. If the former is large enough that you don’t want to disrupt them, that seems like an argument that this shouldn’t be added to stdlib in the first place (because it’s insufficient for most users).

For the initial transition, I expect users to import _both_ the stdlib and third-party implementation, depending on Python version to maintain compatibility. I expect the popularity of the `zstd` package to continue until 3.13 is dropped from most packages.

---

<div class="post-metadata">

**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\
**Post date:** [April 7, 2025, 6:36pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/11 "2025-04-07T18:36:54Z")

</div>

> [@emmatyping](#):
>
> I would expect a number of users simply staying on the last release running into errors unaware of why.

Doesn’t this also apply to moving the compression packages to a namespace, though? Even with a very long deprecation period, it _will_ break people – whereas it is always possible to find an unused name for a new stdlib package.

I also find the rationale in the PEP for hypothetical future `compression.*` packages uncompelling: there has been a recent trend to be hesitant in adding new packages to the stdlib, even in namespaces. Adding `LZ4`, or any other new compression library would face the same hurdles, of which naming is likely to be one of the least challenging.

A

---

<div class="post-metadata">

**Author:** ![malemburg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/malemburg/32/50_2.png) [@malemburg](https://discuss.python.org/u/malemburg)\
**Post date:** [April 7, 2025, 6:49pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/12 "2025-04-07T18:49:23Z")

</div>

> [@emmatyping](#):
>
> you lose access to those features and that package _entirely_ once there is a module in the standard library taking precedence

It’s probably worthwhile asking the maintainers of the `zstd` PyPI module for their thoughts on this. Likewise, the maintainers of the `pyzstd` module should chime in with their thoughts on having their module included in the stdlib.

This is how we usually manage inclusion of PyPI modules or packages in the stdlib. The last time this has happened (IIRC) was for the `toml` module. The compromise was to use the name `tomllib` instead of `toml`. I’m sure we can find a good compromise for the Zstandard implementation as well.

---

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 7:09pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/13 "2025-04-07T19:09:26Z")

</div>

> [@malemburg](#):
>
> It’s probably worthwhile asking the maintainers of the `zstd` PyPI module for their thoughts on this. Likewise, the maintainers of the `pyzstd` module should chime in with their thoughts on having their module included in the stdlib.

I have corresponded with Rogdham, the pyzstd maintainer, a number of times, and they reviewed the PEP draft before it was posted. Rogdham was enthusiastically supportive of the module’s integration and the PEP.

Reaching out to the `zstd` maintainers is a good idea, I will do that some time today.

> [@malemburg](#):
>
> This is how we usually manage inclusion of PyPI modules or packages in the stdlib. The last time this has happened (IIRC) was for the `toml` module. The compromise was to use the name `tomllib` instead of `toml`. I’m sure we can find a good compromise for the Zstandard implementation as well.

My understanding here is that this worked well because the APIs were compatible. If `zstd` in the stdlib exported an API completely different from the `zstd` PyPI package, that surely will lead to compatibility headaches.

---

<div class="post-metadata">

**Author:** ![hugovk](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hugovk/32/14505_2.png) [@hugovk](https://discuss.python.org/u/hugovk)\
**Post date:** [April 7, 2025, 7:20pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/14 "2025-04-07T19:20:56Z")

</div>

> [@emmatyping](#):
>
> My concern with choosing a name that matches an existing package on PyPI is that it would make migration much more difficult for users of that package. If you are using advanced features of the `zstd` package, you lose access to those features and that package _entirely_ once there is a module in the standard library taking precedence.

Previously we’ve renamed the package when it was brought into the stdlib. For example, [PEP 680](https://peps.python.org/pep-0680/) imported the format-reading part of the third-party [tomli](https://pypi.org/project/tomli/) as the stdlib’s [tomlib](https://docs.python.org/3/library/tomllib.html).

If you only need to read TOML files and want maximum compatibility:

```python
if sys.version_info >= (3, 11):
    import tomllib
else:
    import tomli as tomllib

```

But if you want to write TOML (the advanced features), you can still use the third-party tomli directly.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [April 7, 2025, 7:51pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/15 "2025-04-07T19:51:17Z")

</div>

> [@hugovk](#):
>
> Previously we’ve renamed the package when it was brought into the stdlib.

Historically we’ve done this to use a “better” name, or a more consistent name. `tomllib` is an anomaly, where we chose a distinctly _worse_ name (you misspelled it 😉 ) with _less_ consistency (with the other serialization format packages we have).

There’s no way to appeal to precedent on this. We just have to choose the name pragmatically, or come up with another clever idea (such as having `site-packages` shadow the stdlib but make `stdlib.<module>` always bypass it to the stdlib, or a `compression` not-quite-namespace package).

---

<div class="post-metadata">

**Author:** ![gpshead](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/gpshead/32/54_2.png) [@gpshead](https://discuss.python.org/u/gpshead)\
**Post date:** [April 7, 2025, 9:17pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/16 "2025-04-07T21:17:44Z")

</div>

> [@malemburg](#):
>
> I don’t think these two things should go into the same PEP.

I disagree. We’re not likely to add `zstd` without it going in a proper submodule because our top level module name space is already polluted and claimed by PyPI packages and we’d end up with an unfriendly module name. Which is why I suggested do the same thing I did for `hashlib.` and make a similar concept for our compression batteries. Thus `compression.`.

It seemed like a pretty obvious choice to me.

Namespaces (as in `.`s and submodules, I don’t care about PEP 420 “namespace packages” in this context) are a good thing.

---

<div class="post-metadata">

**Author:** ![gpshead](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/gpshead/32/54_2.png) [@gpshead](https://discuss.python.org/u/gpshead)\
**Post date:** [April 7, 2025, 9:24pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/17 "2025-04-07T21:24:18Z")

</div>

> [@pf\_moore](#):
>
> I have no opinion on whether `zstd` should be under `compression` or kept at the top level, but I’m definitely concerned about renaming the existing compression libraries. That’s the sort of change, being done “to keep things tidy”, that is typically discouraged because it introduces real breakages for little or no practical benefit.

The PEP gives a 10 year timeline for this. That is plenty of time to back out if future maintainers decide they’re rather keep our history of just squatting in the top level package namespace for arbitrary things instead of grouping related functionality under a topical name as we’ve successfully done with hashlib.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [April 7, 2025, 9:34pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/18 "2025-04-07T21:34:35Z")

</div>

> [@gpshead](#):
>
> The PEP gives a 10 year timeline for this.

Sorry - I misread the relevant section of the PEP (“standard deprecation timeline” distracted me 🙁). Accepted, that does seem like a reasonable timeline for 3rd party code, even code that aims for extreme backward compatibility.

I’m no longer against the move, although I’d still describe myself as neutral rather than actively in favour.

---

<div class="post-metadata">

**Author:** ![ngoldbaum](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ngoldbaum/32/12276_2.png) [@ngoldbaum](https://discuss.python.org/u/ngoldbaum)\
**Post date:** [April 7, 2025, 10:07pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/19 "2025-04-07T22:07:10Z")

</div>

For what it’s worth, including zstandard bindings in the standard library would also help out with free-threaded support.

That said, a cursory look indicates that pyzstd isn’t testing the free-threaded build. Might there be any thread safety issues due to assumptions around the GIL?

As far as I understand, it’s not safe to share zstd compression or decompression handles between threads.

Of course all of that can be fixed, but it would be good IMO if it were already handled before the code is added to CPython so it doesn’t need to happen later.

---

<div class="post-metadata">

**Author:** ![emmatyping](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/emmatyping/32/36516_2.png) [@emmatyping](https://discuss.python.org/u/emmatyping)\
**Post date:** [April 7, 2025, 10:42pm UTC](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/20 "2025-04-07T22:42:42Z")

</div>

> [@pf\_moore](#):
>
> > [@gpshead](#):
> >
> > The PEP gives a 10 year timeline for this.
> 
> Sorry - I misread the relevant section of the PEP (“standard deprecation timeline” distracted me 🙁). Accepted, that does seem like a reasonable timeline for 3rd party code, even code that aims for extreme backward compatibility.

Ah sorry if that was confusing! I’ll update it to remove “standard deprecation timeline” to emphasize that starts once the last Python without `compression` leaves support, in 5 years, and that means the modules will co-exist for a total of 10 years.

[Next page](https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377.md?page=2)