# Restricting "open ended" releases on PyPI? **URL:** \ **Category:** Packaging\ **Tags:** PyPI\ **Created:** [January 18, 2024, 10:50pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566 "2024-01-18T22:50:28Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![woodruffw](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/woodruffw/32/37228_2.png) [@woodruffw](https://discuss.python.org/u/woodruffw)\ **Post date:** [January 18, 2024, 10:50pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/1 "2024-01-18T22:50:28Z")
CCing @sethmlarson, since this came up as a (small) roadbump in [PEP 740](https://github.com/python/peps/pull/3618) (which is still in draft). I’m filing this as a discussion thread, since there may be consequences to this idea that I haven’t thought through. ## Context PyPI’s data model for packages looks like this: - Projects have names (e.g. `requests`) - Each project has zero or more _releases_, corresponding to versions (e.g. `requests==2.31.0`) - Each _release_ has one or more _files_, corresponding to individual installable “distributions” of that release (e.g. `requests-2.31.0.tar.gz` for the sdist, `requests-2.31.0-py3-none-any.whl` for the wheel) - Many projects have \>2 _files_ per release, corresponding to wheel builds for various Python versions, architectures, host OSes, etc. A [recent version of cryptography](https://pypi.org/project/cryptography/41.0.7/#files) has 23 release files (1 sdist, 22 wheels). Files within a release can “overlap” in terms of platform compatibility, and the package installer (e.g. `pip`) is responsible for selecting the “best” release file. I believe this “best” selection is not explicitly documented anywhere, but is probably something like “pick the most specific wheel for the current host, followed by any compatible wheel, followed by the source distribution.” ## The Problem The data model above is fine, in terms of ensuring that installers can select a compatible non-source distribution. However, PyPI’s implementation of it has a quirk: every release is _open ended_, in the sense that a maintainer of project `spam` can add _new_ files to `spam==1.2.3` long after the first (or last) previous file was uploaded. This has a few arguably negative effects: - Maintainer confusion on uploads: a maintainer looking to publish a new version of an old package may forget to change the version string, and instead silently upload new wheels to an old package version. This is both hard to debug (since the index reports no error) and can cause downstream confusion, since the new distributions are not necessarily compatible with the old ones (and worse, may be silently prefered during installation due to their greater specificity). - Unnecessary pessimization during resolution: an optimal backtracking dependency resolver needs to pessimistically assume that the set of files for each release under consideration may change at any point, potentially causing a cascade of recomputed dependencies if a release gains an additional file in the middle of resolution. I’m not sure if current major installers _attempt_ to be “optimal” in this way (it’s arguably not worth it!), but having a guarantee that the release’s files will not change _would_ enable further optimizations/caching here, I think. - Security: in its current form, PyPI is _uniquely_ vulnerable to a form of dependency compromise that other packaging indices are not: an attacker who wants to compromise users of `spam` without signaling their presence via a new release can instead create new more-specific distributions in _current_ releases, relying on dependency resolvers to faithfully select the most specific wheel for use. This even affects users who pin (e.g. `spam==1.2.3`) since version pins have no effect on wheel resolution, although it doesn’t affect users with pre-existing hash pins. ## Proposed solution My proposal: PyPI should put a cap on the “open ended” nature of each release. What that looks like needs further discussion, but here are some initial ideas to get the conversation started: 1. Prevent additional file uploads to a release, beginning `X` hours after the _first_ file upload. This effectively gives packagers `X` hours to upload all files to a release, after which the release is “frozen” and can no longer be modified (in terms of attached files). The value of `X` would need to be decided. 2. Prevent additional file uploads to a release, beginning `X` hours after the _last_ file upload. This is like (1), except that it keeps the release “open” potentially indefinitely, so long as the uploader continues to upload files within the window\[1\]. The primary advantage of (2) over (1) is that it allows a _little_ more flexibility in the event of long-running publish jobs or release mishaps. 3. Prevent additional file uploads to a release, as soon as a new stable (i.e. non-beta/rc/etc) release is made. In other words: once `spam==1.2.4` is released, `spam==1.2.3` is considered “frozen”. 4. Some combination of (1)+(3) or (2)+(3)? I’m curious to hear what people think of this idea! In particular, I’d like to hear about legitimate use cases for “open ended” releases, and whether this is something anybody in the community is currently relying on. * * * 1. This is not _technically_ indefinite, since the user would eventually run out of valid wheel tag combinations. But it would be much longer than the hard `X` hour window, if a user is determined to keep a release open. ---
**Author:** ![alex\_Gaynor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/alex_gaynor/32/17_2.png) [@alex\_Gaynor](https://discuss.python.org/u/alex_Gaynor)\ **Post date:** [January 18, 2024, 11:01pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/2 "2024-01-18T23:01:57Z")
Hi Will, A pretty frequent request is the ability to “stage” uploads. Right now as soon as I upload my first wheel, that’s the latest version and everyone will try to download that version, even though it’ll take at least a few seconds to upload the rest of the artifacts. It seems to me that solving that problem is a great way to solve the open ended problem. If, instead of just uploading things one at a time that immediately went live, the workflow was something like: - Create “staged” release - Upload all artifacts to that release - Publish that release Then it’d be straightforward and obvious to say “no new artifacts after that”. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [January 18, 2024, 11:03pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/3 "2024-01-18T23:03:47Z")
At one point there was a discussion about “draft” releases, and as I recall one of the ideas was that any release would be in draft (essentially yanked-by-default) until it was “finished” by some explicit action (another API call). That would be a neat way to resolve the _feature_ side of this, as you could then freeze file uploads after the finishing step. You could also just allow an API call to lock the release, which is extensible to the full “draft” feature if that is ever agreed upon. In terms of doing things automatically, most automated build/publish processes are going to do all the builds, then some basic testing, and then publish all the files together, so those could easily handle quite a short window after the first publish. However, I’ve previously advocated/suggested that distributing a manual build process (like we do with CPython itself, where one person tags the sources and others produce the binaries) should be encouraged for where a maintenance team doesn’t feel confident covering all platforms. I’m not sure how much this is actually happening, but if I were doing it I’d want a 48 hour window at least. That feels long, and an opt-out option feels expensive. I think (3) is ruled out by bugfix releases. People need to release `1.1.2` and `1.2.3` simultaneously, and you can’t consider `1.1.2` locked immediately just because there’s already a `1.2.2`. The same window applies as above, I believe, so it doesn’t really gain anything. What should _definitely_ be available though is the publish date/time of each file, and I think it’s reasonable to highlight (either in PyPI directly, or dedicated scorecards/tools) when there’s a significant discrepancy between release times. It’s not necessarily _bad_, but it’s worth manually checking whether it’s expected. ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [January 18, 2024, 11:26pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/4 "2024-01-18T23:26:56Z")
There are significant upsides to this proposal for security as well as caching, so I like the idea in principle. I’d like to point out though that it completely removes the use of build numbers, and using those to deal with an issue in a single wheel - see [What to do when you botch a release on PyPI](https://snarky.ca/what-to-do-when-you-botch-a-release-on-pypi/). Since you also can’t delete individual wheels, all that’s left is yanking or deleting the whole release. ---
**Author:** ![davidism](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/davidism/32/3295_2.png) [@davidism](https://discuss.python.org/u/davidism)\ **Post date:** [January 18, 2024, 11:31pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/5 "2024-01-18T23:31:54Z")
MarkupSafe (and probably other projects with platform wheels) currently takes advantage of this property. When a new version of Python is released, we can start a build for the wheels for that version. The publishing workflow is the same as the original wheels, using trusted publishing to upload to PyPI. In this way, we can add wheel support for the currently released version without having to create an entire new release. MarkupSafe doesn’t change much anymore, so an entire new version just to add a new Python version would just cause unneeded bumping and downloads for all other users. ---
**Author:** ![oscarbenjamin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/oscarbenjamin/32/1209_2.png) [@oscarbenjamin](https://discuss.python.org/u/oscarbenjamin)\ **Post date:** [January 19, 2024, 12:39am UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/6 "2024-01-19T00:39:28Z")
Whatever happened ti the draft releases? I remember seeing this announcement and expecting that it might happen soon: > [@PEP 694: Upload 2.0 API for Python Package Repositories](https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879): > > Riffing off the work done in PEP 691, I’ve gone ahead and used a similar pattern to write a new upload API for PyPI, this time documented as a standard PEP rather than implementation defined like the existing upload API. The new API is a bit more complicated (it would be pretty hard to get less complicated than the old API), but with it we unlock a host of new features: We move to an asynchronous API, that allows servers to respond quickly to requests, and enable any sort of extended processi… ---
**Author:** ![sethmlarson](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sethmlarson/32/1644_2.png) [@sethmlarson](https://discuss.python.org/u/sethmlarson)\ **Post date:** [January 19, 2024, 3:07pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/7 "2024-01-19T15:07:28Z")
Conceptually I’m in favor of moving in this direction for PyPI, the “every release is open-ended” is a common gotcha for folks designing security systems and policy since we’re one of the only ecosystems (if not only) that has this property. Draft releases would be the best solution since it offers benefits alongside the removal of capabilities. > [@rgommers](#): > > I’d like to point out though that it completely removes the use of build numbers, and using those to deal with an issue in a single wheel Relatedly, I recently [added a warning about interoperability problems](https://github.com/pypa/packaging.python.org/pull/1277) stemming from wheel build numbers and not being able to reference them. > [@davidism](#): > > When a new version of Python is released, we can start a build for the wheels for that version. IMO this is the use-case that’s most served by PyPI having open-ended releases that doesn’t have a replacement in “draft” releases or release windows expiring. I’m less convinced by the build numbers or “fixing a subset of wheels” use-cases, if something is actively broken then a new version seems like a good idea? ---
**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\ **Post date:** [January 19, 2024, 3:22pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/8 "2024-01-19T15:22:32Z")
> [@sethmlarson](#): > > Conceptually I’m in favor of moving in this direction for PyPI, the “every release is open-ended” is a common gotcha for folks designing security systems and policy since we’re one of the only ecosystems (if not only) that has this property. Are we? Do the other ecosystems explicitly disallow it, or is that just a consequence of them not supporting binary artifacts? ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [January 19, 2024, 3:27pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/9 "2024-01-19T15:27:37Z")
> [@woodruffw](#): > > Security: in its current form, PyPI is _uniquely_ vulnerable to a form of dependency compromise that other packaging indices are not: > [@sethmlarson](#): > > since we’re one of the only ecosystems (if not only) that has this property. This doesn’t seem true - pretty much any ecosystem that deals with distributing and installing binaries has the option of updating builds. Linux distros, Conda-forge, Nix, Spack, etc. all work this way. Perhaps you’re only thinking of source releases on language-specific packaging indexes? ---
**Author:** ![woodruffw](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/woodruffw/32/37228_2.png) [@woodruffw](https://discuss.python.org/u/woodruffw)\ **Post date:** [January 19, 2024, 3:33pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/10 "2024-01-19T15:33:11Z")
> [@davidism](#): > > MarkupSafe (and probably other projects with platform wheels) currently takes advantage of this property. When a new version of Python is released, we can start a build for the wheels for that version. The publishing workflow is the same as the original wheels, using trusted publishing to upload to PyPI. In this way, we can add wheel support for the currently released version without having to create an entire new release. MarkupSafe doesn’t change much anymore, so an entire new version just to add a new Python version would just cause unneeded bumping and downloads for all other users. Thanks, this is helpful! I suppose the two follow-on questions here are: 1. Do you think doing `abi3` builds would help here? I see that MarkupSafe has a bunch of individual per-CPython-version wheel builds, which should be collapsible into singular `abi3` wheels. That wouldn’t completely eliminate your matrix, but I think it’d avoid the need to push new files at all when new Python versions are released. 2. In your estimation as a package maintainer, would you consider this an acceptable tradeoff to make? In other words: if PyPI were to (not that it necessarily will) go down this route, do you think your project would be the target of ire from users who receive the occasional spurious bump (which should be even more spurious with `abi3`)? > [@rgommers](#): > > This doesn’t seem true - pretty much any ecosystem that deals with distributing and installing binaries has the option of updating builds. Linux distros, Conda-forge, Nix, Spack, etc. all work this way. Perhaps you’re only thinking of source releases on language-specific packaging indexes? Yep, I was thinking of language-specific packaging indices in that original comment. However, for more general indices: do those indices support new _variants_ of the same release in the same way that PyPI does? My understanding is that many non-language indices support some kind of rebuild + reupload, but that the rebuild _overwrites_ the existing artifact for a specific target. This is distinct from the dependency compromise vector I originally mentioned, which involves both artifacts still being present but the dependency resolver selecting the newer one because Python packaging allows for multiple compatible distributions. ---
**Author:** ![jeanas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jeanas/32/8115_2.png) [@jeanas](https://discuss.python.org/u/jeanas)\ **Post date:** [January 19, 2024, 3:39pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/11 "2024-01-19T15:39:10Z")
> [@woodruffw](#): > > Maintainer confusion on uploads: a maintainer looking to publish a new version of an old package may forget to change the version string, and instead silently upload new wheels to an old package version. This is both hard to debug (since the index reports no error) and can cause downstream confusion, since the new distributions are not necessarily compatible with the old ones (and worse, may be silently prefered during installation due to their greater specificity). Couldn’t this be solved easily by an extra prompt in twine: ```python Note: you are uploading a new distribution for an existing release. Is this what you want? [y/n] ``` whenever the user tries to upload a new distribution for a version that already has distributions which were all uploaded more than (say) a day ago? IMHO, this doesn’t need outright forbidding such uploads. > [@woodruffw](#): > > - Security: in its current form, PyPI is _uniquely_ vulnerable to a form of dependency compromise that other packaging indices are not: an attacker who wants to compromise users of `spam` without signaling their presence via a new release can instead create new more-specific distributions in _current_ releases, relying on dependency resolvers to faithfully select the most specific wheel for use. This even affects users who pin (e.g. `spam==1.2.3`) since version pins have no effect on wheel resolution, although it doesn’t affect users with pre-existing hash pins. If security is a concern, isn’t it necessary to check hashes anyway? To me, this sounds like the real solution here is to pick up the work on standardizing a lock file format. > [@sethmlarson](#): > > I’m less convinced by the build numbers or “fixing a subset of wheels” use-cases, if something is actively broken then a new version seems like a good idea? It can be useful when you repackage software that you don’t have control upon. For example, Ninja is [redistributed](https://pypi.org/project/ninja/) on PyPI by the scikit-build folks. The PyPI package is not maintained by the upstream Ninja authors. In such cases, having to ask for a new release upstream if you botch something is not great. ---
**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\ **Post date:** [January 19, 2024, 3:42pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/12 "2024-01-19T15:42:54Z")
> [@woodruffw](#): > > However, for more general indices: do those indices support new _variants_ of the same release in the same way that PyPI does? My understanding is that many non-language indices support some kind of rebuild + reupload, but that the rebuild _overwrites_ the existing artifact for a specific target. This is distinct from the dependency compromise vector I originally mentioned, which involves both artifacts still being present but the dependency resolver selecting the newer one because Python packaging allows for multiple compatible distributions. Wouldn’t overwriting the existing artifact for the specific target have the same compromise vector? A new artifact is uploaded and is selected, either because it’s been overwritten or because we’ve allowed multiple compatible distributions. ---
**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\ **Post date:** [January 19, 2024, 3:47pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/13 "2024-01-19T15:47:07Z")
I guess I should be explicit. I think that this proposal does close a real capability that could be used to attack people in a way that is, by default, pretty “quiet”. I also think that capability can, at least in theory, be used for useful and positive reasons. I don’t think that any real decision one way could be made here unless we get some real numbers behind how often people actually use that capability for useful and positive reasons, and even then we should consider if there are other mechanisms we can put into place to mitigate without removing that capability \[1\]. * * * 1. Lock files (with hashes) do solve this, but unless we get to a place where they are emitted and used by default, that provides a much more limited impact than disallowing them does. That’s not to say that is a bad solution, that’s just one of the trade offs of that solution. ---
**Author:** ![woodruffw](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/woodruffw/32/37228_2.png) [@woodruffw](https://discuss.python.org/u/woodruffw)\ **Post date:** [January 19, 2024, 3:49pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/14 "2024-01-19T15:49:07Z")
> [@dstufft](#): > > Wouldn’t overwriting the existing artifact for the specific target have the same compromise vector? A new artifact is uploaded and is selected, either because it’s been overwritten or because we’ve allowed multiple compatible distributions. Similar, but not identical: one is arguably more surreptitious than the other, since the attacker in PyPI’s case can pinpoint specific targets to deploy a compromised file to, versus having to risk general discovery. (This is arguably a pretty thin hair to split. But I think it’s worth pointing out as part of how PyPI’s model for names/releases/files is subtly different!) > [@dstufft](#): > > I don’t think that any real decision one way could be made here unless we get some real numbers behind how often people actually use that capability for useful and positive reasons, and even then we should consider if there are other mechanisms we can put into place to mitigate without removing that capability Fully agreed! I don’t think an actionable PEP can come out of this discussion until we have concrete numbers (which I’ll work on). ---
**Author:** ![ntessore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ntessore/32/10786_2.png) [@ntessore](https://discuss.python.org/u/ntessore)\ **Post date:** [January 19, 2024, 3:58pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/15 "2024-01-19T15:58:37Z")
> [@jeanas](#): > > If security is a concern, isn’t it necessary to check hashes anyway? To me, this sounds like the real solution here is to pick up the work on standardizing a lock file format. I think so, since once can always upload a new release with any number of `.0` appended to override a pin: ```py >>> Version("1.2.3.0.0.0") in SpecifierSet("==1.2.3") True ``` Even though the OP is more worried about an attacker sneakily inserting their package without creating a new release, the only way to make sure you get what you specify is a hash. Edit: Or maybe this is rubbish and PyPI also considers these versions equal? I thought we fixed a release like this in the past, but now I am not sure if I’m not misremembering the situation. ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [January 19, 2024, 4:17pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/16 "2024-01-19T16:17:03Z")
> [@woodruffw](#): > > However, for more general indices: do those indices support new _variants_ of the same release in the same way that PyPI does? My understanding is that many non-language indices support some kind of rebuild + reupload, but that the rebuild _overwrites_ the existing artifact for a specific target Yes, some do (not sure how many exactly, but that’s also not that relevant). Here’s an example for conda-forge (output of `mamba search scipy`): ```python scipy 1.11.3 py39h89ddb1b_0 conda-forge scipy 1.11.3 py39h89ddb1b_1 conda-forge ``` You see it has a version number and a build number - in this case containing info on the exact Python interpreter built against, plus an increasing `0, 1, ...` build number that should be bumped when rebuilding the same version. What exactly changes in a rebuild is flexible. You may add new architectures for example, e.g. via changing skip conditions in the recipe: ```yaml build: number: 0 skip: true # [py<39] skip: true # [(aarch64 or ppc64le) and python_impl == "pypy"] ``` > [@woodruffw](#): > > This is distinct from the dependency compromise vector I originally mentioned, which involves both artifacts still being present but the dependency resolver selecting the newer one because Python packaging allows for multiple compatible distributions. It’s very similar though, both may silently change the result one may get from an install command. I wouldn’t worry about the differences here. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [January 19, 2024, 4:17pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/17 "2024-01-19T16:17:32Z")
To be clear - is the vulnerability here in the situation where a malicious party has managed to gain the ability to upload to PyPI a new artifact for an existing project and release? Because if that’s the case, surely they already have sufficient power to do all sorts of damaging things, and this one is probably the least of our worries. I’m unclear why this attack vector is considered significant, when (for example) we’re still awaiting an implementation of [PEP 708](https://peps.python.org/pep-0708/), to address dependency confusion attacks (which it seems to me are a much more serious and realistic threat). I don’t want to look like I’m arguing that because we haven’t fixed one problem, we should ignore all others. But on the other hand, if we’re trying to make progress with limited resources, maybe we’d be better focusing on existing work, and giving the PyPI maintainers a chance to keep up? ---
**Author:** ![woodruffw](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/woodruffw/32/37228_2.png) [@woodruffw](https://discuss.python.org/u/woodruffw)\ **Post date:** [January 19, 2024, 4:43pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/18 "2024-01-19T16:43:28Z")
> [@rgommers](#): > > It’s very similar though, both may silently change the result one may get from an install command. I wouldn’t worry about the differences here. I think there’s still an important distinction here: a build number is part of an explicit ordering during dependency resolution. “Post-releases” play a similar role to build numbers in PEP 440, and have an [explicit ordering](https://packaging.python.org/en/latest/specifications/version-specifiers/#post-releases). In contrast, there is no explicit ordering between compatible release distributions. Package installers generally try to select the “most” specific distribution, but even specificity is not super well defined (e.g., it’s unclear whether a wheel with a single tag set “should” be ordered over a different wheel with a compressed tag set that contains the former). This is arguably very pedantic. But I think these kinds of quirks are worth enumerating, even if we ultimately don’t consider them sufficient justification for making changes to PyPI’s release behavior 🙂 > [@pf\_moore](#): > > I’m unclear why this attack vector is considered significant, when (for example) we’re still awaiting an implementation of [PEP 708](https://peps.python.org/pep-0708/), to address dependency confusion attacks (which it seems to me are a much more serious and realistic threat). I think I’ve caused a lot of confusion by including a security point in my original post 😅. To make things explicit: I **do not** think this is a significant attack vector, especially compared to other kinds of dependency attacks. A summarized version of my position is this: the fact that PyPI releases are “forever open” feels like a quirk to me (one that the community has evolved practices around!), rather than something that was originally intended. The proximate reason for that quirk is a limitation in PyPI’s “legacy” upload API around single file uploads, not an explicit design decision to keep releases open forever. This ends up having weird consequences for Python packaging’s data model and ordering logic, a _tiny_ part of which might have security consequences. But 99% of the “itch” here is it just being a bit quirky. I don’t think fixing this is a significant priority, and I’m certainly not volunteering the PyPI maintainers (or myself) to do any immediate work on this here – there’s plenty of otherr stuff that needs doing 🙂 ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [January 19, 2024, 5:36pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/19 "2024-01-19T17:36:11Z")
> [@woodruffw](#): > > Package installers generally try to select the “most” specific distribution, but even specificity is not super well defined (e.g., it’s unclear whether a wheel with a single tag set “should” be ordered over a different wheel with a compressed tag set that contains the former). This is a fair point. The wheel specification is unfortunately vague in a number of areas. It was created at a time when the stakes were a lot lower, and the level of precision that’s now necessary wasn’t the norm. A proposal to tighten up this area would be welcome. > [@woodruffw](#): > > A summarized version of my position is this: the fact that PyPI releases are “forever open” feels like a quirk to me (one that the community has evolved practices around!), rather than something that was originally intended. Again, I think it’s more that it was a design that made sense when PyPI was developed (originally, PyPI didn’t even do the hosting - it was literally just an index - and it was almost entirely a source-only repository). Not so much a “quirk” as a design choice that was made under different constraints. And now, while we could change that design, we need to accept that backward compatibility concerns make it harder than it would be if we designed something from scratch. It’s a fair point to make, but it’s certainly not something I can get worked up about, personally. ---
**Author:** ![Kwpolska](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kwpolska/32/10542_2.png) [@Kwpolska](https://discuss.python.org/u/Kwpolska)\ **Post date:** [January 19, 2024, 5:36pm UTC](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566/20 "2024-01-19T17:36:28Z")
> [@woodruffw](#): > > - Maintainer confusion on uploads: a maintainer looking to publish a new version of an old package may forget to change the version string, and instead silently upload new wheels to an old package version. This is both hard to debug (since the index reports no error) and can cause downstream confusion, since the new distributions are not necessarily compatible with the old ones (and worse, may be silently prefered during installation due to their greater specificity). Do you have any examples of this happening in real life? This seems unlikely to happen, especially if the maintainer prepared wheels for the same platforms as for the previous version (as it seems unlikely that there would be no common OS and Python versions between two releases). > [@woodruffw](#): > > - Unnecessary pessimization during resolution: an optimal backtracking dependency resolver needs to pessimistically assume that the set of files for each release under consideration may change at any point, potentially causing a cascade of recomputed dependencies if a release gains an additional file in the middle of resolution. I’m not sure if current major installers _attempt_ to be “optimal” in this way (it’s arguably not worth it!), but having a guarantee that the release’s files will not change _would_ enable further optimizations/caching here, I think. Using different data to resolve the same package in different places sounds more likely to cause unresolvable dependencies. If the resolver were to fetch details of a package X multiple times, what if the dependency specification was open-ended (no maximum/specific version) and the maintainer of X uploaded a new release while dependencies were being resolved? [Next page](https://discuss.python.org/t/restricting-open-ended-releases-on-pypi/43566.md?page=2)