It should be noted that the client installer must also respect the Yank flag, the behavior you describe requires at least Pip 22.0 to work in all situations, I am unsure how other tools (e.g. Poetry) respect Yanking.
While I get that itās frustrating and should have been yanked instead of deleted, the reasoning people are presenting here are a bit confusing. One way or another it sounds like this was going to stop working. Why is it an issue with PyPI that you didnāt see warnings, you wouldnāt have seen them if the package remained and the API stopped either.
As I understand it, if it had been yanked (and potentially earlier in the deprecation cycle if needed to keep the API working, if it did in fact break compatāwhich is not clear to me that it did), is that it would have been just as visible as a removal (unlike a deprecation message), but:
- Allowed for at least a temporary, drop-in opt-out during the transition
- Provided a much clearer, less confusing/alarming and more specific yank message (vs. a sudden generic indication that the package name was not found, which is confusing until users go look it up)
- Avoided a malicious third party taking over the name
- Kept the old versions of one of the most popular packages on PyPI for historical posterity
As @uranusjr lays out above:
This was a conscious consideration when the flag was designed. Installers without support would simply treat the release as normal (not yanked). They wonāt notice anything as long as you donāt go on and delete the releases (which would be disruptive either way and is not recommended under any circumstances).
To make it clear (since I havenāt), I think the recent codecov incident is a perfect example that we really should stop allowing deletion, or at least put it behind a hurdle that involve PyPI admin intervention.
6 Likes
There is another reason authors might want to delete their packages that I didnāt see mentioned here: they might want to monetize them. You need to decide whether to support that.
The specific package that I have encountered is colorio, now renamed coloria, for which the author now demands a license be purchased. (Of course, there are older Github forks that one can clone, as I have done, but itās not possible to rollback to a working release using pip.)
1 Like
Good point. Completely deleting the packages or the entire project is a rather disruptive and community-hostile change if its widely enough installed to fall under this protection, and could potentially hold a whole network of dependents hostage, which is the kind of thing this change is trying to avoid. Furthermore, it is unlikely to buy the author the trust or goodwill that could translate into sales.
And Iām not sure I see how allowing deletion benefits authors or the community much in this caseāFOSS licenses (like the GPLv3 that appears to be the original license of colorio) are by definition perpetual and irrevocable, so any existing code and packages remain under that license (as you mention). Therefore, if the original package is deleted, anyone (like you, in this case) can republish it under a different name, or even the same name if the author deletes the whole project. If the FOSS version(s) of the package is developed further (as you have), users will be less incentivized to switch to the new paid version, and it will hurt rather than help the authorās business model. Conversely, if one or more of these forks (or worse, the project under the original name) contains malware, this causes direct harm to the user community.
python-sonarqube-api just deleted all of its released versions from pypi except for a new 2.0.2 release. This new release breaks backwards compatibility (which is understandable for a major version number change) and also removed a bunch of features. Therefore, those who can upgrade to 2.0.2 have a significant amount of work ahead of them due to the API breakage, and many users cannot upgrade because the feature they relied upon have been removed.
An issue for this insanity was reported in the python-sonarqube-api GitHub project, and the project owner replied explaining that the deletion of all of these releases was intentional to force users to pay for a new āproā version.
Pypi should not facilitate this kind of extortion.
Users should be able to trust that once a version is available, it will always be available. It is simply not right that a python project that once worked suddenly stops working because a project has decided to delete a release. As someone who has spent hours dealing with the fallout of this (in my opinion) malicious behavior, I feel strongly that it is in the best interest of the community to prevent this kind of attack on user from occurring again. Other package managers donāt allow deletion of releases and projects - pypi should protect its users and do the same.
7 Likes
Agreed. I suppose an enterprising user could fork the repo (source is available, right?) and regenerate the missing releases, but even that would still require users to diddle with their installation tool chain.
Sure seems like this question ought to get an actual answer. I think we established earlier in this thread that the PyPI Terms of Use does not clearly address removals. The clause for non-free licenses say PSF and others canāt have their right to publish/distribute revoked; I guess the implication is supposed to be that if itās under an open source license, that license already grants that. However, events like this havenāt revoked the right to serve up those copies, theyāve just been physically removed from PyPIā¦ (as we discussed, what that wording actually means is clearly open to individual interpretation).
In any case, in the modern world itās not always straightforward: some packages (like this one) interact with web and other servicesā¦ if the remote service itself changes (e.g. API changes) to render the old packages not useful, leaving them around creates an extra burden on the maintainer responding to (or at least rejecting) āhey, why doesnātā¦?ā issue reports, and doesnāt really help users who may download old versions (perhaps because of Python version constraints or other good reasons) and they still donāt work.
I think my opinion on this is that the latest proposal I suggested is still the best path forward for this change, but should also incorporate the additional context I added later.
That proposal had several items marked as blockers that I felt should have to be done before I personally felt comfortable officially proposing that solution, which are not done so I havenāt made an official proposal yet.
This is still on my radar, thereās just been more pressing concerns for my time so I havenāt had time to work on it. If someone else feels like picking up that work, I suggest reading this entire thread to see how the idea evolved and what peopleās opinions and concerns were at various stages, to at least get an understanding of why specific decisions were made in my latest proposal, and either take that proposal or come up with your own.
This change is impactful enough to the entire ecosystem that I personally, and I doubt any of the other PyPI admins would want to make the change unilaterally, so it would need someone to write a PEP. I have plans to work on one, but I have to finish PEP 708 first (sorry @pf_moore) and I have another PEP Iāve got in my drafts that Iām working on as well.
2 Likes
Just for the record, it seems like the issue itself was deleted by the owner (not just closed or locked, completely deleted, which is normally only supposed to be done in extraordinary circumstances), which is even more suggestive of the community-hostile behavior PyPI should prevent. (Fortunately, I managed to grab a copy from Googleās cache prior to the deletion).
Internet Archive copy:
2 Likes
Above screenshot, copy from google cache and from internet archive do not contain all comments prior to issue closing, then deleting by project owner. There you have it with full thread:
For the context - module author/maintainer also scrapped/squashed full history then force pushed changes to github, but looks like he realized that this move is harming attribution of contributors, so at the moment when iām writing that he already re-force pushed the version of history without mentioned squash. Still without historical tags (which can be found in some of forks).
In this specific case, there may be legal implications that are outside the scope of any sort of PyPI/PyPA/PSF questions. From this perspective though, Iām mainly concerned with the trustworthiness of the ecosystem - if I publish my own tool/library with a requirements.txt that pulls in something else, what do I need to concern myself with? Will existing versions stop, well, existing? Can someone change what code āsomelibrary==1.2.3ā installs?
At very least, if something gets removed, can I be confident that there will be a stub left behind that marks that it was there, and prevents different code from being uploaded with the same version number later?
4 Likes
If individual releases get deleted, then yes you can, assuming youāve pinned down to the full version (the original author could make a new post release with the same point versionāthis is also the case without deleting the release, and additionally only without deleting it they could upload a new wheel build without a new release, I believe). If the entire project was deleted, then at least up to the incident that prompted this thread (and AFAIK, still true as of now), then anyone else could create a project with the same name, release that particular version and boom, youāve got malware.
Having a lock file would prevent all of that (as it pins down to the hash), but unfortunately thereās been no agreement on that yet.
And for the record on the legal front, while Iām not a lawyer, unless they either:
- Have signed CLAs from every contributor that assign all copyright interest to the lead maintainer (which I see zero indication of), or
- Every non-trivial contributor has formally signed off on the relicense
They can relicense the code as AGPL, but MUST retain the MIT license and copyright notice in full, with clear indication of what it applies to, which they did not.
Additionally, without an explicit CLA for all future contributors, they cannot use those contributions in the premium version in any form, as it is stated to be āMIT licensedā.
Yet, because of that, anyone who does get access to the āpremiumā code can use, modify and distribute it at will to anyone and the author can do nothing about it because the license gives them explicit irrevocable permission to do so, so only one person needs to pay for the whole world to have access.
Anyway, if people want, I can split this discussion (minus Mats and Donaldās comments) into a new thread, and link that in this one.
1 Like