I just saw a notice from a popular package which I think is based around a misunderstanding. I’m going to reach out to the maintainer to recommend a resolution, but I’d like to make sure I have good information first.
Here’s the unabridged text from the start of the current ruamel.yaml readme:
If you are interested in future upgrades of ruamel.yaml please check the documentation on installing, since at some point I might not be able to upload a new version to PyPI with updated information.
ruamel.yaml was intentionally named as yaml in a namespace ruamel. The namespace allows the installation name to correspond unchanged to how the package is imported, reduces the number of links I have to create in site-packages of a Python install during development, as well as providing a recognisable set of packages my company releases to the public.
However, after uploading version 0.18.7, I got an email from PyPI, about having to change the project name to ruamel_yaml to comply with PEP 625, sometime in the future. The email doesn’t say if namespace packages are no longer allowed, or how to deal with the very real clash with the pre-existing package ruamel_yaml.
I might not be able to adapt ruamel.yaml, in a way that does not negatively affect the 0.5 million daily downloads (and my own usage of the package) in time. My experience with other such service downgrades (Bitbucket, Readthedocs), has not been entirely positive.
So what we’ve got here, trying to remove the elements of opinion and do a little bit of digging, seems to be:
the package is published as ruamel.yaml and uses sdist names like ruamel.yaml-0.18.10.tar.gz
there is no registered package on pypi named ruamel
this maintainer uses sourceforge and engages with users via StackOverflow[1], probably isn’t on this forum and may be isolated from other maintainer experiences
If I follow correctly, there’s actually no issue here. I’d like to ask the maintainer to remove this note (I’ve interacted on his ticket tracker before and will hopefully have good rapport) as I think it indicates an unnecessary level of alarm.
Additionally:
I think the maintainer should lay claim to ruamel as an empty package (even if it’s never used, just to assert ownership) and intend to recommend this.
No changes should be needed to the dist filenames.
If anyone knows what triggered an email to the package publisher about PEP 625, I’d love to know what the criteria are, so that I can pass that information along (even if the logic may have flaws, information will probably defuse this a bit).
Have I missed any key details here? Does this maintainer have any actual reason to be concerned, and if so what should he be doing to resolve those concerns?
I note this even though it’s slightly OT because it helps inform communication with this particular package maintainer. He has some strong opinions about tooling and platforms, as we are all entitled to have. ↩︎
I don’t know the details of emailing people (or whether there was actually any such official e-mail from PyPI in the first place) but PEP 625 is only concerned about source distribution names and not project names. The project can still be named ruamel.yaml, and there’s nothing that needs to change there.
What really should change though is the filenames of the source and binary distributions from ruamel.yaml-* to the normalized ruamel_yaml-*, because both do not follow standards. The former was already done in setuptools a while ago, and it seems that fixing wheel naming has finally started moving forward.
And I don’t have a clue what the claimed ruamel_yaml project might be. PyPI doesn’t allow two projects to use the same normalized name, and redirects ruamel_yaml to ruamel.yaml.
If there’s any risk here, is that people who have hardcoded the sdist name will have to update it. However, all standard tools should handle this just fine. In fact, some tools right now (such as Gentoo packaging) need to apply workarounds for source distributions using old names.
Wait, did I misunderstand this? I thought ruamel.yaml-* was correct because these dists provide ruamel.yaml as a package in a namespace package. Wouldn’t naming the dists ruamel_yaml-* only be appropriate if the provided package name were ruamel_yaml?
Yes, I forgot to include that. I couldn’t find a package with that name on or off of PyPI, so I’m not sure what that’s about.
Project names have no actual relationship to import names, although by strong convention they are the same. But packages live in a flat namespace, so ruamel.yaml is “just a name” , unrelated to the package “ruamel”. By package name normalisation, all of the names ruamel.yaml, ruamel-yaml, and ruamel_yaml are the same, and which you choose is purely a matter of presentation.
Distribution names must use a normalised form of the project name - in part, because the hyphen is a field separator when parsing the name so we must avoid it within fields, but also to make filenames deterministic. There’s an outstanding setuptools bug because they don’t do this - I believe it was slow to get resolved because the setuptools maintainers had a disagreement with the standard, but that’s been addressed now, so I think it’s just a matter of resource availability.
I don’t know why PyPI has started issuing warnings before the setuptools bug is fixed - giving users no option[1] for dealing with the message isn’t reasonable IMO.
What’s the advice about “claiming” the name “ruamel”? I know there’s no technical reason to do so, but it seems like it might avoid some confusion. Is it acceptable to make an “empty” distribution named “ruamel”, and what would be in it?
Well, technically that could be considered “name squatting”, but it’s relatively common these days and I’m not aware of anyone having any major problem with that so far. It’s mostly used to prevent scammers from using “familiar” names to distribute malware.
In my opinion, the cleanest solution is to create a source distribution that throws an error while building, suggesting the correct package names.
Okay, thanks all for helping me get clear on the issue! My refined message to this maintainer will be
The dist filenames need to be normalized. Contents do not change.
Using the latest setuptools to build is sufficient.
PyPI already does not allow for conflicts, so don’t worry about name conflicts in the normal case.
I personally recommend claiming ruamel as a separate matter, but nothing requires it.
I’m more or less positive that there’s an unusual build environment in use here – the project has a peculiar source tree layout with some files at the root which get packed into the built package.
I’ll reach out today with the above and see what kind of response I get.
Yeah, I took a quick look at the setup.py and it has custom sdist and bdist build code. I don’t understand what the code is trying to achieve, but it is fiddling with project names.
So the maintainer will need to modify their customisations to respect the standards. It’s hard to advise on how, because (a) I’m not sure what they are tring to do, and (b) I don’t know enough about customising setuptools to advise anyway.
I don’t know why PyPI has started issuing warnings before the setuptools bug is fixed - giving users no option[1] for dealing with the message isn’t reasonable IMO.
Yes, it seemed like a premature decision given the percentage of established projects relying on SetupTools. As someone who’s an owner contact for a fairly active ecosystem of projects, I’ve personally counted 54 such warning notifications received since November 19. Thankfully that represents only a small fraction of our projects affected or else my inbox would really be getting flooded, but I’m hopeful that it will just stop once SetupTools is finally normalizing wheel filenames.
The latest version of setuptools fully supports PEP 625 for source distributions and produces normalized source distribution filenames. The bug is closed.
PyPI only warns about noncompliant source distribution filenames.
PyPI does not warn about noncompliant wheel filenames. We plan to, but I have marked the PR to start warning about these as blocked on there being a setuptools release that users are able to upgrade to.
The email we are currently sending to users about PEP 625 project names is here. If anyone had suggestions about how to make this more clear given the OP’s example, please feel free to submit a PR.
Thanks, that looks perfectly reasonable. I’m surprised that @fungi is seeing so many notifications. The only reason I can think of is that they pin their version of setuptools, for some reason, and they need to update to the latest version.
The OP may have worse problems, as they customise the sdist build process. But I don’t think a generic message should need to cover that - IMO, it’s perfectly fair to assume that anyone customising setuptools at that level should know what they are doing and the implications (which will include tracking the appropriate standards). Of course, sadly, I suspect that often isn’t the case, but there’s only so much we can do in that situation.
I’m hopeful that it will just stop once SetupTools is finally normalizing wheel filenames.
Actually, looking back, it was sdist filenames not getting normalized. The good news is that an in-flight improvement I have to switch our release automation over to relying on the pyproject-build tool drags in a new enough SetupTools to solve this now.
The latest version of setuptools fully supports PEP 625 for source distributions and produces normalized source distribution filenames. The bug is closed.
Thanks for the details, looks like that should be solved for setuptools 70.2.0 and later (circa 2024-07-01). Based on that, PyPI did wait an additional 4 months to begin emitting warnings. Surveying platforms I have handy, venvs created with stdlib from CPython 3.11 or older bundle a preinstalled SetupTools too old to have this fix, so I expect that’s where our warnings were originating. As noted in my other followup, using pyproject-build to perform isolated sdist builds (even with older CPython) ends up pulling in newer SetupTools and solving the problem.
PyPI only warns about noncompliant source distribution filenames.
Yes, apologies, it’s definitely only sdists we’re getting warnings about. Also our wheels do seem to be properly normalized anyway, checking over the filenames we’ve been uploading.
Thanks, that looks perfectly reasonable. I’m surprised that @fungi is seeing so many notifications. The only reason I can think of is that they pin their version of setuptools, for some reason, and they need to update to the latest version.
Not “pinning” per se, just using an older (CPython 3.10 from Ubuntu 22.04 LTS) stdlib venv without any separate step to explicitly upgrade the SetupTools it supplies.
For a long while I’ve been publishing namespace packages as foo.bar, which PyPI preserves as the display name on those packages’ PyPI page[1].
Of course as others have said, the sdist and wheels that get uploaded have their names normalized to underscores, which is fine, supported by my package manager of choice[2] and doesn’t affect pip install foo.bar commands at all. I like that the PyPI page and documentation use the import-name-matching name rather than the normalized name, but YMMV and it really doesn’t matter practically speaking anyway, at least IME.
there was a bug about this a couple of years ago, but as it’s been fixed, it’s not worth finding a reference to ↩︎
).