1

I have a package called flufl.i18n. Notice the dot in the distribution package name and file names. My last upload was version 4.1.1 on Sep 5, 2022. I upload via a GitLab action when I tag a release. This has been working well for a while now. I use PDM as my package manager.

Lately I’ve been updating my meager remaining stable of packages to drop Python 3.7 and adopt Ruff. I have a slightly different common action file to handle the ruff bits [1], but nothing relevant has changed in the publishing bits. I was able to successfully upload my atpublic package after this conversion last week, but notice that there’s no dot in the package name.

I was also able to upload flufl-enum a resurrected version of my flufl.enum package. What I missed last time was that the PyPI package name doesn’t have a dot in it: flufl-enum. The package is named flufl.enum and PDM 2.7.1 [2] builds distribution packages called flufl_enum-6.0-py3-none-any.whl and flufl_enum-6.0.tar.gz and those are exactly the files that got uploaded on Jun 6, 2023. Even with the inconsistent naming, pip install flufl.enum works as intended, as does pip install flufl-enum [3].

With that background, here’s the problem. I’m trying to make the same updates to flufl.i18n but the upload stage is failing. The error message says

Uploading flufl.i18n-5.0-py3-none-any.whl
See /tmp/pdm-publish-3blvmixr.log for detailed debug log.
 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 kB • 00:00 • 2.2 MB/s
[PublishError]: 400 Client Error: Start filename for 'flufl-i18n' with 'flufl-i18n'. for url: https://upload.pypi.org/legacy/
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

I have so many questions!

  • Why did this succeed back on Sep 5, 2022 the last time I uploaded? AFAICT, I didn’t change anything about my package’s source configuration that should have affected this, although of course all the external tools involve have seen updates along the way.
  • Why does flufl.i18n generate sdist and wheel files with dots in their names, but flufl.enum generates files with dashes in their names, even though both use the dotted-name version in their pypackage.toml files?
  • Has PyPI’s legacy upload API recently changed its naming policy? Has it gotten stricter lately? Or is this rejection an accidental narrowing of the policy? I don’t remember seeing an announcement about such a change, but it’s obviously possible I missed it. Maybe I was just getting away with something that was deprecated and now no longer allowed. Note that the packaging guide explicitly allows dots in distribution package names.
  • Am I just doing something wrong that I can’t see because I’ve stared at this stuff for far too long?
  • How do I get out of this jam?

TIA for helping clarify the situation.

  1. bad pun intended ↩︎

  2. yes, I know it’s been yanked, but that’s still the version in brew ↩︎

  3. the latter being advertised on the PyPI page. Depending on the outcome of this conversation, I would ideally like to rename it to flufl.enum for consistency, but that’s a separate topic ↩︎

2

I don’t have a link handy to the past discussion, but basically a series of things have put the ecosystem into an inconsistent state. I haven’t debugged your issue specifically, so not all of these may apply to your projects, but basically:

  • Originally the wheel spec was inconsistent in it’s requirements for names, it listed two different, mutually exclusive requirements, one of which was impossible to satisfy and one of which was not. All of the tools at the time implemented the not impossible one.
  • Warehouse has a bug (and has for a long time) that does not treat - and . as equivalent when checking that the filename matches the project name, it assumes they must be the same.
  • The wheel spec was updated with basically no real discussion, and no PEP, to require normalization when making the filename, IOW a filename like flufl.i18n-5.0-py3-none-any.whl is invalid according to the current wheel spec.
    • I believe the sdist spec then also inherited this requirement because it said “whatever wheel said”.
  • Some newer build backends, basing things off what the spec says not what the old PEPs said, have implemented things such that they force normalize filenames.
    • Some newer build backends don’t do this as well!
  • I believe some build backends have also chosen to forcibly normalize the project name in the emitted metadata.
    • Whether this is allowed I think is ambiguous. PEP 621 says that tools should normalize it as soon as it is read, but the static/dynamic differentiation was supposed to mean that the static values in pyproject.toml would be emitted as is into METADATA, but it’s unclear whether that meant byte for byte or just “equivalent”.
  • Twine used to use setuptools to escape the filename and the project name, which worked by just looking for “unsafe” characters, but in an attempt not to depend on setuptools it made it’s own copies of those functions, but they now instead work more like normalization than escaping.
  • Some tools, like Artifactory, rely on interpreting the . in a filename to indicate namespaces.

Which means we’re in a weird situation where:

  • Some tools will normalize the project name before emitting them to either METADATA or to PyPI.
  • Some tools will normalize the filename before writing them, either to the filesystem or to PyPI.
  • The above two sets of tools do not perfectly overlap.
  • Some tools (PyPI) require that the filename and the project name match, without taking normalization into account.
  • Some tools (Artifactory) require that the filenames are not normalized.

It’s a mess.

There’s an open issue on Warehouse to relax our requirements, but I’ve -1’d fixing it because right now we’re broken for projects that implemented themselves using the spec, post un(der)discussed update to require normalize AND are using a . in their name. Updating ourselves to match what the spec now requires would mean inverting that breakage so that everyone using a . in their name with an older tool (like setuptools) would break instead.

We could ignore the spec and use the original rules from the PEP (the ones that were possible to implement, not the impossible ones), but then PyPI is just ignoring the spec, and we start going back towards implementation defined behavior again.

So I’ve -1’d it because we don’t have a reasonable path forward, and any change we make is just likely to make the current situation even more confusing and broken. What we need is someone to write a PEP that fixes the current situation, picks an option and describes how all of the various tools are intended to get from where we are today to where the PEP decides to go.

I think there is only two real options:

  • Keep the normalization requirement, setuptools (and any other project that did it) to start forcibly normalizing the output filename. Figure out how tools like PyPI are supposed to function with old and new names.
  • Revert to the previous behavior where normalization was not a requirement, but tools were expected to normalize before comparing, then PyPI just has a bug where it’s not doing that, which needs to be fixed.

Personally I’m a fan of the second option, it maximizes comparability. Tools like pip are going to have to accept unnormalized names forever anyways, because there’s a huge set of names that aren’t normalized. I think it makes the most sense. Maybe if we were starting an ecosystem from scratch then force normalization would make sense, but I don’t think it does here.

However, I don’t actually care, if someone figures out a strategy for moving the ecosystem to the first option that actually works, then that’s totally fine with me. But “just changing the spec without any plans on how to do that”, is not a viable option.

To answer your specific question, in light of all the above:

 Uploading flufl.i18n-5.0-py3-none-any.whl
See /tmp/pdm-publish-3blvmixr.log for detailed debug log.
 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 kB • 00:00 • 2.2 MB/s
[PublishError]: 400 Client Error: Start filename for 'flufl-i18n' with 'flufl-i18n'. for url: https://upload.pypi.org/legacy/
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Looking at the source code, your project is named flufl.i18n and whatever emitted your filename (PDM I guess? I don’t use PDM so I dunno) does not forcibly normalize the filename, so it wrote a wheel named flufl.i18n-5.0-py3-none-any.whl.

However, whenever whatever tool is uploading to PyPI (PDM again I’m guessing from the output), it’s passing the project name as flufl-i18n not flufl.i18n. Your project is named flufl.i18n on PyPI, but PyPI will implicitly rename the project for you if you upload it with a different name [1], so as part of uploading, PyPI is renaming flufl.i18n to flufl-i18n [2].

As part of uploading, PyPI makes sure that the project name and the filename are equivalent, due to a historical bug [3] it doesn’t treat - and . as equivalent, it sees the difference and fails the upload.

If you want to fix things, I think you have 3 options:

  1. Don’t use a . in your name.
  2. Don’t use any tools that forcibly normalize . to -.
    • This could be as simple as patching PDM not to normalize the name before emitting to PyPI, which it shouldn’t be doing anyways IMO.
  3. Write a PEP that fixes the mess.

  1. It still has to normalize to the same thing of course. ↩︎

  2. Again, because whatever is doing the actual upload to PyPI is going "hey, here’s a wheel for flufl-i18n. ↩︎

  3. The bug at the time of writing that code was that it wasn’t fully normalizing filename before checking… but with the changes to the spec, the bug is now that it’s not enforcing that it is normalized. ↩︎

1 Like
3

FWIW, my view is that the form of the name as stated by the project author should be used in all places where a UI displays the “project name”. When a user enters a project name, any name that normalises the same as the metadata name should be accepted. Those are the only two rules that really matter to me. I don’t care how filenames or URLs get normalised, as long as the rule is well-defined and consistently defined.

  • I consider (to be explicit, this is my personal view only) normalising the name before emitting it to METADATA or PyPI to be a bug.
  • I’m fine with normalising the name for filenames, but I think behaviour should be consistent. Currently, the spec says normalise[1], so tools that don’t normalise simply haven’t been updated to match the latest spec. Such things happen, and consumers should be prepared for this - after all, the historical files won’t ever go away…
  • I don’t have a view on what indexes should do. We’ve never really tried to standardise the upload side of the index protocol, so trying to impose order there might be problematic.

As long as the two options you mention are explicitly just about filenames, I don’t mind much which approach we take. But I’m strongly -1 on broadening the normalisation requirement to include forcible normalisation for anything other than filenames, and strongly +1 on clarifying explicitly that the current standards do not support the stance that normalising metadata is the correct approach[2].

I don’t have the energy to create a PEP for any of this myself. I suspect such a PEP could be difficult to get agreement on, because a lot of tool developers will have a vested interest in having their current interpretation declared as correct, even though in the grand scheme of things getting consistency is a lot more important than which consistent interpretation gets chosen :slightly_frowning_face:

  1. That should have gone through the PEP process, but unfortunately didn’t. At this point, though, what’s done is done. ↩︎

  2. I’m stopping just barely short of declaring normalising metadata as contrary to the standards - I consider the statement in PEP 621 as misleading and unfortunate (I always read it as simply reiterating “normalise before you use this internally”, but clearly others didn’t :slightly_frowning_face:) ↩︎

2 Likes
4

Here’s the issue: https://github.com/pypi/warehouse/issues/10030

5

Wow, it sounds incredibly complicated and sticky.

I am using PDM (@frostming) and there’s been some churn there, with 2.7.1 getting yanked and 2.7.2 getting released today-ish. But also brew still has 2.7.1, so I’ve tried to do some comparisons with pipx installed versions too. I’m not sure this churn is in any way related to this issue though.

There’s also a lot of “names” being referenced. There’s at least:

  • Canonical definitive project name as specified in pyproject.toml
  • PyPI project name as defined when it was registered (and not renameable AFAICT).
  • sdist and wheel file names generated by the build backend (in my case pdm-backend)
  • Project name as captured in the sdist/wheel metadata files.

None of these seem consistent.

I totally agree. As a maintainer, this is the name I want to promote as my package name, and thus what ends up in the metadata. There are two sides to this, one being the name I use in my pyproject.toml but the other is the name I tell my users to pip install. These two shouldn’t be normalized, but PyPI should definitely treat e.g. flufl.enum and flufl-enum project names as equivalent, otherwise we’re just inviting typosquatters.

I’m less concerned personally about the actual file names as long as it all works, but note that sometimes that could be troublesome. E.g. sometimes you hand a consumer a specific .whl file and tell them to pip install that into a particular environment. That’s a pretty uncommon use case (IME) so I’m less concerned if that’s a little jarring.

What I didn’t understand is why PDM was behaving differently with flufl.enum and flufl.i18n. In both cases, I very explicitly use the dot in the project name in pyproject.toml, but for some reason, the flufl.enum metadata name and filenames were getting normalized to flufl-enum and flufl_enum-6.0{.tar.gz,-py3-none-any.whl} respectively, but flufl.i18n was not getting normalized.

There’s only one explanation, and I’ve confirmed this through experimentation. PDM must be talking to PyPI before generating the metadata and building the distribution packages. The only place flufl-enum string shows up is on PyPI. I confirmed this by running pdm build on a clean git repo with my networking turned off, and yep, pdm build fails with a connection error in that case. Ergo, PDM must be deciding to normalize based on the PyPI project name. I bet if I had registered it as flufl.enum instead, I’d have seen this error earlier.

This plus PDM’s magical PyPI lookup is likely what’s breaking flufl.i18n uploads now. It goes to build that project’s metadata and filenames, find that the project name on PyPI is flufl.i18n and decides not to normalize either name. But then because of the Warehouse bug you mentioned (or some other PyPI behavior), those unnormalized [1] names get rejected at upload time.

AFAICT, there’s no switch to pdm-backend that tells it to normalize the metadata and filenames unconditionally. So between all that, the inability to rename PyPI project names, and my strong desire not to change my canonical project name in pyproject.toml, I think I’m stuck.

  1. but I’d argue correct ↩︎

6

What, no. PDM-backend will always normalize the wheel filename to lowercase_underscore. It fails without network because pdm build needs to install the build requirements from PyPI.

IMO PyPI should accept filenames in unnormalized forms and store them in the same lowercase-hyphen namespace.

7

Ah, my bad, thanks for correcting that.

Then I’m still confused! Why does flufl.enum get normalized but flufl.i18n does not?

flufl.i18n; no normalization:

% head -2 pyproject.toml 
[project]
name = 'flufl.i18n'
% pdm build
Building sdist...
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /.../flufl/i18n/.venv
Built sdist at /.../flufl/i18n/dist/flufl.i18n-5.0.tar.gz
Building wheel...
Built wheel at /.../flufl/i18n/dist/flufl.i18n-5.0-py3-none-any.whl

flufl.enum; normalized:

% head -2 pyproject.toml 
[project]
name = 'flufl.enum'
% pdm build
Building sdist...
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /.../flufl/enum/.venv
Built sdist at /.../flufl/enum/dist/flufl_enum-6.0.tar.gz
Building wheel...
Built wheel at /.../flufl/enum/dist/flufl_enum-6.0-py3-none-any.whl
8

Different version of pdm? Looking at the code @frostming linked to, it looks like the normalisation changed about 4 months ago…

9

The snippets above were generated with PDM 2.7.4.

10

It appears that you’ve specified a different build backend for each of these:

11

Well there ya go! Pebkac it is! :pray: I think I saw a warning about the old build backend in a different repo, but missed it in this one.

12

I’ve been observing the situation for a while since we need to generate filenames from Gentoo package names. However, I don’t fully agree with the narration given here.

From my perspective:

  • all modern build systems do follow the “new” wheel naming and PEP 625
    • whenever they did not, it was considered a bug and fixed
    • hatchling provides an option to disable normalization to workaround PyPI
  • setuptools is the only build system that doesn’t follow the modern spec
    • a major blocker for this is probably that the name comes from distutils
  • there is no ongoing effort to change the standards and normalization rules (and I’m really happy about that because the last thing we need right now is fourth standard on how to name files)

In my opinion, the real problem here is that @dstufft is single-handedly battling the agreed-upon standard and using PyPI to block it. At this point, I’ve been slowly considering appealing to the higher authorities, or at least trying to establish if there are “higher authorities” relevant to that. In my opinion, it’s not acceptable for an admin position for a key point of Python packaging infrastructure to be used to counter accepted standards.

13

Donald is trying to implement a fix that hurts the least number of real users.

We don’t (usually) use “standards” to beat users over the head and make their stuff fail until they fix it.[1] We invest the effort ourselves to make things work for users where they are at.

Keeping an ecosystem running isn’t as trivial as just writing a spec. Donald’s doing a great job finding the balance.

  1. Occasionally some of us do that, and usually some/many of us try to oppose it and lose. :frowning: ↩︎

5 Likes
14

I’m sorry if I’m missing something but I really do find it hard to believe that rejecting normalization of . to _ in sdist (but accepting all other aspects of PEP 625 normalization) is actually likely to hurt a meaningful number of users.

So far the only “breakage” mentioned is some tool called Artifactory, and from the description given it sounds like it’s making bad assumptions anyway. At this point, it sounds to me like keeping things in limbo is actually making it worse for the users.

Package authors are users too. Whenever they can’t upload an artifact, they suffer from this state of things. Whenever they have to work around the problem and end up publishing a semi-broken release archive (e.g. because they rename the file to workaround the problem), we all suffer.

Leaving things as-is is only leaving us with bigger mess to clean up in the future.

To put it bluntly, someone needs to do the work. If you can’t convince anyone to do it for you, you have to do it yourself. That’s how volunteer open source works.

Blocking things in order to force someone to do the work just isn’t going to work. It’s creating the kind of hostility that you are witnessing right now. If you can’t get anyone to do the work and you can’t do it yourself, then it’s time to let go and agree that things won’t be perfect.

In any case, the cat’s out of the bag for a long while now, the standards are what they are and even if they change again in the future, what PyPI should be doing is being lax in what it accepts and not try to prevent people from shooting themselves in the feet. That’s not its purpose.

15

Let’s not make this a matter of individuals, please.

How about this for a simple way forward and action plan? @dstufft I’m willing to convert this to a PEP if you feel it’s necessary. Personally, I think it’s almost entirely just reiterating the rules as they stand, so I don’t think it’s worth it, but if writing a PEP is what’s needed to get things unblocked, I’m fine with that.

  1. The specification for wheel filenames stands, in particular the statement that normalisation with underscores is the standard format for wheel filenames.
  2. Tools producing wheels MUST use that format.
  3. Tools consuming wheels MUST[1] be prepared to accept files with mixed case and with full stop (period) characters in place of underscores. Again, this is noted in the existing spec.
  4. Indexes are in a slightly odd position, as they are (in a sense) both consumers and producers. To that extent, we have some explicit rules for them.
  5. Indexes MUST accept normalised wheel filenames for upload. They MAY accept other filenames as described in (3). This is essentially just “indexes are consumers”.
  6. Indexes MUST serve wheel filenames using the normalised form, and MUST use the normalised form when publishing the wheel filename (for example, in the simple index pages), whether as a standalone name or as a component of a URL. The same applies to names derived from the wheel filename, such as metadata files. It’s an implementation choice whether they store the name as uploaded and convert when needed, or convert the filename on upload. This one is basically “indexes are producers”.
  7. Indexes MAY continue to serve files previously uploaded with unnormalised names under their existing names, but MUST NOT do this for new uploads.
  8. Consumers of index data SHOULD (for backward compatibility reasons) accept unnormalised wheel filenames and URLs, to support working with older indexes that don’t follow these rules and indexes taking advantage of the provision in (7). The consumer MAY warn in circumstances like this[2].

Both PyPI and setuptools (technically, wheel) can implement these rules as soon as they want. PyPI would obviously choose to accept old-form uploads (although they have the option to phase this out over time, if they wish to do so). Setuptools can hold off if they have legacy constraints making it hard for them to switch.

The fact that these rules mean that I can upload flufl.i18n-5.0-py3-none-any.whl and PyPI will store it as flufl_i18n-5.0-py3-none-any.whl is mildly uncomfortable, but if we view the name of a wheel file as mostly an implementation detail (which IMO we should, because it’s designed for machine consumption first, and user readability second) it’s acceptable.

The same rules can be applied for sdists, although I’m not sure how widely PEP 625 has been implemented. It was mostly a clarification of existing behaviours, so tools may not have consciously adopted it. That’s not a reason for it not applying here, just a note that there may be more tools which find they don’t apply the correct rules…

There’s nothing in this proposal about driving producers to adopt the new rules. That’s intentional - projects have to make their own prioritisation calls. It sucks if tools are slow in adopting new standards, but that’s unfortunately normal, and no tool has a right to complain here, as we’ve all been the “late adopter” at some point or another.

  1. I’d be willing to downgrade this to SHOULD, but the existing spec does say “must”. ↩︎

  2. Although in practice, such warnings are not likely to be helpful, as the user can’t do anything useful about them. ↩︎

2 Likes
16

“Some tool called Artifactory” is probably the one of the most widely deployed non-PyPI implementations of the package index standards, so it’s hardly a minor consideration, even if it’s not implementing things quite how we’d like.

1 Like
17

Wouldn’t renaming wheels and sdists result in the *.dist-info/ folders in wheels and root folders in sdists having different (pre-normalized) project names than their filenames? I don’t think the standards say anything about that situation at the moment, so it should probably be documented that consumers should accept such mismatches.

1 Like
18

Damn. Good catch, yes that’s something I hadn’t considered (and it’s probably why nobody, myself included, should assume this is simple to fix). I start to see why @dstufft has concerns about all this :slightly_frowning_face:

I could argue that the wheel spec can be interpreted as saying that the .dist-info directory should use the unnormalised name. But the spec for recording installed packages explicitly states that the installed .dist-info directory must be normalised according to the “change dashes to underscores” method, shich implies that the same directory in the wheel should follow that pattern as well. I suspect plenty of packages don’t follow that rule correctly, either :slightly_frowning_face:

Edit: Yes, a quick scan shows Pygments, MarkupSafe, Jinja2, Babel, jaraco.classes, ConfigUpdater, PyScaffold, PyYAML, jaraco.context, jaraco.functools, jaraco.text, CacheControl, Unidecode, PyRSS2Gen, as well as probably others all incorrectly normalised in packages I have installed locally).

Wow, once you start looking, it’s a real mess…

Unfortunately, the wheel spec definitely shows its age in some ways - it’s far looser than we’d accept for a standard these days. But we’re a long way from having enough resource to do any but the most essential of “tidying up” on existing specifications, I’m afraid…

19

What’s Jinja2 doing incorrectly? I’ve just been using setuptools and twine, nothing has changed in years (except switching to pyproject.toml, not sure if that’s released yet).

20

It’s not specific to sdists, it affects wheels as well.

Currently, everyone using a . in their name with setuptools (which I believe is still by far the most used build backend) works when uploading to PyPI. The tools that implement the specs as written have never been able to upload to PyPI.

If PyPI switches to enforcing the specs as written, then we suddenly break all of the existing projects who have been successfully using . in their name, sometimes for 10+ years, in order to fix the projects who have never been able to successfully upload to PyPI.

If PyPI just relaxes the rules, and accepts anything, then it’s likely that we never align the build backends into a single behavior, and regardless of what the spec says, the de facto implementation will be what PyPI implements (i.e. that normalization is not required). It also leaves a “missing stair”, we know there are widely deployed tools in the wild that rely on there being a . in the name, those tools are just going to randomly break for users depending on what build backend they happen to use for their project.

all modern build systems do follow the “new” wheel naming and PEP 625
setuptools is the only build system that doesn’t follow the modern spec

When I looked in February of this year, it was not just setuptools. In fact in this very issue you can see the filename that Barry got was flufl.i18n-5.0-py3-none-any.whl from PDM.

Here’s my results from Feb 25th, 2023, this may have changed since then, I haven’t re-run these to see.

Build System sdist filename wheel filename
flit-core Test.Project-1.tar.gz test_project-1-py2.py3-none-any.whl
hatchling test_project-1.tar.gz test_project-1-py2.py3-none-any.whl
meson-python test_project-1.tar.gz test_project-1-py3-none-any.whl
pdm-pep517 Test.Project-1.tar.gz Test.Project-1-py3-none-any.whl
poetry-core test_project-1.tar.gz test_project-1-py2.py3-none-any.whl
setuptools Test.Project-1.tar.gz Test.Project-1-py3-none-any.whl

Note that, at that time, setuptools, pdm-pep517, and flit-core all produced either sdists or wheels that were not normalized.

I haven’t checked how many of those projects always emit normalized versions, PyPI doesn’t enforce normalized versions, which is also part of the spec, and I assume there’s a non zero number of them that don’t normalize versions.

I’ve repeatedly said that I don’t personally care, at all, what the filename format is. It is basically the least interesting thing in the world to discuss. What I care is that we don’t have some flag day that breaks large numbers of users and that we don’t create a situation where to avoid that, the tools necessarily have to ignore the spec.

The specs as written, have no plan whatsoever for dealing how to introduce the now stricter requirement to the ecosystem, and afaict took no efforts to minimize the breakage, because the backwards incompatible change to the wheel spec was not really discussed, and was effectively added “by accident”.

I’m also not the only person with a commit bit on PyPI. I’ve voiced my opinion that I think the specs force us to ignore them or have a flag day that breaks users that are currently working, and because of that I’m not willing to make a change to PyPI. If the other admins disagree with me they’re able to merge a change without my approval, we have no requirement for unanimous consent for merges in PyPI.

Weird how you’re demanding that PyPI does the work of figuring out how to implement a backwards incompatible change that was introduced without any migration plan. I guess PyPI doesn’t count as volunteer open source?

Like this is basically a fundamental part of change management, determining how to actually roll a change out in the ecosystem. That was skipped for these backwards incompatible changes, and that is what is causing the problem here. I know that the people involved did not intentionally do that, and I believe it was entirely accidental.

The need to manage the roll out of the change doesn’t go away because it was skipped though. All I’ve ever asked is that someone figure out how in the world this backwards incompatibility is supposed to get rolled out in an incremental way across the ecosystem, and get agreement on that being the path forward, because there’s open questions on what that should look like.

  • Should PyPI just allow any form of the name normalized or not? If so this means that it’s unlikely that tools are going to be able to assume that names will be normalized, because one of the ways we’ve shifted the ecosystem is through enforcement on PyPI. Is that fine?
    • If the rule isn’t important enough to enforce it on PyPI, is it really a MUST?
  • Should PyPI rename the files upon upload? Should we apply it retroactively? If we do this does it matter that the filename and the .dist-info will not have matching names, will that break anything? Do any major tools out there rely on PyPI not renaming the files upon upload?
    • A quick skim of twine looks like this would break at least the feature to skip uploading existing files. Is that fine? No idea if there are other tools or even places in twine.
  • Should PyPI hard reject non-normalized filenames? What timeline makes sense to do that in? How many projects would that break? Are we as a community fine with that amount of breakage?
  • Are the backwards compatibility concerns too hard to sort out and should we relax the spec to allow non-normalized names as well?
  • Something else?
2 Likes