PEP 825: Wheel Variants, Package Format (split from PEP 817)

Packaging Standards
41

I understood what you’re proposing, what I’m saying is that given everything else is allowed to be mutable and to be added to over time without any sort of up front declaration makes doing that a confusing footgun for people. It’s going to go against all of their expectations.

Adding constraints like that needs to be done holistically, in a way that actually makes sense in a big picture, not just slapping it on a random piece because that happened to be the last piece that was added.

I also don’t believe it actually allows any sort of optimizations to take place given that you still can’t assume other platform tags aren’t going to be used. At least I’m not able to think of any optimization where knowing that a different variant (and again, specifically variants) might be used in the future is actually helpful– but it does impose at least somewhat of a cost.

It’s not even something that is technically possible to require generically. An installer can’t go backwards or forwards in time to see what used to be there or what is there now. I honestly even think the wording in the current PEP is probably too strong here, and the requirement should mostly be around making sure that the current state of the index is consistent (e.g. you can’t have 2 wheels for the same (project, version) that try to define a given variant label differently.

On PyPI that would naturally imply that you can’t change the definition of a label, because deleting files on PyPI is unusual and you can’t re-use a filename once you do delete it which further makes it unlikely people are going to delete them.

On personally managed indexes it might mean that you redefine those labels constantly (particularly as you’re first implementing variants for a project!), but that’s OK because those are your personally managed indexes and you can already do pretty much anything you want.

1 Like
42

I’ve just opened a PR with two clarifications (related to the discussion here):

Long story short:

  1. We clarify that the values in variants are sets (that get converted to lists as a JSON limitation), and mandate that they are sorted, so that implementations can use simple == comparison on the whole dictionary without having to convert values to sets.
  2. We update the rules for merging variant metadata from individual wheels into index-level metadata to use a simpler “the result is the same irrespective of the order”. This should be much easier to follow than the previous “resolution results within a subset of variants do not change“.
43

@pf_moore

Hi all, we - the authors - would like to request pronouncement on PEP 825 . It feels like the discussion has largely converged, the sharp edges have hopefully been sanded down, and the PEP seems ready to roll toward a decision.

6 Likes
44

I’ll take a look and let you know my decision. For personal reasons, my time is limited right now, so apologies in advance if it takes a little longer than usual to do so.

2 Likes
45

Fantastic ! Thanks a lot Paul, we appreciate the help.

46

I’ve started looking at the PEP. At the moment this doesn’t constitute a formal response, but it is flagging some initial concerns I have that will factor into my ultimate decision. For now, I’ve only focused on the change to the wheel filename.

Like it or not, the new filename is not backward compatible. There is no formal constraint that a platform tag can’t be numeric, so a wheel foo-1.0-1-none-any.whl is valid. Add a variant label and you get foo-1.0-1-none-any-var.whl, which will be interpreted incorrectly. And yes, I checked this:

>>> packaging.utils.parse_wheel_filename("foo-1.0-1-none-any-var.whl")
('foo', <Version('1.0')>, (1, ''), frozenset({<none-any-var @ 1209214139968>}))

This is simple to fix - the PEP just needs to require that platform tags MUST begin with a non-numeric character (which is fine, because all existing tags follow that rule). But it needs to be made explicit, both in the PEP and in the updated specs once the PEP is accepted[1].

There’s a more general point here as well - the “backward compatibility” section needs to be a lot more through on the impact of this change. The point of that section is not to claim that everything will be fine, but rather to fairly assess what issues will arise from the change, and establish that the disruption caused is acceptable, ideally discussing how the transition will be managed. For example:

  1. Like it or not, code will parse wheel filenames by counting the dashes. The packaging library does. I know I’ve written lots of code like that. And I don’t check the build number is valid[2]. My code will break. Probably not badly, but that’s not the point. The point is, is it worth breaking that code to implement this PEP? Do the benefits justify it? Probably[3].
  2. The packaging function parse_wheel_filename will need to change its signature which returns the parts of the wheel filename, because it might now have to return a variant label. That’s a breaking change to a public function. What’s their compatibility policy? How do they want to handle that API change? Do they have any idea how widely that function is used?
  3. What about the transition period? Will tools that are not prepared to handle variant wheels need to pin their dependency on packaging when the updated version which does handle variant wheels gets released? Will a delay in implementing variants in packaging cause a knock-on delay in adopting the PEP? Should a PR for packaging be a prerequisite for adoption of the PEP?

To be clear, I’m fine with the solution adopted in the PEP. But the questions above need to be considered when approving the PEP, and IMO it’s the responsibility of the PEP authors to consider those questions, and document their conclusions in the PEP, so that I don’t have to :slightly_smiling_face:

To summarise, there are two concrete actions here:

  1. Add “platform tags must not start with a digit” to the PEP.
  2. Update the backward compatibility section to discuss the breakage the filename change will cause. Ideally without making the section a lot longer - most of the paragraph discussing the filename compatibility can simply be replaced, as once you discuss what breakage could occur, there’s no longer any real need to make the point about “most of the time, it’ll be OK”.

  1. Updating the specs will be a non-trivial task - the existing specs are frustratingly vague, and incorporating PEP 825 without accidentally introducing new constraints, like the one about platform tags, will be hard. Someone should be planning what’s needed there, possibly even starting to prepare a PR for the specs repo right now. ↩︎

  2. Nobody actually cares about build numbers :slightly_smiling_face: ↩︎

  3. Although there’s an argument that we should design an extensible change, so we don’t have to go through this again for the next change, and the benefits are that much higher as we’ve “solved the problem once and for all”. I’d accept the response “that’s too complicated to tackle now” for that one, though. ↩︎

8 Likes
47

Thanks a lot Paul. Give us a few days to send a PR to the PEP. We’ll tag you in the PR in review so that you can see & review the changes

2 Likes
48

I’ve done my best to address your requests in PEP 825: Address filename compatibility concerns from DPO by mgorny · Pull Request #4890 · python/peps · GitHub.

I’ve started the dialog with packaging authors in `parse_wheel_filename()` vs. wheel variants (PEP 825) · Issue #1148 · pypa/packaging · GitHub (which is also linked within the updated PEP). Long story short, the main idea is that the current API will reject variant wheel filenames (as it does now), and support for variant wheels will involve some kind of opt-in: either using a new API, or passing a flag that says “I support variant label”.

3 Likes