Python tags - specific version of interpreter or minimum version?

Why would the wheel need to do that? The wheel should say “I can be used on anything that claims it’ll handle cp37 wheels” and it’s up to the target environment (technically, the installer for that environment) to say if it can deal with those wheels. What’s the use case for saying “please ignore any claims from the target that it supports cp37 wheels, and only install on…” what, exactly? On things that claim cp37 support and don’t report a Python version other than 3.7? You can do that with Requires-Python, but I’m still unclear why you would.

As a matter of implementation, yes, it’s up to the tool consuming the wheel to decide what it accepts. But the meaning of the cp37 tag - what environments it is compatible with in principle - is defined by the spec. Just, in this particular detail, not very precisely defined. :slightly_smiling_face:

Good point. My recollection is that it’s vague because at the time we were originally discussing this, it wasn’t at all clear what the right answer should be (would PyPy, which had its own version numbering, be able to support cp37 wheels, for example). Now, we’ve got a lot more experience and the answers to questions like this seem to have settled down (at least for now :wink:). But we have backward compatibility to contend with now, which we didn’t then.

I don’t think it’s possible to standardise cp37 as meaning anything other than “runs on CPython versions 3.7 and later”. But wording that needs some care - we don’t know that CPython 4.0 (or even 3.12) will still be compatible with cp37 wheels.

And there’s also the more immediate oddity around how platform and ABI interact, too - the list I showed above included cp39-cp39-win_amd64 and cp39-none-win_amd64, but not cp37-cp39-win_amd64, cp37-cp37-win_amd64 or cp37-none-win_amd64. Why? I’ve no idea. But that demonstrates that cp37 doesn’t simply mean “CPython 3.7 or later”…

Binary compatibility is hard. @ncoghlan used to make that point a lot, and he’s far more knowledgeable about the area than I am :slightly_smiling_face:

As I said, I’d support getting this written down as a standard that documents what packaging.tags does, in a way that explains the logic so that it can be extrapolated to future releases and implemented by other libraries. But so far no-one has been sufficiently motivated to do so.

PS I should also say that I’m really pleased that Pynsist, by having its own tags implementation, triggered this discussion. It’s a great example of why everyone relying on a single reference implementation is bad, because we then don’t question the details.

1 Like

Reading PEP 425 again:

Importantly, major-version-only tags like py2 and py3 are not shorthand for py20 and py30. Instead, these tags mean the packager intentionally released a cross-version-compatible distribution.

My interpretation is this implies py20 is not cross-version-compatible, i.e. does not claim to work on Python 2.1 or later. So maybe the intention is actually to make cp37 mean just CPython 3.7? A tool installing into CPython 3.8 can claim the target interpreter is compatible with cp37 if it wants to, but the tag itself does not claim CPython 3.8 compatibility.

Whatever the intention was when PEP 425 was written, I think it’s most practical now to codify how packaging has interpreted it, unless there’s a really solid reason that that needs to change.

Wheels for CPython 3.7 specifically can be tagged with cp37-cp37 (i.e. using the ABI tag to make it more specific). Wheels on PyPI or other (PEP 503) indexes can also be selected using the requires-python metadata, which is more flexible and descriptive.

1 Like

In keeping with what we’ve done for other spec changes (e.g. escaping wheel filename components), I guess we should copy the content of PEP 425 to the appropriate spec page on packaging.python.org, and then propose changes there?

Generally, we need a PEP revision that notes the new location is now the canonical definition, followed by future changes being made as PRs to that location.

However, please note that this does not mean that we have the discussion on the new wording on the PR. There’s still a requirement for consensus on the change to be agreed here. See here for the specifics of the process (distutils-sig should be replaced by Discourse nowadays…) It’s also worth noting the following section which prohibits backwards-incompatible changes.

The key point is

If a change being considered this way has the potential to affect software interoperability, then it must be escalated to the distutils-sig mailing list for discussion, where it will be either approved as a text-only change, or else directed to the PEP process for specification updates.

I’ve generally been relatively relaxed about not insisting on the PEP process for small changes, mostly because people tend to perceive the PEP process as bureaucratic (even though it needn’t be in practice).

In this case, though, I feel like writing up the details will be a fairly difficult process, so I’m inclined to say that if we want to consolidate the tag specs on packaging.python.org as part of adding this information, we should have a PEP that:

  1. Defines the rules for how installers should declare what tags they accept, as a supplement to PEP 425.
  2. States that the canonical location for the packaging tags specification will be moved to packaging.python.org.

The PEP should link to a PR that updates the tags spec page here. Note that this will be a non-trivial rewrite (it certainly won’t just be a “lift and shift” of PEP 425) because that page links to the various manylinux specifications as well as to PEP 425, but it doesn’t clearly state (for example) that py3-none-any is a valid tag on Linux - so it’s not (IMO) currently in a state that would qualify it as the definitive spec for tags… I’d consider that PR as a required part of the PEP, as it’s a relatively complex rewrite.

Sorry - that’s quite a lot of work. But if we’re to consolidate the tag specs, I think it’s what needs doing.

The alternative, which I’d also be OK with accepting, is to create a new PEP, called something like “Compatibility tag support in package installers” which is purely a spec of how installers must list what tags they support, and which acts as a supplement to all of the existing compatibility tag specs. That would make it essentially just an implementation-agnostic spec of what packaging.tags.sys_tags() returns. That could go through the PEP process as an independent spec, and we could ignore the need to consolidate the specs for a while longer :slightly_smiling_face:

==

Because when I researched and consolidated all the tag code out there no one could give me a motivation for what a CPython-specific wheel that didn’t require a specific ABI was meant for (and neither could PyPI; I think I found like 3 of those sorts of wheels).

This has come up often enough that I have given up arguing against it, but no one has cared enough to put the tag back in either :slight_smile: . And if anyone does decide to put it back they will have to figure out in what priority location does it go back in (e.g. what’s a better fit: cp36-abi3-* or cp38-none-*?).

See above for the cp*-none-* tags being left out. As for cp37-cp39-*, that’s simply not possible. The CPython 3.7 release does not have CPython 3.9 ABI compatibility (nor does any other release; CPython-specific ABIs are only compatible with that specific release; hence why abi3 was created).

To be clear, are you saying that packaging & cryptography are using this incorrectly? Cryptography’s cp36 wheels are for >= 3.6.

(I’m specifically talking about the Python tag here, not the ABI tag - though I admit I could have made this clearer given that they use a very similar format.)

If cp36 means “for Python == 3.6” then that would imply that packaging.tags should not include cp36-abi3-win_amd64 in the tags for Python 3.9. But I interpret @brettcannon’s reply as meaning cp36 is matched using == against the supported tags, but you can’t view that in isolation, you also need to consider that CPython 3.9 considers cp36 as compatible.

And as a practical point, if cryptography is tagging its wheels wrongly, what tags should it use? There’s no cp3 tag, and clearly it’s impossible to tag as cp36.cp37.cp38… because you can’t know all future tags. And publishing one wheel per Python version is unnecessary duplication, and precisely what having a stable ABI is intended to remove the need for.

I call practicality vs purity on this - even if people think it looks weird, the current behaviour where cp36 wheels are accepted by all later versions of Python satisfies an important use case. Maybe we wouldn’t be having this discussion at all if we’d used cp36+ rather than cp36 in the first place. But it’s too late for that, and honestly it would be a minor improvement at best.

Taking it back to the original point here, which is pynsist not recognising cryptography wheels, my view is that pynsist is wrong here, and it should accept the cryptography wheels. The correct behaviour is as implemented in packaging - pynsist can use that and not have a problem. If pynsist prefers to implement its own logic, it needs to replicate what packaging does - that should be documented independently of the packaging source code, but unfortunately it isn’t at the moment. We can remedy that, certainly, but we shouldn’t be trying to invent a different set of rules here - the logic in packaging is long-established, and has proved effective in practice. Changing that logic has serious backward compatibility risks, and no obvious benefit. If documenting the rules is hard, I’m fine with them being defined by the implementation in packaging for now, until someone has the energy to write them down in a spec.

Hi there.

For completeness I assumed there could be a cpxx-none-any. We don’t know what a cpython specific no-abi wheel could be or how to write a transpiler to jython compatible code or whatever.

But if you figure it out the tag is waiting for you.

In the meantime cp36 means “and newer cpython”

Actually that’s backwards. The wheel knows it works on cp36. Future Python decides whether it can still run cp36 code.

There is no “cp36 and not newer” interpreter tag. The abi tag has always taken care of that.

You could publish additional cp37-* wheels of you were really worried about the wrong interpreter getting the older ones.

Wheel tags have the fun feature of being not intuitive. Not a bug though.

2 Likes

Assuming @pf_moore is right about how I should understand @brettcannon’s statement (Brett, please confirm that, because I’m still a bit confused), it seems like no-one really wants to argue that a cp36 tag is specific to CPython ==3.6. @uranusjr thought that might have been the intention (as did I, evidently ;-), but I don’t see anyone saying that we should try to assert that meaning & change packaging.

If no-one contradicts this in a couple more days, I’ll change Pynsist to implement the >= meaning.

I hope to get round to clarifying it in the spec as well, if that’s possible without too much work.

I guess something that used CPython’s C API without compiled code, using ctypes or similar, might qualify? Or something that compiled code against the C API at runtime. In principle, you could use it even for something that was pure Python but relied on CPython specific features, e.g. if it really needed ref counting. Not major use cases, but I think they roughly fit with a cp36-none-any wheel being compatible with newer CPython, in a similar way to abi3 wheels.

I think the key intuition is what @dholth said - cp36 means the wheel knows it works on CPython 3.6. It’s up to CPython 3.9 (or the installer, on the implementation’s behalf), to say whether it can still run cp36 wheels - the tag spec is silent on that question. That knowledge (about what older versions newer Pythons can run) is currently captured in packaging. But using a >= test is sufficient here (and hopefully won’t ever be wrong), so you should be fine with that if you want to avoid depending on packaging.

This just feels like a rhetorical dodge, to be honest. Whether the wheel works on Python 3.9 is largely up to what’s in the wheel. Every Python release has a list of minor changes which will, in very specific cases, break code that worked on an earlier Python. You can’t definitively say that cp36 code is or is not compatible with CPython 3.9.

So we have a heuristic (a cp36 distribution will be used on CPython >=3.6, <4). But it’s important that this is written down and different tools consuming wheels work the same way. Saying “it’s up to the installer” what a tag means doesn’t really help anyone, because there are real practical problems if they use different meanings.

Yeah, but that’s basically the limitation to software versioning. All a cp36 wheel can claim is it works on CPython 3.6 (because it’s built on it), and whether a host CPython 3.9 works with a wheel built to run on CPython 3.6 can only be decided by that host. So for all practical purposes of pynsist specifically, I think something like >=3.6,<4 is the best you can do. If we’re going to codify this for installers in general, it can probably be something like

  • A cp36 language tag promises the wheel works on (is built against?) CPython 3.6.
  • Since CPython maintains langauge compatibility, an installer running on a newer CPython version (e.g. 3.8) should generally consider a cp36 wheel compatible for installation.
  • If a package is explicitly built against multiple langauge versions (not sure how, but theoratically it can), it should tag all the langauges it’s built against (e.g. cp36.cp37.cp38) so installers running on all those CPython versions know the wheel is compatible with their language versions.

Yes, it’s common to expect the tags to assert compatibility. We don’t really know if the wheel will work on CPython 3.9, or whether it will work at all. The important question is only “will the installer download the best candidate?”. Suppose you are choosing between a cp36-none-any wheel and the sdist. As long as the sdist is not somehow more compatible with CPython 3.9 than the wheel then it is acceptable to install the wheel.

That’s exactly what I meant: ==, but you have to remember that packaging.tags.sys_tags() spits out a lot of tags. So cryptography-3.4.6-cp36-abi3-win_amd64.whl is totally accurate and works for CPython 3.9 because cp36-abi3-win_amd64 is a valid tag for that interpreter version (and thus why packaging.tags.sys_tags() includes it).

Yep: cp36-abi3-* is specific, but newer versions of CPython happen to declare support for that tag as well.

This came up when we were cleaning up the tagging code and the answer given was that wheel tags are best-effort/more-than-likely compatibility. So there’s no promise things will work perfectly, but there’s at least a reasonable chance the wheel will work. Otherwise the expectation is that a better, tighter-fitting wheel will be provided to deal with any compatibility concerns.

To try to put all of this in code, this is roughly the expectation that an installer goes through when given a list of potential files to install for a project:

wheel_map = {}

for file_name in possible_wheels:
    _, _, _, tags = packaging.utils.parse_wheel_filename(file_name)
    for tag in tags:
        wheel_map[tag] = file_name

for tag in packaging.tags.sys_tags():
    if tag in wheel_map:
        print("Wheel found! 🎉", wheel_map[tag])
        break
else:
    print("No compatible wheel found 😢")

Notice how as long as the wheel tag shows up in packaging.tags.sys_tags(), it’s considered compatible. So while we mentally might all be thinking "cp36-abi3-* means >= cp36", from a programmatic/technical POV it’s ==.

I thought this was just another case of ‘specification needs to be clarified’, but it sounds like we’re actually saying that the meaning of tags is meant to be unspecified, and the packaging.tags module is officially blessed as defining the standard, not just the de-facto standard implementation. Is that right?

If that’s the case, I’ll stop trying to reimplement the logic, because that’s only a useful activity if we’re trying to have a written specification independent of implementations. I’ll also push for PEP 425 to point to packaging.tags, because if the official standard is an implementation, the thing that looks like a specification should probably say that. :wink:

I think it’s simply that the rules for saying what tags a given Python implementation supports have not been standardised (yet). We have an implementation in packaging.tags that represents the general consensus, and which works well in practice, but it’s not a standard. Anyone wanting to do the work of proposing an actual standard would be well advised to make the standard compatible with packaging.tags, but there’s no requirement for that.

Following from that, what I would say is that I consider changing PEP 425 to define packaging.tags as the actual standard as a substantive change to the spec, and as such couldn’t just be handled as a textual edit to the existing PEP, but would need to go through the PEP process in its own right - just as changes like new metadata fields or new PEP 517 hooks do.

2 Likes

I don’t think I’m likely to find the time to go through the PEP process any time soon, so I’ve copied & modified from packaging.tags for now.

(Copied rather than using directly, because looking at the code in packaging.tags, lots of places will silently default to checking the Python they’re running on, which is not what I want.)