Python tags - specific version of interpreter or minimum version?

Packaging
1

TLDR: does a cp36 (or py36) Python tag in a wheel filename indicate it’s for Python == 3.6 or for >= 3.6?

PEP 425 is somewhat ambiguous about this, to my mind:

The Python tag indicates the implementation and version required by a distribution.
…
The version is py_version_nodot.

The example in the Use section suggests that generic Python py36 tags mean >= but CPython cp36 tags mean == - the list of compatible tags for installing on Python 3.3 includes py32 down to py30 but not cp32. But this distinction is not written explicitly anywhere.

This has come up because I have assumed the == meaning in Pynsist, but Cryptography (or packaging tooling it uses) seems to be releasing wheels assuming the >= meaning (all their wheels have a cp36 tag, but are meant to work on newer versions of Python). I’m getting bug reports about it. I imagine that pip must implement the >= option, otherwise Cryptography developers couldn’t do this. But e.g. PyQt5 uses cp36.cp37.cp38.cp39 to indicate compatibility with multiple versions of CPython.

Edit: to be clear, cryptography is not unique, there are some other smaller libraries doing the same thing. But most wheels with compiled parts (like NumPy) also specify the corresponding version-specific ABI tag, like cp36m, and have a separate wheel for each supported Python version. The ambiguity only arises for wheels with a less specific ABI tag (like abi3).

I’m happy enough to be told that I’ve got it wrong, but either way I think it’s worth clarifying this in the tags spec.

1 Like
2

The responsibility is on the consumer here. I’m not sure that was a good design, but it’s the reality we have.

Specifically, an installer has to list all of the compatibility tags that it will accept. The reference implementation is in packaging.tags, and that returns “older” tags, so for example, on my PC running Python 3.9, I get

>>> [str(t) for t in sys_tags()]
[
    'cp39-cp39-win_amd64',
    'cp39-abi3-win_amd64',
    'cp39-none-win_amd64',
    'cp38-abi3-win_amd64',
    'cp37-abi3-win_amd64',
    'cp36-abi3-win_amd64',
    'cp35-abi3-win_amd64',
    'cp34-abi3-win_amd64',
    'cp33-abi3-win_amd64',
    'cp32-abi3-win_amd64',
    'py39-none-win_amd64',
    'py3-none-win_amd64',
    'py38-none-win_amd64',
    'py37-none-win_amd64',
    'py36-none-win_amd64',
    'py35-none-win_amd64',
    'py34-none-win_amd64',
    'py33-none-win_amd64',
    'py32-none-win_amd64',
    'py31-none-win_amd64',
    'py30-none-win_amd64',
    'py39-none-any',
    'py3-none-any',
    'py38-none-any',
    'py37-none-any',
    'py36-none-any',
    'py35-none-any',
    'py34-none-any',
    'py33-none-any',
    'py32-none-any',
    'py31-none-any',
    'py30-none-any'
]

Note that cp36-abi3-win_amd64 is included - that’s what determines whether pip on my PC will install a cp36 wheel, rather that cp36 somehow implying “any version after 3.6 as well”.

I suspect the simplest solution for you is to use packaging.tags in pynsist. But I can completely understand if you feel uncomfortable that the knowledge is buried in the implementation of packaging.tags - that seems contrary to the drive to make sure that we rely on standards rather than implementations :frowning:

2 Likes
3

I’m quite happy if we codify the de-facto standard from pip/packaging into PEP 425. In many ways, that’s the best answer for me - it’s much easier to fix something in my own code than chase around trying to change what everyone else does. But I’d like to get consensus and then adjust the spec to make this unambiguous.

From your list: is it expected that (for instance) cp38-abi3 is there but cp38-none is not? Surely if wheels which use the stable ABI can be used on a newer Python, wheels which don’t rely on the ABI at all should be equally acceptable?

Part of the reason I do reimplement stuff like this is that I think it’s worthwhile to find and clear up these ambiguities so we have useful specs rather than just relying on what a favoured implementation does.

1 Like
4

I have no idea TBH. It’s what packaging does, so it’s the reality we have to deal with. As far as I know, the logic isn’t documented anywhere.

@brettcannon did most of the work codifying the existing practice for the packaging implementation, maybe he can clarify some of the reasons behind how things work.

5

This was not an issue before abi3 existed because CPython did not provide ABI compatibility across version anyway, so (say) cp37-cp37m would always only support CPython 3.7 no matter how the Python version part is interpreted. In the compatibility sense, my intuition is cp37 should imply >=3.7 (and a virtual upper cap where Python breaks syntax backwards compatibility, like how py2 does not cover Python 3.x).

Assuming this interpretation, the next question would be how a wheel can say “just Python 3.7, not later”. Is this semantic needed?

6

Why would the wheel need to do that? The wheel should say “I can be used on anything that claims it’ll handle cp37 wheels” and it’s up to the target environment (technically, the installer for that environment) to say if it can deal with those wheels. What’s the use case for saying “please ignore any claims from the target that it supports cp37 wheels, and only install on…” what, exactly? On things that claim cp37 support and don’t report a Python version other than 3.7? You can do that with Requires-Python, but I’m still unclear why you would.

7

As a matter of implementation, yes, it’s up to the tool consuming the wheel to decide what it accepts. But the meaning of the cp37 tag - what environments it is compatible with in principle - is defined by the spec. Just, in this particular detail, not very precisely defined. :slightly_smiling_face:

8

Good point. My recollection is that it’s vague because at the time we were originally discussing this, it wasn’t at all clear what the right answer should be (would PyPy, which had its own version numbering, be able to support cp37 wheels, for example). Now, we’ve got a lot more experience and the answers to questions like this seem to have settled down (at least for now :wink:). But we have backward compatibility to contend with now, which we didn’t then.

I don’t think it’s possible to standardise cp37 as meaning anything other than “runs on CPython versions 3.7 and later”. But wording that needs some care - we don’t know that CPython 4.0 (or even 3.12) will still be compatible with cp37 wheels.

And there’s also the more immediate oddity around how platform and ABI interact, too - the list I showed above included cp39-cp39-win_amd64 and cp39-none-win_amd64, but not cp37-cp39-win_amd64, cp37-cp37-win_amd64 or cp37-none-win_amd64. Why? I’ve no idea. But that demonstrates that cp37 doesn’t simply mean “CPython 3.7 or later”…

Binary compatibility is hard. @ncoghlan used to make that point a lot, and he’s far more knowledgeable about the area than I am :slightly_smiling_face:

As I said, I’d support getting this written down as a standard that documents what packaging.tags does, in a way that explains the logic so that it can be extrapolated to future releases and implemented by other libraries. But so far no-one has been sufficiently motivated to do so.

PS I should also say that I’m really pleased that Pynsist, by having its own tags implementation, triggered this discussion. It’s a great example of why everyone relying on a single reference implementation is bad, because we then don’t question the details.

1 Like
9

Reading PEP 425 again:

Importantly, major-version-only tags like py2 and py3 are not shorthand for py20 and py30. Instead, these tags mean the packager intentionally released a cross-version-compatible distribution.

My interpretation is this implies py20 is not cross-version-compatible, i.e. does not claim to work on Python 2.1 or later. So maybe the intention is actually to make cp37 mean just CPython 3.7? A tool installing into CPython 3.8 can claim the target interpreter is compatible with cp37 if it wants to, but the tag itself does not claim CPython 3.8 compatibility.

10

Whatever the intention was when PEP 425 was written, I think it’s most practical now to codify how packaging has interpreted it, unless there’s a really solid reason that that needs to change.

Wheels for CPython 3.7 specifically can be tagged with cp37-cp37 (i.e. using the ABI tag to make it more specific). Wheels on PyPI or other (PEP 503) indexes can also be selected using the requires-python metadata, which is more flexible and descriptive.

1 Like
11

In keeping with what we’ve done for other spec changes (e.g. escaping wheel filename components), I guess we should copy the content of PEP 425 to the appropriate spec page on packaging.python.org, and then propose changes there?

12

Generally, we need a PEP revision that notes the new location is now the canonical definition, followed by future changes being made as PRs to that location.

However, please note that this does not mean that we have the discussion on the new wording on the PR. There’s still a requirement for consensus on the change to be agreed here. See here for the specifics of the process (distutils-sig should be replaced by Discourse nowadays…) It’s also worth noting the following section which prohibits backwards-incompatible changes.

The key point is

If a change being considered this way has the potential to affect software interoperability, then it must be escalated to the distutils-sig mailing list for discussion, where it will be either approved as a text-only change, or else directed to the PEP process for specification updates.

I’ve generally been relatively relaxed about not insisting on the PEP process for small changes, mostly because people tend to perceive the PEP process as bureaucratic (even though it needn’t be in practice).

In this case, though, I feel like writing up the details will be a fairly difficult process, so I’m inclined to say that if we want to consolidate the tag specs on packaging.python.org as part of adding this information, we should have a PEP that:

  1. Defines the rules for how installers should declare what tags they accept, as a supplement to PEP 425.
  2. States that the canonical location for the packaging tags specification will be moved to packaging.python.org.

The PEP should link to a PR that updates the tags spec page here. Note that this will be a non-trivial rewrite (it certainly won’t just be a “lift and shift” of PEP 425) because that page links to the various manylinux specifications as well as to PEP 425, but it doesn’t clearly state (for example) that py3-none-any is a valid tag on Linux - so it’s not (IMO) currently in a state that would qualify it as the definitive spec for tags… I’d consider that PR as a required part of the PEP, as it’s a relatively complex rewrite.

Sorry - that’s quite a lot of work. But if we’re to consolidate the tag specs, I think it’s what needs doing.

The alternative, which I’d also be OK with accepting, is to create a new PEP, called something like “Compatibility tag support in package installers” which is purely a spec of how installers must list what tags they support, and which acts as a supplement to all of the existing compatibility tag specs. That would make it essentially just an implementation-agnostic spec of what packaging.tags.sys_tags() returns. That could go through the PEP process as an independent spec, and we could ignore the need to consolidate the specs for a while longer :slightly_smiling_face:

13

==

Because when I researched and consolidated all the tag code out there no one could give me a motivation for what a CPython-specific wheel that didn’t require a specific ABI was meant for (and neither could PyPI; I think I found like 3 of those sorts of wheels).

This has come up often enough that I have given up arguing against it, but no one has cared enough to put the tag back in either :slight_smile: . And if anyone does decide to put it back they will have to figure out in what priority location does it go back in (e.g. what’s a better fit: cp36-abi3-* or cp38-none-*?).

See above for the cp*-none-* tags being left out. As for cp37-cp39-*, that’s simply not possible. The CPython 3.7 release does not have CPython 3.9 ABI compatibility (nor does any other release; CPython-specific ABIs are only compatible with that specific release; hence why abi3 was created).

14

To be clear, are you saying that packaging & cryptography are using this incorrectly? Cryptography’s cp36 wheels are for >= 3.6.

(I’m specifically talking about the Python tag here, not the ABI tag - though I admit I could have made this clearer given that they use a very similar format.)

15

If cp36 means “for Python == 3.6” then that would imply that packaging.tags should not include cp36-abi3-win_amd64 in the tags for Python 3.9. But I interpret @brettcannon’s reply as meaning cp36 is matched using == against the supported tags, but you can’t view that in isolation, you also need to consider that CPython 3.9 considers cp36 as compatible.

And as a practical point, if cryptography is tagging its wheels wrongly, what tags should it use? There’s no cp3 tag, and clearly it’s impossible to tag as cp36.cp37.cp38… because you can’t know all future tags. And publishing one wheel per Python version is unnecessary duplication, and precisely what having a stable ABI is intended to remove the need for.

I call practicality vs purity on this - even if people think it looks weird, the current behaviour where cp36 wheels are accepted by all later versions of Python satisfies an important use case. Maybe we wouldn’t be having this discussion at all if we’d used cp36+ rather than cp36 in the first place. But it’s too late for that, and honestly it would be a minor improvement at best.

Taking it back to the original point here, which is pynsist not recognising cryptography wheels, my view is that pynsist is wrong here, and it should accept the cryptography wheels. The correct behaviour is as implemented in packaging - pynsist can use that and not have a problem. If pynsist prefers to implement its own logic, it needs to replicate what packaging does - that should be documented independently of the packaging source code, but unfortunately it isn’t at the moment. We can remedy that, certainly, but we shouldn’t be trying to invent a different set of rules here - the logic in packaging is long-established, and has proved effective in practice. Changing that logic has serious backward compatibility risks, and no obvious benefit. If documenting the rules is hard, I’m fine with them being defined by the implementation in packaging for now, until someone has the energy to write them down in a spec.

16

Hi there.

For completeness I assumed there could be a cpxx-none-any. We don’t know what a cpython specific no-abi wheel could be or how to write a transpiler to jython compatible code or whatever.

But if you figure it out the tag is waiting for you.

In the meantime cp36 means “and newer cpython”

Actually that’s backwards. The wheel knows it works on cp36. Future Python decides whether it can still run cp36 code.

There is no “cp36 and not newer” interpreter tag. The abi tag has always taken care of that.

You could publish additional cp37-* wheels of you were really worried about the wrong interpreter getting the older ones.

Wheel tags have the fun feature of being not intuitive. Not a bug though.

2 Likes
17

Assuming @pf_moore is right about how I should understand @brettcannon’s statement (Brett, please confirm that, because I’m still a bit confused), it seems like no-one really wants to argue that a cp36 tag is specific to CPython ==3.6. @uranusjr thought that might have been the intention (as did I, evidently ;-), but I don’t see anyone saying that we should try to assert that meaning & change packaging.

If no-one contradicts this in a couple more days, I’ll change Pynsist to implement the >= meaning.

I hope to get round to clarifying it in the spec as well, if that’s possible without too much work.

I guess something that used CPython’s C API without compiled code, using ctypes or similar, might qualify? Or something that compiled code against the C API at runtime. In principle, you could use it even for something that was pure Python but relied on CPython specific features, e.g. if it really needed ref counting. Not major use cases, but I think they roughly fit with a cp36-none-any wheel being compatible with newer CPython, in a similar way to abi3 wheels.

18

I think the key intuition is what @dholth said - cp36 means the wheel knows it works on CPython 3.6. It’s up to CPython 3.9 (or the installer, on the implementation’s behalf), to say whether it can still run cp36 wheels - the tag spec is silent on that question. That knowledge (about what older versions newer Pythons can run) is currently captured in packaging. But using a >= test is sufficient here (and hopefully won’t ever be wrong), so you should be fine with that if you want to avoid depending on packaging.

19

This just feels like a rhetorical dodge, to be honest. Whether the wheel works on Python 3.9 is largely up to what’s in the wheel. Every Python release has a list of minor changes which will, in very specific cases, break code that worked on an earlier Python. You can’t definitively say that cp36 code is or is not compatible with CPython 3.9.

So we have a heuristic (a cp36 distribution will be used on CPython >=3.6, <4). But it’s important that this is written down and different tools consuming wheels work the same way. Saying “it’s up to the installer” what a tag means doesn’t really help anyone, because there are real practical problems if they use different meanings.

20

Yeah, but that’s basically the limitation to software versioning. All a cp36 wheel can claim is it works on CPython 3.6 (because it’s built on it), and whether a host CPython 3.9 works with a wheel built to run on CPython 3.6 can only be decided by that host. So for all practical purposes of pynsist specifically, I think something like >=3.6,<4 is the best you can do. If we’re going to codify this for installers in general, it can probably be something like

  • A cp36 language tag promises the wheel works on (is built against?) CPython 3.6.
  • Since CPython maintains langauge compatibility, an installer running on a newer CPython version (e.g. 3.8) should generally consider a cp36 wheel compatible for installation.
  • If a package is explicitly built against multiple langauge versions (not sure how, but theoratically it can), it should tag all the langauges it’s built against (e.g. cp36.cp37.cp38) so installers running on all those CPython versions know the wheel is compatible with their language versions.