Python tags - specific version of interpreter or minimum version?

[dholth (Daniel Holth)]

Yes, it’s common to expect the tags to assert compatibility. We don’t really know if the wheel will work on CPython 3.9, or whether it will work at all. The important question is only “will the installer download the best candidate?”. Suppose you are choosing between a cp36-none-any wheel and the sdist. As long as the sdist is not somehow more compatible with CPython 3.9 than the wheel then it is acceptable to install the wheel.

[brettcannon]

@pf_moore

If cp36 means “for Python == 3.6” then that would imply that packaging.tags should not include cp36-abi3-win_amd64 in the tags for Python 3.9. But I interpret @brettcannon’s reply as meaning cp36 is matched using == against the supported tags, but you can’t view that in isolation, you also need to consider that CPython 3.9 considers cp36 as compatible.

That’s exactly what I meant: ==, but you have to remember that packaging.tags.sys_tags() spits out a lot of tags. So cryptography-3.4.6-cp36-abi3-win_amd64.whl is totally accurate and works for CPython 3.9 because cp36-abi3-win_amd64 is a valid tag for that interpreter version (and thus why packaging.tags.sys_tags() includes it).

@pf_moore

I think the key intuition is what @dholth said - cp36 means the wheel knows it works on CPython 3.6. It’s up to CPython 3.9 (or the installer, on the implementation’s behalf), to say whether it can still run cp36 wheels - the tag spec is silent on that question.

Yep: cp36-abi3-* is specific, but newer versions of CPython happen to declare support for that tag as well.

@takluyver

Whether the wheel works on Python 3.9 is largely up to what’s in the wheel. Every Python release has a list of minor changes which will, in very specific cases, break code that worked on an earlier Python. You can’t definitively say that cp36 code is or is not compatible with CPython 3.9.

This came up when we were cleaning up the tagging code and the answer given was that wheel tags are best-effort/more-than-likely compatibility. So there’s no promise things will work perfectly, but there’s at least a reasonable chance the wheel will work. Otherwise the expectation is that a better, tighter-fitting wheel will be provided to deal with any compatibility concerns.

To try to put all of this in code, this is roughly the expectation that an installer goes through when given a list of potential files to install for a project:

wheel_map = {}

for file_name in possible_wheels:
    _, _, _, tags = packaging.utils.parse_wheel_filename(file_name)
    for tag in tags:
        wheel_map[tag] = file_name

for tag in packaging.tags.sys_tags():
    if tag in wheel_map:
        print("Wheel found! 🎉", wheel_map[tag])
        break
else:
    print("No compatible wheel found 😢")

Notice how as long as the wheel tag shows up in packaging.tags.sys_tags(), it’s considered compatible. So while we mentally might all be thinking “cp36-abi3-* means >= cp36”, from a programmatic/technical POV it’s ==.

[takluyver (Thomas Kluyver)]

I thought this was just another case of ‘specification needs to be clarified’, but it sounds like we’re actually saying that the meaning of tags is meant to be unspecified, and the packaging.tags module is officially blessed as defining the standard, not just the de-facto standard implementation. Is that right?

If that’s the case, I’ll stop trying to reimplement the logic, because that’s only a useful activity if we’re trying to have a written specification independent of implementations. I’ll also push for PEP 425 to point to packaging.tags, because if the official standard is an implementation, the thing that looks like a specification should probably say that.

[pf_moore (Paul Moore)]

I think it’s simply that the rules for saying what tags a given Python implementation supports have not been standardised (yet). We have an implementation in packaging.tags that represents the general consensus, and which works well in practice, but it’s not a standard. Anyone wanting to do the work of proposing an actual standard would be well advised to make the standard compatible with packaging.tags, but there’s no requirement for that.

Following from that, what I would say is that I consider changing PEP 425 to define packaging.tags as the actual standard as a substantive change to the spec, and as such couldn’t just be handled as a textual edit to the existing PEP, but would need to go through the PEP process in its own right - just as changes like new metadata fields or new PEP 517 hooks do.

[takluyver (Thomas Kluyver)]

I don’t think I’m likely to find the time to go through the PEP process any time soon, so I’ve copied & modified from packaging.tags for now.

(Copied rather than using directly, because looking at the code in packaging.tags, lots of places will silently default to checking the Python they’re running on, which is not what I want.)

[brettcannon]

@takluyver

looking at the code in packaging.tags , lots of places will silently default to checking the Python they’re running on, which is not what I want.

This might somewhat tie into plugin system for tags · Issue #161 · pypa/packaging · GitHub. Various people have talked about wanting to make sure the API in packaging.tags let one specify a different platform’s details, so we can either try to do it in ‘packaging’ first, or if you go ahead and do this, Thomas, please keep notes so we can have an informed decision about this later.

[takluyver (Thomas Kluyver)]

I think that issue is actually kind of the opposite, in that it’s about adding another way to influence the tags considered compatible for the current Python, whereas I’m trying to be independent of that.

I think that what I want is actually entirely possible with packaging.tags. But if you forget to pass some information (or maybe if functions grow new arguments in the future?), you get values based on your current interpreter. E.g. if I call cpython_tags(python_version=(3, 7), platforms=['win_amd64']) without abis=, it creates the ABI tag based on the running Python (and the specified Python version). It will still gives the right answer most of the time, but that makes it hard to see the rare problem.

Of course selecting tags for the running Python is the typical use case, so it makes sense to cater to that. But for Pynsist, I want to ensure the tag selection is entirely consistent for given inputs, regardless of the interpreter running that code. Building a Windows installer from Linux is supported. And it’s easier to do that by cutting out the ‘inspect current interpreter’ code than by carefully passing parameters to step around that code.