Is there a canonical list of all possible PEP 425 tags (or patterns)?

For context, I have been looking for ways to derrive PEP 508 environment markers from a wheel’s python tag abi tag and platform tags.

From what I can tell, the packaging.tags implemenation is geared towards generating compatible tags for the active environment and does not provide a way to list all possible tags for all environments.

PEP 425 itself does not seem to list all possible values/patterns, but rather only identifies a subset as examples.

Should I try and build this list from using the packaging.tags source or is there a better way to apprach this? Or is this a futile attempt?

I was really hoping to find a canonical list somewhere, but failed to do that so far.

There is not. Python packaging does not predefine a list of supported platforms, but platforms declare themselves to support Python packaging. So packaging tools query the platform to get what tag to use (usually via packaging.tags as you already know), not the other way around.

You may also be intereted in the semi-related topic came up just yesterday: PEP 425 Platform tag

Thanks @uranusjr. I did notice that one yesterday, and I am following that. Definitely interested in that topic.

So, in theory I would have to know all the possible platforms out there in order to define a canonical list. Which is, well infeasible.

I guess the next best alternative is to build a list for well known platform values and then have some fallback for “unrecognised” platforms if encountered in the wheel’s tag. Is there any obvious reason why this might not work?

Why do you need a list of all possible tags? I’m having trouble thinking of an application which would need to know an exhaustive list.

This is in relation to being able to determine package requirements from platform specific wheels uploaded to PyPI.

In the case of poetry, when adding a new dependency to a project and PyPI metadata of the package is incomplete or deemed incorrect, we inspect the wheels to determine the package dependencies. This is straight forward for non platform specific wheels.

When platform specific wheels are identified, we need to mark any of the detected dependencies with platform specific tags, especially the ones that are only required for a particular platform.

Right now we fallback to attempting to use the sdist, but this means only the platform on which the lock file is generated is considered when determining dependencies. If we inspect the wheels and simply determine the requirements from the wheel, we also have an issue where we start requiring dependencies that may not be required for the platform on which this lock file is consumed.

I wanted to do the latter, but mitigate the issue by adding environment markers derrived from the wheel’s platform tag.

In order to do this, I need to be able to reliable map PEP 425 tag to PEP 508 environment markers. I wanted to avoid relying on too much “smarts” to do this mapping. Having a list of possible values means the mapping can be done 1:1 if required.

A reliable map from PEP 425 tags to PEP 508 markers is needed, but that does not mean you need all the possible tag values. Platform tags are ultimately formatted based on platform information, and it should be be possible to establish an algorithm to convert a tag back to the information which generated it.

Or, even better, maybe we can just add a new marker platform_tag (or some other name) that allows you to do something like

requests; platform_tag == 'macosx_10_9_x86_64'

This, plus putting the tag logic into the stdlib (so the value can be evaluated on install-time), would provide all that’s needed here.

I agree. And that is what I was about to do, but figured I would ask first. I guess I should just bite the bullet and do it.

If this is something that is feasible to do, it would definitely help. How does one go about doing this?

You’d ask in this forum (or I guess you can just modify the title of this thread to reflect the proposal so relevent people would be interested in reading), and submit a PEP at some point based on others’ feedback if that is indeed feasible (people here will tell you if it’s not).

1 Like

Not necessarily because a lot of it also comes straight from the OS/build itself, e.g. variables set when generating the Makefile. Like if you look at the generic tag support that’s used by PyPy you will see a lot of it is simply asking the interpreter to tell us something that was set at compile time which can ultimately be anything.

It kind of is already, but then kind of isn’t :wink:. For instance, even if we had the API to get this stuff it doesn’t won’t tell you what the validate tag triples are or what their priority order is. And then if you add that you have packaging.tags. :smile:

The idea of being able to use PEP 508 markers to specify a part of the tag seems like a nice solution to me.

Here I was thinking this would be simple. :joy:

I agree. Would be great to be able to do something like marker.platform_tag in map(str, packaging.tags.compatible_tags()) to check for this at install time.

@abn FYI, this is something I’ve thought a lot about for Hatch, especially during the last few months since I’m re-writing it.

A few weeks ago I decided that for v1, I’m not going to support cross-platform dependency locking, for 2 reasons:

  1. I don’t have a sense of what users want to happen when the resolving becomes too complex e.g. if a package only uploads an sdist mixed with PEP 621’s allowance of dynamic dependencies (basically back to the problem).
  2. I’m waiting for a lock file to be standardized in a PEP, which hopefully will shed more light on 1.

I beleive this will come back to the build backends really. This will always be a limitation if full metadata is not published by the package maintainer for all supported platforms of the package, because there is no reliable way to exhaustively determine cross-platform dependencies, especially in cases where these are determined at install time. Additionally, the platforms supported are also not finite as per this discussion. Even when dealing with a subset of known platforms, this information will have to be generated explicitly either by PyPI (which has been made clear is not going to happen, because resource investment is too high) or by a third-party service or in a distributed fashion.

Trying to solve all use cases before standardising a lockfile format is a bit like going down a rabbit hole. Much rather accept the limitations and cater to the majority use case.

That said, the intent here is not necessarily to look at the determining all platforms, but rather reliably being able to map the information that we receive from the wheel’s platform tag into a requirement specification such that it is imilated to this platform.