Absolutely! I expect to publish a PR tomorrow.
1 Like
The PR for the updated PEP itself is now available, together with its preview rendering.
4 Likes
@steve.dower, @pf_moore, does the updated version work for you?
I’ll be honest, I find the way the sys.abi_info data is “backported” to versions before 3.15 to be very confusing. Why do the attributes introduced in Python 3.15 get this special treatment, but something introduced in Python 3.17 doesn’t? Or is the intention that we do the same sort of “backport” for any new attribute added in future? If we have to define how to calculate sys_abi_info values for older versions of Python, I don’t really see what advantage there is in having sys.abi_info in the first place. A simple list of “available features” didn’t have this problem - any newly introduced features simply aren’t there in older versions.
I’m not sure I see how pointer_bits is usable, given that it can in theory take an arbitrary value. For all practical purposes, if it’s ever anything other than 32 or 64, any marker specs that use it will almost certainly fail to work correctly - so what’s the point in having that flexibility if it can’t be used in practice? (The byteorder attribute doesn’t have this issue - it states that there are only two valid values, so it’s the same as a boolean attribute, just spelled differently.)
1 Like
It’s also almost certainly encoded in the platform tag, and it’s incredibly unlikely that you’d enable a dependency for both x86-64 and ARM64 but disable it for x86. But I can see the completeness reason for including it.
I think in this case, we’d only be adding new features because they don’t currently exist, whereas the things that are going in now were already discoverable through a range of methods. It’s kind of like how PEP 514 defined logic for default values for our pre-3.5 releases that didn’t include it. But it does mean that anything new we add has to have a logical interpretation for “absent”.
1 Like
Agreed, from a packaging point of view, it’s probably pointless. But like you, I can see that making it accessible for other uses is fine. The real question is if other attributes that can assume arbitrary values get added, they probably won’t work well as environment markers, for the reasons I stated above.
The problem is that there’s a tension between the packaging use case (where boolean “available or not” states work best) and the general “provide useful information” case. Maybe the PEP should say that only boolean-valued attributes will be exposed as environment markers? We’d lose byte_order, but I can’t see anyone caring that much…
OK, that makes sense, but the PEP should probably explain that more clearly. How about moving that whole discussion into the “Backward Compatibility” section?
For Python versions before 3.15, the
sys.abi_infoattribute did not exist. In order to allow expressing the conditions this PEP enables on older versions of Python,sys_abi_infowill be calculated as if asys.abi_infovariable existed with only the following attributes:free_threaded,debug
(I’d argue that there’s little point “backporting” endianness or pointer length.)
Agreed, and that should be documented (in the PEP at least). Or we could go with my suggestion that we only include boolean values, and then the interpretation of “absent” is obvious (False). We could even go back to just a set of feature names, only including boolean features with a value of True in the set.
1 Like
My assumption ts that new attributes would get the same “backport”, if they are available and make sense on older Python versions.
In fact, the design of abi_info intentionally allows this. The initial proposal called for a set of flags like 32-bit, 64-bit or free-threading; the attributes with values allow an “unknown” state. IMO, the PEP should provide “fallbacks” on the attribute level, so different interpreters (and CPython versions) can leave stuff out if needed.
Thanks for the feedback! It seems there are two main concerns, namely the backporting of information to Python versions older than 3.15, and the use of arbitrary values, particularly in the pointer_bits example.
I think both are affected by the mechanics proposed in the PEP and the reference implementation, which we need to explain better, so let me clarify two points here.
First, note that the reference implementation targets pypa/packaging, which is where the parsing and handling of environment markers lives. That means that the connection between Python versions and the PEP is necessarily a bit fuzzy, as Packaging is not likely keen to introduce 3.15 as a lower limit on the supported Python version.
Second, the implementation translates every field name with value value in the sys.abi_info namespace into a single string f"{name}::{value}", collects these strings in a frozen set and in this way allows for tests of the form "pointer_bits::64" in sys_abi_info. This means that every test is a boolean and that the author of the specs is free to decide the desired default behavior in case of absence of the bit of abi information to a degree by choosing a not in test, though we could also augment the syntax to allow for example "pointer_bits" in sys_abi_info to test only for presence of the name with no impact of the value.
These two points in mind, let me address the two concerns brought up.
On the question of backporting to Python versions older than 3.15, I think it is clear that Packaging needs to do something as the situation that a new version of Packaging gets installed on an older version of Python is highly likely.
We could say that in this case simply no field should be present, which would simplify the implementation and the explanation. However, as @steve.dower pointed out, all the information presently in the sys.abi_info field can be determined by other means via the C API, though, crucially, not in the context of environment markers, which in turn means that it is relatively easy to provide this added benefit also for older Python versions. Still, if we come down on the side of cleanliness, we could remove the section on backporting and the corresponding implementation part, though I personally think that the advantage particularly for one of the initial motivations of supporting early adoption of free threading outweighs the downside of adding this.
On the question of pointer_bits and other arbitrary values, I hope that my explanation on the mechanics above have convinced you that they pose no fundamental problem as every value translates into a simple string allowing for tests of equality and non-equality, though a pure test on presence may be added.
The particular example of pointer_bits was added because with the current environment markers it is not possible to determine whether one is on a win64 or win32 Python, both of which are widely used, where popular packages like scipy are only available for one of the two. Hence, the situation arises where, for example, scipy is an optional dependency and should be omitted where not available due to this limitation.
1 Like
The problem with this motivating example is that pointer_bits isn’t sufficient to distinguish between amd64 and arm64 Windows, so you can’t actually use it when you need to filter by platform, and can only use it to filter pure-Python packages that rely on the size of the machine word but without using any native code (e.g. they use ctypes against system libraries).
More crucially, you determine it in different ways for each item on each platform, which means you get into very complex code with a lot of conditionals, for things that are trivially known at CPython compile time.[1] Simply storing the values directly and making them available at runtime (or through the build-details.json) is long-term the better way.
And presumably other implementations’ compile time, but I’m not going to speak for them on this point. ↩︎
1 Like
It’s not something that the packaging library needs to do, it’s something the PEP has to cover. You cannot assume that all tools use packaging - for example, uv is implemented in Rust, and will need to make their own implementation of the rules in this PEP.
Ultimately, if we plan on having rules for how fields that are in sys.abi_flags get calculated for Python versions before they were introduced into sys.abi_flags, those rules will need to be documented in the packaging specs, and those specs will need to be kept up to date as sys.abi_flags changes. Who do you expect to do that? I don’t think we can realistically assume that the core developers will take on that responsibility.
Furthermore, even if everything is handled perfectly, you still have to cater for the possibility that a user will be using a tool that contains an older implementation of the marker logic. The “backward compatibility” section of the PEP doesn’t cover that at all. Suppose for instance that a project declares a dependency on foo; "field::value" in sys_abi_features. There are two situations that need to be considered here:
- The most serious one, this project will not be installable with tools that use a version of the marker spec that predates PEP 780. If the project is a popular one, that could be a major disruption to the ecosystem. Do we plan on releasing the new spec, but telling projects to not use the new features until everyone is on a suitably up to date version of pip, uv, packaging, etc, etc.? How do we know when that will be the case? And how do we prevent people from ignoring our advice? Once a package using the new spec is released to PyPI, it’s near impossible to retract it.
- Less serious, but also important, if
fieldwas introduced after PEP 780 is in common use, there will inevitably be users who are using tools that haven’t updated to a marker implementation that knows about the new field. They will see that field as missing in Python 3.14. We need to ensure that the documentation explains clearly that when creating marker expressions, projects need to allow for that possibility (use"field::value" in sys_abi_featuresif you want to default to “no” if the field is missing, but use"field::value" not in sys_abi_featuresif you want to default to “yes”). That sort of logic will be hard to follow, and easy to get wrong.
So while I’m fine with the idea that we add words to the spec explaining how to “manufacture” the correct sys_abi_features values for Python versions before 3.15, you don’t actually gain anything in terms of compatibility by doing so[1].
I’m afraid not. Are you sure there will be no need for a marker expression that says “pointers must have 64 bits or greater”?
All your current proposal does is convert sys.abi_flags.pointer_bits into a set of boolean flags in sys_abi_features: pointer_bits::32 and pointer_bits::64. But how is the user meant to know that there isn’t a pointer_bits::128? Or pointer_bits::36?
I feel like this is a feature with no real use case, which has been forced on us because the core devs didn’t accept the original proposal where ABI features were all boolean. You mentioned the case of scipy, but I’m not convinced. Without seeing that example worked through fully, it feels more like handwaving to justify the feature, rather than the feature having been designed to address the use case. Steve pointed out that pointer_bits can’t even be used to distinguish between amd64 and arm64 Windows, so it sounds like even when it might have been useful, it isn’t enough.
If the PEP contained explicit details on how to calculate the values for Python versions earlier than 3.15 (something which I pointed out above is required before the PEP can be used in practice) then this issue would presumably be glaringly obvious. Which is another reason I want to see a lot more precision in the proposal than we currently have[2].
3 Likes
To add a concrete example, uv fails when trying to install a wheel with an unknown marker:
$ uv pip install dist/debug-0.1.0-py3-none-any.whl
Using Python 3.14.0rc2 environment at: /home/konsti/projects/uv/.venv
× Failed to read `debug @ file:///home/konsti/projects/uv/debug/dist/debug-0.1.0-py3-none-any.whl`
├─▶ Couldn't parse metadata of debug-0.1.0-py3-none-any.whl from debug @ file:///home/konsti/projects/uv/debug/dist/debug-0.1.0-py3-none-any.whl
╰─▶ Expected a quoted string or a valid marker name, found `sys_abi_features`
tqdm; 'free-threading' in sys_abi_features
^^^^^^^^^^^^^^^^
I’m very much in favor of adding new markers, we to find some way to being able to add new markers, and how tools lacking those markers should behave.
I see this problem as python version independent, as currently e.g. dependencies = ["tqdm; 'free-threading' in sys_abi_features"] means that sys_abi_features needs to be in Requires-Dist verbatim.
3 Likes
I’m also interested in this problem, since it will affect wheel variants, and possibly other future specs that attempt to add new environment markers. That said, I’m not sure if this doesn’t deserve a topic of its own, given it’s not really specific to PEP 780, but it seems to be a general limitation of how environment markers were implemented.
So does pip, through packaging. What’s particularly bad about both cases is that you basically get a syntax error rather than a suggestion that you need to upgrade the installer.
There’s also a related problem: given that pip vendors its own packaging version, the user may upgrade pip and therefore be able to install a wheel using the new markers, but be left with old packaging or other tools that will explode when processing the metadata from installed packages. It’s a “non-obvious” case, and I think in most of the cases it won’t be handled gracefully.
The problem could partially be worked around by including the new markers only when they are actually applicable to the wheel in question. For example, the free_threaded::true could be omitted from, say, abi3 wheels. However, FWIU this goes against universal resolution which demands that metadata is consistent across all wheels.
It may well do. But that would mean putting PEP 780 on hold until we solve the bigger issue. I’m hesitant about blocking progress while we spend our time tackling fundamental issues, if we can make useful progress on an incremental basis.
On the other hand, we do need to look at the big picture, and if we keep working on individual features we’re just ignoring the technical debt. I’d like to know what the community consensus is here - keep progressing and work on technical debt when we can, or face the large scale blockers and tackle them now, to set ourselves up better for the long term?
2 Likes
I’m sorry, I didn’t mean to block it. Just were wondering if we should discuss it in this thread or separately from it. In fact, I think it would be great if PEP 780 would set a precedent for handling this.
Let’s continue to discuss here. If it becomes a big enough issue we can spin off a separate thread.
I wasn’t suggesting you were blocking anything, it’s just unfortunate that this PEP needs features the current design doesn’t support well. We have to find some solution for this PEP, and it would be good if the solution was general enough to cater for future PEPs as well. We can decide whether we want to compromise with a “good enough for now” solution when we have a better idea what the options are.
1 Like
I don’t have a clear plan yet, but this is critical for both this PEP and wheel variants.
One option is to separate new and old metadata into separate fields by introducing a new field Dependency. Requires-Dist remains backwards compatible and uses only PEP 508 markers.
Requires-Dist: anyio
Requires-Dist: numpy; sys_platform == 'win32'
Dependency: anyio
Dependency: tqdm; 'free-threading' in sys_abi_features
Dependency: numpy; sys_platform == 'win32' or 'free-threading' not in sys_abi_features
The PEP 621 rules are amended so that Requires-Dist must only be a copy of project.[optional-]dependencies for entries that use only PEP 508 markers. For entries that uses non-PEP 508, the entry may not appear in Requires-Dist at all or may appear with a modified marker that is PEP 508 compatible. Dependency parsers must ignore unknown fields, evaluating them to false.
This scheme is designed for backwards compatibility with wheels. URL dependencies and source distributions, for which tools read project.[optional-]dependencies are not supported.
2 Likes
I think that’s an interesting solution. Presumably the build backend would be responsible for translating Dependency entries into Requires-Dist? I would personally also lean towards requiring the translation to explicitly follow the same rule as Dependency processing for unknown markers — i.e. always assume false in this case. Otherwise, we may end up with pre-Dependency installers installing different set of dependencies than an installer supporting Dependency but not any of the non-PEP 508 markers markers used in it.
I think it also would make sense to ponder a bit about the rule for unknown markers. “False” is better than an error, I guess, but it could be confusing to end users and results in logical contradiction — i.e. where 'free-threading' in sys_abi_features and 'free-threading' not in sys_abi_features both evaluate to false. Perhaps it would make sense to include some way of declaring defaults for when the evaluate is unsupported or querying whether it is supported, e.g.:
tqdm; 'free-threading' in sys_abi_features or 'sys_abi_features' not in supported_markers
numpy; sys_platform == 'win32' or 'free-threading' not in sys_abi_features
which would mean tqdm is installed if sys_abi_features is not supported (i.e. default “true”), but numpy would not (default “false”).
A key question for this design: Is it acceptable if old tools (with strict PEP 508 support) ignore releases with new markers entirely (considering them incompatible) or do we need a design that supports both old tools and new markers in the same version?
Just to recall some syntax, after the core addition, the marker should now be called sys_abi_info, even though individual bits of information may be referred to as features. Also recall that the proposed syntax is "{name}::{value}" [not] in sys_abi_info, i.e. the value part is mandatory to detect True or False, giving a clear candidate syntax of "{name}" [not] in sys_abi_info to detect presence of the feature if we want to add this to the spec.
@pf_moore, I do see the point about future Python releases. At the same time, I think it’s not too hard to add fields where it is already possible.
How about instead of tying it to the Python version, like 3.15 or earlier, we spell out a list of core features that must be supported.
Then it is up to the implementation (Pip, Uv, …) to provide that and we can add a comment to the relevant source in cpython to please consider updating the list if an addition to abi_info warrants that.
This leaves the larger issue fundamentally unresolved (see my next answer about that), but allows us to progress here.
Regarding the wider issue of future proofing dependency specifiers, I think it goes beyond this PEP.
I’m not sure I like the Dependency field proposed by @konsti, as that would put us back in the same situation for possible future additions.
Usually when dealing with backwards compatibility, we rely on versions, so maybe we should version the dependency specification itself, or possibly only the environment marker part of it.
Indeed, looking at the overall Package Distribution Metadata, we might use the Metadata-Version field of the core metadata for this purpose.
I could not find a clear rule on what changes to the metadata should trigger an updated Metadata-Version, but to me it does not seem outlandish to increase the minor version number for a change in the dependency specification.
Alternatively, we could add a new Dependency-Spec-Version field with similar semantics to Metadata-Version, that might also be exposed as a dependency_spec_version environment marker variable.
This would allow build backends to support communicating required marker support and could even allow package authors to be explicit about their needs.
It would be desirable to clarify the concepts of backward and forward compatibility and the semantics of Metadata-Version; all of this could be done in a dedicated PEP.