PEP 780: ABI features as environment markers

Packaging Standards
1

Hi!

We are excited to present PEP 780—ABI features as environment markers. We believe there is a need to make ABI features like free-threading available as environment markers for dependency specification and beyond that to improve accessibility of ABI features.

This PEP is strongly influenced by an earlier discussion.

We look forward to everyone’s feedback and interesting discussions!

Abstract

This PEP defines using ABI features as environment markers for project dependencies, through a new sys_abi_features environment marker and sys.abi_features attribute in the sys module.
PEP 508 (later moved to the Python Packaging User Guide) introduced environment markers to specify dependencies based on rules that describe when the dependency should be used.

This PEP extends the environment markers to allow specifying dependencies based on specific ABI features of the Python interpreter. For this, it defines a set of ABI features and specifies how they are made available via an addition to the Python Standard Library in the form of a new attribute sys.abi_features, as well as for environment markers as a new marker variable,
sys_abi_features.

An example of use in dependency specification is

cython >3.1.0a1; "free-threading" in sys_abi_features
cython ==3.0.*; "free-threading" not in sys_abi_features

More examples can be found in the document.

PEP

5 Likes
2

Is there any specification for the sys.abi_features attribute to be a static compile-time constant or dynamically runtime-dependent?

For the former, some feature names can be ambiguous (e.g., gil-enabled). On free-threading build, the GIL can still be controlled by -X gil=0/1 and PYTHON_GIL=0/1.

# static
Py_GIL_DISABLED = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))

# runtime dependent
gil_disabled = not sys._is_gil_enabled()

The latter one needs to implement a sys.__getattr__ function (see gh-127405: Emit a deprecation warning about a future change of `sys.abiflags` availability on Windows by XuehaiPan · Pull Request #131717 · python/cpython · GitHub). I can imagine a potential breakage while users install and run the package with a different PYTHON_GIL environment.

3

To be usable as a marker, the values in sys.abi_features would have to be static, defined by the Python interpreter. Markers are used to decide what code to install in an environment, and if running the interpreter with a different flag, or changing a setting in a running interpreter, makes the code installed invalid, that’s incompatible with the expectations people have of markers.

The point about re-enabling/disabling the GIL at runtime is an important one. What would

python -X gil=0 -m pip install some_pkg

do if run on a free-threaded implementation of Python, if some_pkg depends on thread_utils; "free-threading" in sys_abi_features"? Would it install thread_utils?

python -X gil=1 -c "import some_pkg"

work after such an install command? Or would it fail because thread_utils wasn’t installed?

1 Like
4

It would do exactly the same as without the gil=0 flag:

python -m pip install some_pkg

This is needed for precisely the reason you stated: the installed package needs to be compatible with all the ways that Python might be run in this particular environment. It is the package authors job to make sure that the version of their package that is installed under freethreading would be compatible with gil=0 and/or gil=1 (if they want to support that) or whether any extension modules re-enable the GIL etc.

5

Yes, this indeed. The ABI is static, and won’t be affected by a user passing -X gil or not. Since it’s apparently not 100% clear, we’ll make sure to state this very explicitly in the next update to the PEP.

4 Likes
6

I think this post should have had Packaging and Standards tags, rather than PEPs. Can it still be moved by a moderator?

7

The PEP contains a section “Adding sys.abi_features to the Python Standard Library”, which requires a core PEP.

You may need to split into two PEPs to enable the packaging side of it. Though it’s possible that a PEP could be skipped if it’s literally just going to be to enable x in sys_abi_features on versions of the runtime where the sys attribute exists.

(And just to be 100% clear, I think the post was and the PEP is correctly classified, it just has to talk less about environment markers as specification and move them to examples of how the new sys.abi_features attribute might be used.)

8

This was discussed on https://github.com/python/peps/pull/4315#discussion_r2007997314, with the reviewers agreeing it was too minor. Do you really want to require a full separate PEP just for adding sys.abi_features?

10

I haven’t seen many cases of “arbitrary sets of words we expect everyone to agree on the meaning of” be deemed too minor. That’s not to say that the full list of names for all time needs to be fixed by the PEP, but at least the definition, naming conventions, deprecation process, addition process, and overall rationale ought to be clearly spelled out for future reference (and our format for that is a PEP).

PEP 421 (adding sys.implementation) is a good reference. It might even be the right place for adding abi_features…?

I also suggested on a semi-related PR that a PEP introducing abi_features could also deprecate sys.abiflags, which ought to be adequately covered by sysconfig (for cases where you really just want the exact flags and not the list of features) and has cross-platform complications right now meaning it isn’t a good alternative for the proposed abi_features but could distract from it.

But the main point is that the discussion of this feature is already fragmented, and running it through the normal PEP process will bring it together (or at least formally exclude discussion that doesn’t revolve around the PEP).

Once it exists, I would expect that making it available as a marker is the part that becomes too minor to need a PEP.

11

lol, I can’t help but find this ironic. You think the packaging community is more likely to be able to agree that something is minor and can just be added without a discussion than the core developers?

I’d hope it would be a small and pretty uncontroversial PEP. I’d even be willing to consider a proposal to add it just as a textual change to the relevant standard[1]. But I doubt that would get consensus, and the debate could easily be more effort than just writing a PEP would be.

  1. although we’ve had bad experiences with “obviously uncontroversial” changes made without a PEP in the past ↩︎

2 Likes
12

Of course I don’t think that :smiley: But since there’s already a PEP and a specification with a list of markers, I’d hope that adding a new marker that’s been thoroughly defined by someone else would only require a spec update and not an entirely new motivation.

Though the “Unknown variables must raise an error” part of that spec may cause issues when adding a variable that isn’t available on earlier versions but could be reasonably inferred…