Is there a PEP 508 environment marker for free threading? Unsurprisingly, my packages need newer versions of build dependencies (some prereleases) when building for free threading. I haven’t been able to find an environment marker to be able to specify the different dependencies under the right conditions.
Related, I suppose: I don’t see any mechanism in PEP 508 for defining new environment markers as the need arises (as I believe it has with the arrival of free-threading). Does each environment marker addition warrant a whole PEP process?
I don’t think this is something we anticipated when we specified markers.
Yes, a new marker would need a spec amendment, including tooling changes and a transition period (we don’t have a way to say “you must use tools that understand this revision of the marker spec” either ). That will almost certainly require a PEP.
Such a PEP should probably include an extension mechanism so we don’t ever have to go through this again…
It’s not the biggest deal, and perhaps not enough for someone to go through the process, but thanks for confirming it’s not there. Once the latest stable versions of packaging tools have free-threading support, there likely won’t be a much ongoing practical need for this one. In my case, free threading needs cython>=3.1dev where I already have cython>=3. Once Cython 3.1 is out, nobody will probably notice.
I think checking for Python >= 3.13 is the closest you can get. Hopefully the dependencies will feature an actual release before Python 3.13 final is out, so perhaps that marker would be good enough.
Until it’s the default, free-threading will be marked by an ABI flag (t). So perhaps a way of checking the ABI flags as environment markers is the easiest/most generic option?
Hi all, my team at Quansight working on free-threading would like to take on writing this PEP and the implementation work needed in some of the key libraries and tools - e.g., packaging, pip, pypa/install and the Python Packaging User Guide.
We are seeing an increasing number of cases where an environment marker for free-threading is needed. To give two prominent examples:
Cython has support for free-threading only in its master branch, and is used by a lot of projects that already publish cp313t wheels. Picking up the wrong Cython version is causing a lot of obscure build failures or runtime crashes. It would be quite beneficial if the metadata could express, e.g., >3.1.0a1 # free-threading-only.
CFFI has no support yet, and the maintainers have stated that it may be a good idea to fork cffi, ensure it starts working with free-threading, and only come back to the CFFI project with a single large PR that adds support after the functionality “is reasonably well-tested (either as tests or, better in this case, tested by being in use in various other projects)”. There are a lot of projects that depend on cffi. They are likely fine to start depending on a fork for free-threading only, however depending on a fork for >=3.13 or for all Python versions seems like a much larger ask, and more disruptive for distro packagers for example.
We expect that there will be other packages, especially older projects with maintainers who don’t have much bandwidth, that will be in such a situation and can benefit from friendly forks rolling out for a while.
I completely agree that a PEP should be as forward-looking as possible, and not just focus on free-threading. I think there are two options:
Do a single larger update adding all markers that people want/need. E.g., I’m aware of the need to distinguish 32-bit from 64-bit Python on Windows, that currently isn’t possible.
Do (1), plus add an extension mechanism.
It’s not clear to me whether an extension mechanism is desirable for everyone as a final solution, because it’s inherently dynamic (I think, unless there’s a static flavor a la perennial manylinux) and it may prevent use cases for tools that generate lock files for example. So even if a good extension mechanism is found, there may be a future need for another PEP to add more markers that can be statically recognized.
Questions we have at the moment:
Are there other environment markers that should be considered for inclusion?
Are there other tools in addition to packaging, pip that seem to be on the critical path and we should consider working on?
For the suggested extension mechanism, did anyone already think in more detail about this and have a suggested implementation direction (static vs. semi-static vs. completely dynamic, syntax, …)?
I think it would make sense to essentially expand sys.abiflags into a list with proper feature/variant names, which could live in sys or sysconfig. The marker, then, would be something like 'free-threading' in sysconfig.abi_features.
Technically, we could do 't' in sys.abiflags, but we could really use proper standardized names for ABI flags — no more is it “free-threading” or “nogil”? — so we might as well take the opportunity to add this API.
Backwards compatibility wouldn’t be much of an issue, on older versions, we just translate sys.abiflags to the names chosen in the new API.
I think that’s a bundle of complexity on its own, I wonder if it wouldn’t be better off as a separate PEP.
That said, I really like the idea. We implemented something similar in Conda. Here, for example, the CUDA version is implemented as a plugin. In this case, a first-party one, but 3rd parties could do the same.
Thanks for the input @FFY00! This is an interesting option. I do agree that wanting to spell it as free-threading, rather than t or hardcording an individual wheel abi tag value like cp313t, is nice. Minor detail: marker variables have no dots, so it’d be sysconfig_abi_features.
It looks to me like a similar approach would work for 32-bit vs. 64-bit Python on Windows.
Agreed I think, unless we can come up with a nice and concise mechanism.
There were a few very long threads on support for CUDA and other accelerator hardware in Python packaging already, which illustrates how complex a fully generic and dynamic extension mechanism is indeed.
With help from @rgommers we’ve been working on this over the past weeks and we are now getting ready to submit a PEP PR.
If you are interested, you are very welcome to have a look at our proposal, which contains also links to the reference implementations for the sys.abi_features attribute suggested by @FFY00 (thanks!), the environment markers implementation in packaging, and an example package.
I’ve only skimmed the PR, but I didn’t see any discussion of how this proposal would handle the point where free-threading becomes the default. At that point, presumably we’d lose the concept of “free-threaded Python” as a distinct entity, and it would be back to being just “Python”. It’s probably worth including a discussion of this, even though it’s some way in the future at the moment.
That’s a good point. A possible answer is that nothing needs to change, package authors can simply stop using the free-threading marker.
On the other hand, we don’t yet know what the transition will look like exactly. It’s possible, maybe even likely, that there’s a gradual switchover and that the GIL-enabled build will stay around for some time even when it’s not the default anymore. Which could argue for treating both builds on more equal footing, and also introducing a 'gil-enabled' feature in sys.abi_features - it would then never be empty (the same should perhaps also apply to '32-bit'/`‘64-bit’).
Won’t that mean that if an installer needs to backtrack to an older version of the package, they will get the version that doesn’t have the free-threading marker - and hence requires the GIL (and which therefore may break under free threading)?
@pf_moore I’m not sure what you’re asking exactly, is “that” referring to the first or second paragraph of my last answer? Either way, I think that yes if one backtracks to an old version of some package where metadata that is necessary for a correct install is missing, things won’t work out well. That’s generically true though for any kind of metadata that is being introduced. It seems like we can’t break anything that’s not already not broken today for the same scenario. Wheels still have the t in the ABI tag, so it’s not possible to get the wrong wheel installed.
If there’s a specific worry you have, could you please give a concrete example?
I was referring to your first paragraph. I think I had a different understanding of what will happen when free threading becomes the default than you do.
I think the key thing to emphasise is that even when free threading becomes the default, we’d need to retain "free-threading" in sys.abi_features, even though at that point it’s stating the obvious. Because otherwise the marker will no longer work properly, and it’s impossible in any practical sense to ensure that installers will never encounter old packages which still use "free-threading" in sys_abi_features as a marker. Similarly, we’ll need to retain the t suffix on the ABI tag, even when there’s no longer any non-t version. Because again, as you say, correct installation relies on the ABI tag distinguishing free-threading builds.
So the key thing is that this will put constraints on what “free-threading becomes the default” means. I had assumed that it would mean that the “normal” build would gain free-threading (and therefore in all other ways, it would look like the GIL-enabled build does nowadays). Whereas you (I think) are assuming that we drop the GIL-enabled build, and simply publish the free-threading build as the official Python version.
IMO, both possibilities are reasonable assumptions for people to make, and I think the PEP needs to be explicit in saying that the first approach won’t work.
Looking at the very long term, if we take this approach I can imagine people who aren’t aware of the history not understanding why the “t” on the ABI name, and the fact that we bother to include “free-threading” in sys.abi_features, exist. At that point, we’ll need to face the decision of whether we retain those markers forever, or whether it’s safe to remove them. So to an extent, we’re simply kicking the problem into the future, not actually solving it. (Which, to be clear, is probably a perfectly reasonable thing to do…)
Thanks, that is helpful - you’re making some very good points.
I do think that there was an attempt so far to aim for that, hence there not being any way to identify free-threading with public API. There’s sys._is_gil_enabled() and sysconfig.get_config_var("Py_GIL_DISABLED"), which are both private and hence subject to change. A GitHub code search shows that their use is spreading though, showing that there is a need for distinguishing between the two builds (wholly unsurprising). In practice, if either of these APIs were to be removed suddenly in a few years it is going to lead to fairly widespread breakage, which is undesirable. So effectively it has to be treated as public API anyway, with slow/careful deprecation if there’s a desire to remove it in the future.
Technically this isn’t true - the marker itself is packaging syntax only, it doesn’t rely on the contents of sys.abi_features, or even sys.abi_features existing at all. That’s also why introducing the marker will work fine on older Python versions. It’s just design consistency and the principle of least confusion that makes it a good idea (or at least, we think it is) to actually introduce the new sys module feature.
I do agree that it’d be nicer to keep 'free-threading' in sys.abi_features as long as it’s in use as an environment marker. But if there’s a desire to deprecate it because it’s no longer needed in the (far) future, it’s possible to do so. Packaging tools would then have to deal with that deprecation through a conditional code path for older versions.
Technically also not the case I think. If cp319 is unambiguous, if there’d be only one ABI left for Python 3.19, that would work without the t. Dropping the t in a newer version doesn’t mean breakage for older versions. What I said was the opposite: cp313t will always remain cp313t, so a future change (either a t-drop or a 'free-threading' removal) won’t break that nor will cp313 start being interpreted as free-threading (module bugs in packaging tools of course). Both “keep t” and “drop t” are valid choices in the future here I believe, and this PEP doesn’t change that.
Possibly 't' in sys.abiflags is a concern for dropping the t. Not sure if anyone is using that spelling, but it’s technically public API and since abiflags is a simple string the t can’t easily be deprecated.
Good assessment. The difference is quite small and subtle here. The truth may be somewhere in the middle. I think it has to be looked at per API/feature at the time when free-threading is being made the default. We may want to drop the t from the ABI tag but keep 'free-threading' in sys.abi_features. To me this PEP isn’t really a material change, because effectively sys._is_gil_enabled is already to be treated as public and we’re only introducing nicer API for it.
Agreed. In the future there will be more data to decide on keep vs. deprecate-and-remove, it seems premature and unnecessary to try and decide on that now.
Thank you for justifying why I spent so long arguing towards their current designs We don’t have to remove these in the future, because they reflect what they’re doing.
Equally, other implementations don’t have to provide them at all, because they have a leading underscore. Documented but CPython-specific is a valid reason for this naming. If you see code that’s assuming it’ll be present based on sys.version_info, feel free to let them know that hasattr (or equivalent) is the intended and correct way. (This is also why it’ll stick around, because otherwise we may have to change the meaning of sys.implementation.name == 'cpython' and not hasattr(sys, '_is_gil_enabled') based on sys.version_info.)
This is correct, and also why we no longer have ABI tags for wchar_t width or memory allocator.
Yes, this is true, and has flow-on effects to naming of extension modules. Personally I’m not a fan of looking at this variable directly anyway (it’s platform-specific; absent on Windows), so I’d just discourage this approach.
Thanks for that explanation. That wasn’t clear to me. I see sys._is_gil_enabled is indeed documented in the sys module. Would you accept a docs PR adding a sentence that explains why the _ is present and that this is actually stable API?
). That will almost certainly require a PEP.
We don’t have to remove these in the future, because they reflect what they’re doing.