Environment marker for free-threading

Packaging
7

Hi all, my team at Quansight working on free-threading would like to take on writing this PEP and the implementation work needed in some of the key libraries and tools - e.g., packaging, pip, pypa/install and the Python Packaging User Guide.

We are seeing an increasing number of cases where an environment marker for free-threading is needed. To give two prominent examples:

  1. Cython has support for free-threading only in its master branch, and is used by a lot of projects that already publish cp313t wheels. Picking up the wrong Cython version is causing a lot of obscure build failures or runtime crashes. It would be quite beneficial if the metadata could express, e.g., >3.1.0a1 # free-threading-only.
  2. CFFI has no support yet, and the maintainers have stated that it may be a good idea to fork cffi, ensure it starts working with free-threading, and only come back to the CFFI project with a single large PR that adds support after the functionality “is reasonably well-tested (either as tests or, better in this case, tested by being in use in various other projects)”. There are a lot of projects that depend on cffi. They are likely fine to start depending on a fork for free-threading only, however depending on a fork for >=3.13 or for all Python versions seems like a much larger ask, and more disruptive for distro packagers for example.

We expect that there will be other packages, especially older projects with maintainers who don’t have much bandwidth, that will be in such a situation and can benefit from friendly forks rolling out for a while.

I completely agree that a PEP should be as forward-looking as possible, and not just focus on free-threading. I think there are two options:

  1. Do a single larger update adding all markers that people want/need. E.g., I’m aware of the need to distinguish 32-bit from 64-bit Python on Windows, that currently isn’t possible.
  2. Do (1), plus add an extension mechanism.

It’s not clear to me whether an extension mechanism is desirable for everyone as a final solution, because it’s inherently dynamic (I think, unless there’s a static flavor a la perennial manylinux) and it may prevent use cases for tools that generate lock files for example. So even if a good extension mechanism is found, there may be a future need for another PEP to add more markers that can be statically recognized.

Questions we have at the moment:

  1. Are there other environment markers that should be considered for inclusion?
  2. Are there other tools in addition to packaging, pip that seem to be on the critical path and we should consider working on?
  3. For the suggested extension mechanism, did anyone already think in more detail about this and have a suggested implementation direction (static vs. semi-static vs. completely dynamic, syntax, …)?
7 Likes