Requires-python and pre-release Python versions?

Packaging
1

Currently both pip and uv (and others?) ignore the pre-release section of requires-python.

E.g.

[project]
name = "foo"
version = "0.1.0"
requires-python = ">=3.13"

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Using any Python 3.13 pre-release:

$ pip install --dry-run .
Processing ...
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Would install foo-0.1.0

Yet this does not conform to the specifier spec:

>>> from packaging import specifiers
>>> specifiers.SpecifierSet(">=3.13").contains("3.13.0rc1", prereleases=True)
False

To be fair, I think most users would find this surprising behavior. And the spec actually says very little on requires-python:

This field specifies the Python version(s) that the distribution is compatible with. Installation tools may look at this when picking which version of a project to install.

The value must be in the format specified in Version specifiers.

Perhaps it would make sense, as it is the de facto standard, to say it is normal to ignore the pre-release part of the version on comparison?

3 Likes
2

@pf_moore while we’re on the topic of pre-releases, I would like to submit clarifying language for the spec, something like:

The value must be in the format specified in Version specifiers.

To something like this (I’m not tied to the specific language, just the overall point):

The value must be in the format specified in Version specifiers, tools MAY choose to ignore the pre-release part of the version so that >=VERSION does not exclude pre-releases from that version.

What is the process to submit clarifying language?

3

The process is described here. Basically, if it’s a purely text-only change that doesn’t have the potential to affect interoperability, it can be submitted as a PR to the standards document. Otherwise, it needs to be discussed here first, and it will either be approved[1] as a text-only change, or referred to the PEP process.

In this case, I’m not convinced this is a text-only change. First of all, I don’t recognise the text you want to change - can you confirm precisely where in the standards you’re proposing to make the change? Second, it definitely doesn’t feel like a text-only change to me - why would it be a simple clarification to start allowing tools to choose to ignore information specified by the user?

I suggest you start by explaining why you want to make the change, what exactly you’re proposing to change, and why it won’t affect interoperability (i.e., it doesn’t change allowed behaviour - in particular, “documenting how tools currently interpret an ambiguous part of the spec” is not a clarification, as it enshrines “tools may choose” as the official behaviour, when the reality might be that the original intent was to specify a behaviour[2]).

  1. by the PyPA core reviewers ↩︎

  2. personally, I think we should specify a behaviour, even if we don’t currently know what the best behaviour to require is… ↩︎

4

First of all, I don’t recognise the text you want to change - can you confirm precisely where in the standards you’re proposing to make the change?

It’s the second line of the “Requires-Python” section of the “Core metadata specifications” specifications article, here it is in more context:

This field specifies the Python version(s) that the distribution is compatible with. Installation tools may look at this when picking which version of a project to install.

The value must be in the format specified in Version specifiers.

For example, if a distribution uses f-strings then it may prevent installation on Python < 3.6 by specifying:

Requires-Python: >=3.6

This field cannot be followed by an environment marker.

Second, it definitely doesn’t feel like a text-only change to me - why would it be a simple clarification to start allowing tools to choose to ignore information specified by the user?

As the line above says “Installation tools may look at this when picking which version of a project to install” it is already allowed for tools to completely ignore this line.

Further, the line only states that the format specified must be a Version specifier, it does not state how the tool must or should interpret that Version specifier.

I suggest you start by explaining why you want to make the change, what exactly you’re proposing to change, and why it won’t affect interoperability

The behavior I’m trying to describe is the existing behavior of pip. Because pip ignores the pre-release parts of the Python version, and adds prereleases=True to the version specifier, other tools already match this behavior to keep compatibility with pip. But I don’t expect anyone reading the spec and trying to implement it themselves would come up with this behavior without knowing what pip does.

As the spec has no explicit behavior requirements here, it only implicitly describes the behavior by giving an example that “may” be followed, any clarifying text on what to do with the version specifier can not break existing parts of the spec. Though I think it would be important not to contradict the example, which this text does not do.

1 Like
5

it is already allowed for tools to completely ignore this line.

If the spec already allow tools to ignore the field completely, I see little value in adding that they can also ignore it partially.

The behavior I’m trying to describe is the existing behavior of pip.

Should “describing what pip does in 2024” be the goal of the specs?

Tools whose intention is to emulate pip should look at pip documentation / behaviour - not at metadata specs.

6

Sorry, I’d missed that it was in the title. My bad.

Yeah, the problem here is that the Python interpreter technically doesn’t follow packaging version semantics, so we can’t make any definitive statements about interpretation :slightly_frowning_face:

It’s all a bit of a muddle, to be honest, but in practical terms it works fine. I’d be against changing the spec, if for no other reason than “if it ain’t broke, don’t fix it”. I certainly don’t have any confidence that a change would be an improvement.

No it should not, and I agree that this is not a good justification for this change.

7

No it should not, and I agree that this is not a good justification for this change.

That’s not the justification, it’s the history and the reality of the situation.

The justification is that now users expect tools to match Python prerelease against non-prerelease specifiers, e.g. >=3.14 is expected to match Python 3.14 alpha 2. But if you were to interpret this as a version specifier then >=3.14 does not match 3.14a2.

I thought it would be better to clarify the spec than have tools realize a naive interpretation of the spec doesn’t match user expectations and then have to investigate how pip has implemented it and copy that (as uv had to do). But if that’s not the consensus that’s fine.

8

I’d actually argue that this is a bug in pip. After all, >=3.14 as a version specifier does not match 3.14a2 whether we assume prereleases=True or not.

However, Python versions do not follow the packaging version spec, so assuming the packaging semantics for specifiers applies isn’t actually justified. But that leaves us with the unhelpful “clarification”

The semantics of version comparison for this metadata field are unspecified.

I’d be in favour of defining proper semantics for the Requires-Python field, but it definitely needs a PEP. And given that the only point of contention is pre-release handling, I think it would be better to hold off on such a PEP until we’ve fixed the situation with prerelease handling in packaging version specifiers. I’d hope that we can simply say “The Requires-Python field follows the normal version specifier semantics”. If we can’t, then something’s gone very wrong[1].

  1. Which is basically that pip has implemented different semantics for Requires-Python and “normal” versions, and people have got used to that inconsistency :slightly_frowning_face: ↩︎

9

I’d actually argue that this is a bug in pip. After all, >=3.14 as a version specifier does not match 3.14a2 whether we assume prereleases=True or not.

I believe it’s this line of code that causes the issue, it doesn’t try and extract pre-release information. uv implemented it this way initially and then changed it to match pip, which I think had the side effect it made it easier for them to reason about multi-Python resolutions, so I doubt they would be easily convinced to go back.

One thing that’s definitely surprising from pip’s implementation, is that requires-python = ">=3.14.0a2" will match Python 3.14 Alpha 1: https://github.com/pypa/pip/issues/12821

I’d be in favour of defining proper semantics for the Requires-Python field, but it definitely needs a PEP.

That would be ideal but I didn’t have much hope :slightly_frowning_face:.

I’d hope that we can simply say “The Requires-Python field follows the normal version specifier semantics”. If we can’t, then something’s gone very wrong

I personally think it might be too embedded into the Python packaging metadata ecosystem at this point for Requires-Python to follow Python packaging version specifier pre-release semantics, at least not without a lot of people accusing Python packaging of needlessly and intentionally breaking things, and also for use cases like uv that resolve across Python versions they might not be able to easily implement it.

10

I agree, although on the face of it, it’s a perfectly reasonable way of getting the Python version to match against. It’s just not precise enough if you want to make fine distinctions over pre-releases (which an installer has to, unfortunately).

Yes. That clearly demonstrates that pip’s approach is wrong. It could be made right by restricting Requires-Python to X.Y[.Z] versions, but that’s quite reasonably going to be viewed as a wart. It is also a backward incompatible change (people could have >=3.14.0a2 in their metadata at the moment). Conversely, constructing a precise Python version and using standard packaging semantics will break people who expect 3.14.0a1 to satisfy Requires-Python: >=3.14.

It seems to me there are two options here:

  1. Run away, in the time-honoured tradition of Brave Sir Robin. Do nothing, the status quo isn’t (as far as I am aware) causing any actual problems, it’s just ill-defined and a bit quirky in some (probably never used) edge cases.
  2. Write a PEP that defines proper comparison semantics for interpreter versions. This would probably need to be a core PEP, not just a packaging PEP, as we’d need the core devs to commit to never changing Python’s versioning scheme to something that broke the comparison semantics[1].

I’m in favour of (1), but if you have the energy and feel it’s something you want to work on, you’re welcome to put together a PEP.

  1. that’s not likely, but we really shouldn’t be imposing constraints on the core interpreter without involving the core devs ↩︎

2 Likes
11

but if you have the energy and feel it’s something you want to work on, you’re welcome to put together a PEP.

I do not feel I have the energy for this task! Aside from the issues discussed here, there’s other thorny issues known to me, such as multi-Python version resolvers ignoring any specified upper-bounds.I’m sure there are plenty of issues unknown to me.

What I do have the energy for is writing some small clarifying language or some documentation somewhere (maybe here? Or maybe it’s time I create a blog for this stuff) that lets users know about the dragons of undefined semantics for Requires-Python.

12

OK, and my point is that I don’t think such a “clarification” (which is actually documenting tool-specific behaviour that is arguably a bug) should go anywhere that could be taken as official.

13

Tool specific documentation appears to be in the discussion section, such as pip vs. easy_install features, and if setuptools is depreciated.

But for clarifying language I was meaning explicitly stating that there are no defined Requires-Python version matching semantics.

That would give the reader a hint not to assume Python packaging version matching semantics even though it has the format of a Python packaging version specifier, without the need to reference any tool’s specific behavior.

14

I remain of the view that doing nothing is fine here. I think “there are no defined semantics” is misleading, even if it’s accurate. There are intended semantics, it’s just that nobody ever sorted out the edge cases.

Can I ask again - is there any actual user-impacting problem caused by the current lack of clarity? If so, that might shed some light on the best course of action. If not, then maybe it’s not worth worrying about this right now.

1 Like
15

I remain of the view that doing nothing is fine here.

Fair enough.

Can I ask again - is there any actual user-impacting problem caused by the current lack of clarity?

Just the ones I already mentioned. Surprise behavior using install tools that implement Requires-Python (https://github.com/pypa/pip/issues/12821). And unclear behavior to implement for tool authors trying to implement the spec (https://github.com/astral-sh/uv/issues/4272 / https://github.com/astral-sh/uv/issues/4714).

16

To me, there are two and a half things users want to declare:

One is the minimum required python version: The required syntax features and std api version.

The other(s) are the python versions and interpreters that you want to lock for, and the you want to run tests and other tasks on. These may or may not be the same, there’s design space here for tools. A backend application may run on one specific cpython version and only need to lock and test for that, a library may want to test a range of cpythons plus a pypy release while locking universally. These are just illustrative examples, this is the part is outside the PEP/requires-python’s scope and imho well-suited for tool specific configuration.

For the minimum required python version, we don’t want an upper bound (see discussion at Consider discarding `< 4` upper-bound on `Requires-Python` · Issue #4022 · astral-sh/uv · GitHub), what we’re really interested in is the major.minor interpreter compatibility. This explains the prerelease behavior: 3.13a1 does have the required “API level” 3.13, it’s only that this field has been defined as version specifier instead of <major>.<minor> so that >=3.13 mismatches.

2 Likes
17

We could fix the upper bound being problematic to set in the first place right? If we say that an upper bound not matching isn’t allowed to backsolve by assuming maximum version support done by a comparison (< or <=) rather than exclusion (!=) should be interpreted as not supporting that version yet, and requiring this be monotonically increasing with time when present?

Put it into the specification that the purpose of an upper bound is only to say that a version is not and has not been supported, so backsolving to a version that did not have an upper bound at all is going backward in time, to even earlier in time, and further from any version that might know enough to support the python version.

1 Like
18

I’d prefer this be fixed than keep telling people they shouldn’t set it.

I rely on certain things that are not guaranteed by documentation with asyncio and concurrent.futures regarding actual vs documented thread safety, and keep an upper bound. I keep an eye on changes that affect the related cpython source files (Automation notifying me) and test in advance, but I do not want to receive errors from users for this if they install a prerelease or decide on the minute it releases, to update. I’d rather fail early and cautiously so that users aren’t exposed to that level of potential breakage, especially for a known reliance on something that may cause hard-to-encounter and harder-to-debug issues.

I know the obvious answer to this is “you shouldn’t be doing that”, but the alternatives here involve either

  • vendoring all of the related machinery piecemeal and taking on the responsibility of keeping that in sync for every supported python version.
  • Not be able to mix async and threading effectively.
  • Work to make these guaranteed (unlikely to be a good idea to promise in advance with the current progress on freethreading, and the open questions of asyncio’s desired use pattern in a post-GIL world)
  • Reimplement all of: asyncio.Event, asyncio.Queue, concurrent.futures.Future (which is documented not to be constructed how it is useful), rather than just rely on them in ways that are either not guaranteed or explicitly warned against, but valid and well-tested for the version I support
  • expose users directly to a high likelihood of breaking in ways that may not be obviously broken, as most of them would then involve a race condition involving both threading and asyncio if one of the non-documented invariants wasn’t upheld and a user got to it before I suitably address such a change.

I know I’m not alone in the ecosystem on this, much larger libraries rely on things that change and are private, cython being a great example here. It would be a good thing if setting the upper version had the expected result, rather than backsolving to a point in time before the library author thought to cap it.

1 Like
19

I know I’m not alone in the ecosystem on this, much larger libraries rely on things that change and are private, cython being a great example here. It would be a good thing if setting the upper version had the expected result, rather than backsolving to a point in time before the library author thought to cap it.

I think you’re talking about putting upper bounds on dependencies, not on the Requires-Python field, if so that’s not the topic at hand. This topic is about the lack of defined semantics for Requires-Python, which does have an issue for upper bounds when universal resolvers are filtering Python versions, but I don’t think it causes excessive backtracking (unlike upper bounds on dependencies).

You may be interested in uv’s thread on this which has proposed ideas for better handling: https://github.com/astral-sh/uv/issues/8157#issuecomment-2495757623

And I also have a PR almost ready to, among other things, slightly improve pip’s upper-bound handling (it was previously posted: https://github.com/pypa/pip/pull/13017, but I am now waiting until resolvelib 1.1 lands as I was asked to consolidate all my optimizations in to one PR that has clear commits).

20

No, I was speaking about upper bounds on Requires-Python, and specifically gave examples of my own reliance on things that aren’t guaranteed in python from version to version to not pass this reliance on to my users prior to me testing and verifying, as well as pointing to one of the most used libraries for accelerating python code that also relies on non-guaranteed python internals.

It causes backsolving to an older version if any older version exists without the upper bound, which is clearly not the intent of the person placing the upper bound.