Requires-Python upper limits

seberg · November 7, 2023, 9:37am

IMO, please do merge it, but I will say again that I think it is very much short of this discussion (ignoring the thoughts on mid-/longterm improvements). It would be nice if there was even a semi-authoritative place to discourage it even when it is 100% correct conceptually such as in the case for NumPy. The documentation could be that.

rgommers · November 8, 2023, 2:26pm

+1 to what @seberg suggests.

I’d also add that this there should be a way to have the upper bounds in metadata at some point - either in this field, or in a new to-be-created field. This metadata is definitely relevant, just not for the current state of pip, Poetry & co that are unable to solve for Python version. When working towards metadata that is actually representative of the package itself, rather than only how it’s built for distribution on PyPI, this is a current gap / pain point. Having downstream distro packagers derive the upper bound empirically from the cp3xx wheels that are present on PyPI for a given release rather than looking at a metadata file is not great. And with PyBIs or another such solution for packaged Python releases, it may be relevant for PyPI-focused tools in the future as well.

johnthagen · February 2, 2025, 7:10pm

@henryiii Thanks for bringing up this topic. I was curious if this thread ever landed on a solution/decision.

We are in a situation now where tools like uv and Poetry have diverged.

uvdiscards requires-python upper bounds

But Poetry respects it (though as of 2.0, the published requires-python can be separated from the resolver requires-python)

The pyproject.toml file | Documentation | Poetry - Python dependency management and packaging made easy
Remove upper bound checking on Python version · Issue #10136 · python-poetry/poetry · GitHub

I took me quite a while to dig through threads and issues to get to the bottom of this, and I worry the backtracking issue can still be a tripping point for new developers.

I can appreciate the Poetry team’s reluctance to change behavior before the standard is changed. Perhaps it’s time to consider changing the standard?

notatallshaw · February 2, 2025, 7:46pm

I plan to write a pre-PEP in the next couple of months that formalizes the semantics of Requires-Python, based on the discussion in https://discuss.python.org/t/requires-python-and-pre-release-python-versions/62959

It will specify the semantics as only being the language version part of the version string, i.e. 3.10, 3.11, so not something like 3.11.1, and that the only valid operators as “>”, “>=”, and “!=”, and that tools should warn users when they find requires Python strings that don’t match this, and may throw an error (I’m thinking specifically of tools that build source trees).

We’ll see if it achieves consensus and is accepted.

flying-sheep · September 18, 2025, 9:30am

uv has quite some real resolver issues because of its decision to ignore the upper bound.

E.g. numba has absolutely meaningful upper bounds, as it mucks around with Python bytecode that changes every minor release. How do we make sure that the perfectly valid and meaningful metadata of old e.g. numba releases is respected? I don’t like the idea that legacy packages with bad behavior get accomodated at the cost of breaking legacy packages with good behavior.

I also strongly believe that if this must be changed, a PEP has to have a new way to specify an upper bound. Just removing this feature without replacement completely messes up resolution for packages like numba.

bwoodsend · September 18, 2025, 10:59am

There’s one scenario I don’t see mentioned which in my experience is actually the most common: Someone pins a library to a version they know works, they then upgrade or switch Python version but don’t upgrade the pinned library version, then the upper bound catches it without any backtracking and tells the user exactly what they did wrong.

PyInstaller’s in a similar boat to numba. It reads bytecode, it’s extremely sensitive to all the weird stuff that happens behind the scenes in importlib and Python’s initialisation. It takes about 30 fairly involved commits to get us from one Python version to the next and we’ve been broken by changes in Python’s supposedly stable beta releases phase.

In 2021, I added a Python version upper bound to PyInstaller. I don’t think I’ve ever reduced the bogus bug reports count so much with a one line change as with that one.

pf_moore · September 18, 2025, 12:02pm

I’m in favour of improving the situation here, but I don’t think we should be penalising people who use existing features correctly in order to cater for people who use them incorrectly. Honestly, I don’t think there’s been enough effort made to push back on people using meaningless upper bounds like < 4 (although I accept that because old metadata can’t be fixed, it’s too late in some senses to do anything about this).

I don’t think that simply withdrawing the ability to specify upper bounds is reasonable. There are valid use cases, even if they are rare. Nor do I think that redefining the semantics of an existing field is acceptable - even with a metadata version increment, Requires-Python is still exposed in places like the index API, where there’s no metadata version to disambiguate.

At this point, I don’t think we have a way forward. We’ve clarified the packaging user guide to avoid giving the impression that people should use upper bounds “because they can’t guarantee they will support newer versions of Python”, so hopefully it’s clearer that people should only use upper bounds when there’s a specific need to do so (as in the cases of numba and PyInstaller). But apart from that, the status quo stands. And I don’t think that uv’s approach of simply ignoring metadata that’s problematic is a good solution - it’s their choice how they implement things, but it confuses the situation and (as the posts here demonstrate) it doesn’t actually fix anything, it just changes who gets broken.

IMO, any solution needs to provide a replacement feature that addresses the use cases that Requires-Python was intended to (but doesn’t) handle. The existing Requires-Python metadata should be left alone, and people encouraged to switch to the new approach. And yes, that means there will be some confusion over the status of Requires-Python in script metadata - the PEP for the new feature will need to cover that.

seberg · September 18, 2025, 12:46pm

I think the problem with this (I think the actual text is actually slightly better), is that for many projects it seems better even when we know it won’t be compatible.
In practice nobody can be fully confident that something will be compatible the next Python version, because in practice that isn’t 100% guaranteed (for non C extensions it is probably close to 100% admittedly, for C extensions it is likely they won’t compile).

I am curious what is different about numba/PyInstaller to NumPy, though? NumPy will just fail compiling, honestly, I haven’t seem many complains (except against alpha versions of Python).
So is a runtime (or ideally install/build time) failure via something like:

#if Py_HEXVERSION > ...
#  error "The Python version is too new, maybe there is a newer numba/PyInstaller version?"
#endif

not as good? And if not, what makes PyInstaller/numba different from NumPy, where with NumPy I think the consensus is strongly leaning towards: don’t pin even though we basically know it won’t compile (in practice it just tends to not be the case)?

bwoodsend · September 18, 2025, 1:17pm

I’d be ok with a runtime check for PyInstaller (and it would have to be a runtime check since its wheels are Python-version agnostic) if that’s the accepted workaround.

rgommers · September 18, 2025, 1:59pm

That isn’t correct. The consensus is that we won’t add an upper bound exactly because tools like Poetry and uv have problems with it. And in release notes and/or code comments we’ll say something like “this release supports Python 3.11-3.14”. And then for some other packaging ecosystems like Spack, the maintainers have to deduce from those release notes or from what cp1xx wheels are up on PyPI what the actual upper bounds are.

I like @pf_moore ‘s points, his summary sounds exactly right to me. If there was a safe way to add the upper bounds to metadata, we’d definitely use it in NumPy.

seberg · September 18, 2025, 3:06pm

Right, I lost track of the cause which is uv and Poetry. But is that going to get fixed/changed?

I think my admittedly small grudge is that I don’t think anyone starting a new project right now could guess that the choice NumPy is using right now is good and preferable for the foreseeable future? (And I am very happy to point blame: Because of uv/poetry/… many projects prefer to not even set known correct upper bounds.)

pf_moore · September 18, 2025, 3:20pm

That was my point (I wasn’t clear, sorry). People shouldn’t use upper bounds just in case their code isn’t compatible with some as-yet unreleased Python version. The older text suggested that was a justification for using them, the new text avoids suggesting that.

That’s the sort of use case that any replacement for Requires-Python needs to take into account. There are projects that have a legitimate need for upper bounds^[1] (or at least they have a requirement that currently can’t be handled by any other means than an upper bound) and we have to support those requirements.

If projects have clear examples where they have “known correct upper bounds” then they should present them here, or in a future discussion around a Requires-Python replacement. I think there’s been a lot of examples of people thinking they knew what a correct upper bound was, but being wtong (the Poetry Requires-Python: < 4 example is a case in point - it’s based on a mistaken belief that Python follows semantic versioning). It may be that there’s a way to address those projects’ use cases without an upper bound - and if there isn’t, the use case is valuable to define what capabilities any new mechanism must have.

I’m going to take it as given that NumPy’s need is “legitimate” - I don’t think it’s constructive to turn this discussion into an argument over whether any given use case is valid. Explore how to address the use case, sure, but let’s not dismiss people’s stated needs. ↩︎

h-vetinari · September 18, 2025, 5:17pm

That’s exactly the issue. Any mechanism that can be used correctly (by cases like numpy/numba that genuinely need it), can be mis-used by projects that add spurious bounds due to misunderstandings of various versioning schemes, or because they want to reflect “these are the versions we’ve tested so far”^[1].

Any replacement mechanism will have the exact same problem, because it’s a social issue – in this case, the rampant misuse causing tools like poetry/uv to circumvent it to avoid a bad user-experience on average, which disproportionately punishes those packages (and their users) that would genuinely need it.

What PyPA could do is mandate that those bounds are respected by solvers, and then let projects fix their metadata. But that would be quite disruptive… The only alternative I can see is some kind of _must_be_this_tall_to_use_requires_python option, but that’s a design smell if I ever saw one.

even though there’s no realistic reason to suspect that the next python version would break anything ↩︎

Jelle · September 18, 2025, 6:03pm

Worth noting that numba was called out in the first post of this discussion as an example where an upper bound was causing problems: the trouble was that (at the time) the newest version of numba didn’t support 3.10 yet, so when trying to install numba on 3.10, pip would fall back to a years-old version without an upper bound, which would then fail with a mysterious error.

I don’t know what the solution is, but wanted to mention that even if a package has a “good” reason for using an upper bound, it may still cause problems.

pf_moore · September 18, 2025, 7:03pm

So you’re essentially saying that because a feature can be misused we should remove it? Or that because we haven’t managed to educate people on the correct use of a feature, it’s not worth having? I don’t agree with either of those positions. It’s quite possible that by this point, people have internalised an incorrect understanding of Requires-Python, and therefore we need a new mechanism, to “reset people’s mental model” - but that doesn’t mean we shouldn’t even try to address the undelying issues that people are unsuccessfully trying to solve with Python version limits.

Social issues need social solutions, I agree. We need to educate people. But “fixing” Requires-Python (whether by changing the syntax to disallow upper limits, or by allowing tools to ignore upper limits) is applying a technical solution to a social problem.

But I think there is an underlying technical issue here, which we’re ignoring. And that is that people genuinely do have use cases where they need some way of controlling what version of their package gets installed for a particular Python version. That’s a purely technical question, and all the evidence is that the current technical solution (Requires-Python) has failed to solve it (whether because it fell foul of social problems, or because it’s inadequate technically, isn’t really important at this point).

And I don’t believe that any replacement mechanism must by necessity have the same problem. I’m willing to concede that any replacement built around version caps could have the same problem, but why must it be about version caps? That’s a classic XY Problem - people don’t want version caps, they want to solve their problem. We need to find out what their actual problem is, and stop focusing on version caps.

Agreed, that’s a heavy handed approach. But it’s not that disruptive, insofar as it’s the status quo. The standard defines how Requires-Python works, and tools are supposed to follow the standards. It’s only disruptive in the sense that we’d need to enforce standard compliance on tools like Poetry and uv, which have chosen not to follow the standard. Plus, we’d need to put effort into dealing with the existing bad metadata that’s been introduced into the ecosystem as a result of being insufficiently strict in the past. But that (immutable metadata) is a much bigger issue, and it’s not fair to claim that it’s a reason to abandon any attempt to address the use cases that are currently best served by (correct use of) Requires-Python.

What about the alternative I proposed? That we deprecate Requires-Python and replace it with a better-designed technical solution that addresses the use cases that are currently being handled with Requires-Python? Note that I’m being very careful to say “address the use cases” and not “implement version limits”. I don’t think it’s necessarily the case that version limits are the right solution to the underlying use cases - but we won’t know unless we look at the use cases with fresh eyes.

Again, that’s not directly a problem with version caps, it’s a problem with how tools fall back to older versions when newer versions don’t apply (something which is not standardised, although it’s arguable that there’s “only one reasonable approach”…) combined with the issue that older metadata can’t be edited.

Stopping people using version caps doesn’t solve their problems. It’s just saying "we can’t make version caps work with other (arguably broken) aspects of how the ecosystem works, so tough luck

Understood. Maybe there was (is) a solution using upper bounds. Maybe not. My point is that “upper bounds” are a solution, not a problem. What’s numba’s actual problem here? If it’s a tight dependency on a specific bytecode version, maybe a runtime check of the magic number is enough? Do we even need to automatically pick the right version to install here? And even if we do, maybe letting numba declare “I need bytecode version XXX” is a better solution than the blunt instrument of language version checks.

The wheel variant proposals are suggesting that tools run custom “selector” code as part of the resolution process, to pick the correct wheel. While I have strong reservations about that idea, if it’s good enough for them, why isn’t it good enough here (in some form) as well?

I don’t want to make the perfect the enemy of the good here - blocking any progress simply because there’s a “better” fundamental restructuring that could be done if we ever had the resources isn’t my aim. But I do think that we shouldn’t be so quick to remove a feature that some projects clearly have a need for (in some form), without offering any sort of replacement.

h-vetinari · September 18, 2025, 7:06pm

No magic in the world will make a compatible numba version appear. The only choice is not to play (e.g. not fall back to source installs if there’s no compatible wheel, unless people opt in).

mikeshardmind · September 18, 2025, 7:17pm

We can avoid this case in a way that makes sense, but it still requires a change to the rules of some sort. An upper bound on Requires-Python should be considered as applying to all prior versions of a library, and must be monotonically increasing. This removes the backtracking problem, and frankly, makes sense because it’s a temporal concern. Time moves in one direction, backtracking to an older library version to suddenly support a newer python version makes no sense, and it should be obvious that it’s a case that comes about when a project later realizes they need such a constraint, but can’t change existing metadata.

h-vetinari · September 18, 2025, 7:26pm

Fortunately I wasn’t taking any of those positions either.

If anything, my preference is to stop brushing things under the carpet. Bite the bullet of enforcing the constraints, and encourage projects to fix their metadata. I tend to think about such questions mostly in terms of long-term desiderata, but the transition story is a key part in getting there – and I don’t pretend to have an answer for that right now.

It’s slightly more subtle than that. It’s that people believe that version caps are a solution to their actual problem, and – absent a phenomenal pedagogy effort – will continue to use a version-bound-shaped hammer for problems that look like nails (in their eyes / at first glance / etc.).

Agreed that this is a big issue. But this would be a pretty major change to the current author-centric model, with questions of who can apply metadata fixes to whom and how, and the whole security story of that. At least, I think there are more immediate wins to be had in python packaging than tackling this can of worms (not that I’m trying to stop anyone though!).

pitrou · September 18, 2025, 7:37pm

Perhaps the answer is a separate meta-package a bit like oldest-supported-numpy, which uses environment markers to select different NumPy versions depending on the Python version and architecture:

github.com/scipy/oldest-supported-numpy

setup.cfg

687ffc2f5


      
          install_requires =
              # NOTE: Any platform-related assumptions made here are encoded in the
              #       tests for this package.
          
              # AIX requires fixes contained in numpy 1.16
              numpy==1.16.0; python_version=='3.7' and platform_system=='AIX' and platform_machine!='loongarch64' and platform_python_implementation != 'PyPy'
          
              # IBM i requires fixes contained in numpy 1.23.3. Only Python 3.6 and 3.9
              # are supported on IBM i system, so we're ignoring other Python versions here
              # for simplicity (see gh-66).
              numpy==1.23.3; python_version=='3.9' and platform_system=='OS400' and platform_machine!='loongarch64' and platform_python_implementation!='PyPy'
          
              # numpy 1.19 was the first minor release to provide aarch64 wheels, but
              # wheels require fixes contained in numpy 1.19.3
              numpy==1.19.3; python_version=='3.7' and platform_machine=='aarch64' and platform_system!='AIX' and platform_python_implementation != 'PyPy'
              numpy==1.19.3; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy'
          
              # arm64 on Darwin supports Python 3.8 and above requires numpy>=1.21.0
              # (first version with arm64 wheels available)
              numpy==1.21.0; python_version=='3.7' and platform_machine=='arm64' and platform_system=='Darwin' and platform_python_implementation!='PyPy'

This file has been truncated. show original

pf_moore · September 18, 2025, 7:42pm

There’s a long-running discussion about pip doing that, which is stalled waiting for funding (or some other way of getting resource to progress it) - @rgommers was considering doing something about funding a few years back, but I don’t believe anything came of it.

If it’s something that could ensure correct behaviour in cases where we currently have no solution for people, maybe it’s something we should consider standardising? It’s very close to being an “installer UI” matter, and as such may be inappropriate for standardisation, but I can see a case that we should standardise it to give other standards a stable foundation to build on.