Requires-Python upper limits

Packaging Coordination
21

+1. In a perfect world we would not need a python version cap. But here we are. The ability to cap the python version, and declare a project is 99% sure it will not work until further notice, makes the projects developers’ life less stressful. If the requires-python field of the pyproject.toml is not the correct place to cap the python version, where should it be (in a world where setup.py no longer exists)?

A second question is what the dependency resolver should do with such a cap where ever it may be, which is where @henryiii’s proposed alternatives come in.

Thanks!

22

I will note here that in a purely theoretical sense, a “dependency resolver” should do what pip currently does, and backtrack until it finds a package version with a working set of requirements. The problem here is not that the current behaviour is “wrong”, but rather that our current metadata, as implemented across the full history of existing packages, doesn’t actually express the intent people are saying they want.

Whether we can modify the metadata standards to make the metadata mean what people want it to mean, I don’t know. But I doubt it, as it seems to me very unnatural that adding a Python version cap to version 10.0 of a package changes the meaning of the requirements on version 5.0 (which never had a Python version cap) so that we are now no longer allowed to install version 5.0 on new Pythons.

What we can do, and what @henryiii is suggesting here, is make requirements on how tools implement their dependency resolvers. That’s fine, but we should be very careful here, because “dependency resolver” is a standard CS term, and having Python packaging tools use the term with subtly different implications is going to be very confusing, no matter how many practical benefits we get from doing so. We can probably phrase such a constraint as “installers must only accept certain valid solves that a dependency resolver returns”, in effect adding extra constraints on the solution. This avoids misusing the term “dependency resolver”, but does require us to define how these extra constraints get manufactured from the data in the package metadata.

I think that what people are actually assuming here is that if we agree a behaviour, pip can implement it and we’re done. And I want to strongly caution against that, as the whole point of the standards-based approach we use is so that people have the choice to use installers other than pip. So any resolution to this issue must be careful to frame the requirement as “all installers MUST” rather than as “pip must”. @henryiii hints at this with his phrasing “Pip (and suggested for other solvers)”, but I don’t think that goes far enough - people will claim that an installer that doesn’t implement the behaviour that gets decided is “broken”, so we need to define a standard that requires all installers to follow it.

23

In my experience, upper capping python versions are not needed in 99% of the cases. NumPy and SciPy are large and complicated enough projects to fall outside of this case, especially due to their heavy use of C-extensions, sometimes using more than the C-API exposes directly. That’s fine because allows achieving some things with the currently released pythons and in a quick fashion. However, it does come at cost of the maintainers having to do (almost guaranteed) more work for every new Python release.

TLDR; I think upper capping python versions are heavily discouraged and should be avoided, but allowed for use cases where the library maintainers know the library is highly unlikely to work with a new Python release. If you’re in doubt don’t upper cap. And we should document it accordingly. It does mean that tools need to support upper capping. Not sure where we should add this documentation though.

24

In my experience, upper capping python versions are not needed in 99% of the cases.

I’d put it around 50-75% for binary packages, and high upper 90s for Python-only packages. And for binary packages that use Cython, the broken packages are often “fixed” by updating Cython and pre-rendering the C/C++ code (because they tried to be “nice” and burned it in instead of depending on Cython :angry:). If you use pybind11, we test with the current alphas/betas/RCs, of course, so almost all pybind11 extensions support all new versions of Python as long as they don’t cap pybind11. At least one of our users, matplotlib, is already testing Python 3.11 alphas.

The ability to cap the python version, and declare a project is 99% sure it will not work until further notice, makes the projects developers’ life less stressful.

No it does not. It might seem to, but instead you are adding extra work for every package, extra work for every migration (brew, conda-forge, etc). Testing to see what is failing is harder, because you don’t see the real failure, but instead get a scripted fail. You can’t test prereleases. conda-forge would have to add patches to change Requires-Python so that they could move forward with the migrations for packages that don’t quickly update (and you know they would). I do fear with Option 2, we would “normalize” adding caps so that many maintainers would think exactly this way, that adding a cap frees them from a yearly bother of having to test newer Python versions, especially alphas/betas/RCs. The Python community is expected to keep up with the yearly Python releases - this is not Node. If different extensions could run different interpreters or different “language level modes” in the interpreter, it would be different.

And capping Python version never fixes anything. It’s a scripted failure instead of a possible success. Always. Capping a dependency can “fix” things (sometimes). And there are other reasons to have a scripted failure. Maybe a package doesn’t support Apple Silicon or ppc64le or something else. Or even doesn’t support Windows - none of these things are in the checked metadata, because they are not part of the solve. Upper caps never “change” the solve either (at least they shouldn’t), so why do you want them in the explicit metadata? Trove classifiers are there for metadata that does not affect the solve. If you are sure you fail, you can add your own scripted failure - going to my favorite cap on a package - Numba, since they are truly not able to support future versions - numba/setup.py at 0994f97c33a19d7684471cc98b8c36573762204b · numba/numba · GitHub is already there. It just never triggers because the solve gets an old version due to the upper cap behavior currently.

it seems to me very unnatural that adding a Python version cap to version 10.0 of a package changes the meaning of the requirements on version

That’s not quite what Option 2 or 3 are doing. It says if you try to solve, and get version 10, and version 10 doesn’t support your current version of Python or any newer Python, then quit, the solve cannot be completed. If you asked for version <6, then it would never see version 10 and would not quit the solve, because it doesn’t see a cap on Python. It’s just baking in the practical assumption that you will not “drop” support for a newer Python version by never looking for an older version when it sees an upper cap.

I’d be happy to have the language stronger, all of these options would also need to be implemented by Poetry, PDM, etc. They are currently very broken with upper caps, since they will back solve even on older versions of Python if the master cap is higher or uncapped.

Option 1 could be mixed with a bit of Option 3 - if we disallowed Requires-Python for upper caps, we could also ignore them in solvers.

1 Like
25

Some numbers:

Right now, 2 127 of the top 5 000 PyPI packages specify requires_python and 251 have an upper limit.

Upper limit Count
<4 99
<4.0 82
<4.0.0 24
<3.11 13
<3.9 11
<3.10 11
<3.8 3
<=3.10.0 1
<=4.0 1
<3 1
<3.0 1
<3.4.* 1
<3.5 1
<3.7 1
<3.9.* 1
4 Likes
26

Is there a way to see what build system they use? I’d guess quite a few Poetry ones for the <4. Actually, how did you check that?

What the heck is <=4.0 supposed to be, I wonder? “My code will support Python 4! But only the first one. 4.1 will clearly crash my code!” Or what about <=3.10.0! :crazy_face:

That number is growing, though, I’d expect, since SciPy just added an upper cap, Numba’s had one for the last three releases, and Poetry is gaining in popularity, and it upper caps by default and forces downstream packages to upper cap if a dependency does (infectious capping).

1 Like
27

Also, <4’s are all clearly guessing. So over 200 of the 251 packages are just guessing they won’t be compatible with Python 4, even thought the Python devs have said 4 won’t be a breaking change and that 4 might never come…

28

First of all, you can blame me for all the problem with Requires-Python as I am one of the implementer that pushed it in Pip even if was in the pep long before, and yes it was for dropping Python 2.7 for IPython.

I echo many sentiment here that 1) I hate that some project have to put an upper bound, but 2) they do it because removing the upper bound is worse.

I don’t think preventing maintainers from adding an upper bound is the right choice as there are other ways projects could anyway enforce upper bounds in setup.py, to get at least failures at install time other than run time.

I will agree with C.A.M, that there really is a need to update metadata after a package has been published. Requires python is already stored separately in warehouse database in PyPI as it needs to be put in the /simple page on PyPI, so that pip can check it before downloading the files. And no I don’t want to have to rebuild a .postX for every past releases, and publish them all.
Allowing modification a posteriori will make the index inconsistent with the metadata in the sdist/wheels, but is it really the end of the world ? Isn’t yanking a package already sort-of storing metadata outside of the release files ?

And I don’t think improving the resolver or mandating a resolving algo make sens, if you don’t have the right data you won’t come up with the right answer

There is also some comments here saying that most packages get new version released relatively fast after the release of a new Python version, but that might be because of month of planning and preparation because we know a new version of Python will break the world for a wide majority of our users. IPython for example may look like it current works for most use case on Python 3.10, but we had to find funding and hire a contractor to work on most of the 3.10 compatibility and I don’t expect a properly 3.10-compatible release to appear before 2022. And the new 1year release cadence has not made things easier on many downstream projects either.

So please consider an option 4: Allow projects to update Python-Requires even if only in the /simple page after release.

2 Likes
29

Note that Requires-Python is also held in the wheel metadata, and pip will happily use a cached wheel rather than re-downloading it, so changing the metadata in simple, or even allowing metadata embedded in the wheel to be changed, won’t actually achieve what you’re hoping for (it might come close, but there will be odd failures and confused users).

I can actually understand the appeal of allowing after-the-fact changes to metadata - it better reflects the reality here. But it’s a significant change to a principle that tools have relied on for a long time now, and if anyone wants to do it, it would need a PEP, as well as a very well thought through plan on how to handle the transition, and how to deal with the (significant!) backward compatibility break.

Personally, I don’t think the number or seriousness of the legitimate use cases for this would come even close to justifying the disruption that mutable metadata would cause, but feel free to explore the idea if you disagree.

30

I think the context for that was Python v4.0 was to be the feature version after v3.9 (in place of v3.10).

Regardless, there is the assumption in the community that Python follows semantic versioning (even though it doesn’t). Even I specify requires-Python as ~=3.6 as it feels safer

31

Can you check for “any character is <“, rather than just the starting character? Also, ~= is a relevant operator as well.

32

This is exactly my point! There are other ways to get failures if you don’t support a new Python version, just like there are other ways to fail if you don’t support Windows, if you don’t support some special arch, etc. It’s not part of the solver. Solver’s look for working versions. There is no working version if the most recent version that otherwise fits the solve caps below your current version of Python.

The only reason we are arguing about this is because Requires-Python exists, and it seems like such a nice place to put a full range instead of a lower-bounded range. But it was added for the lower bound range - it was not requested for the ability to upper cap. Option 1 just relates failing because Python is too new to something a package has to implement if it wants it, just like all those others.

No. This is not true. Adding an upper bound currently completely messes up the solve. SciPy is finding ways around it because they didn’t support Python 3.10 (and now are guessing about 3.11) and they think Requires-Python is the correct place to put it, and they expect Pip to “fix” the issue. It makes a mess with Numba too. If pip ignored the upper bound completely, the current release of Numba would improve, you’d get 0.54 instead of 0.51, and you’d get the explicit error about Python 3.10 not being supported from Numba’s setup.py.

1/5 of packages are adding it because they think it’s the right place to specify they have a limit. At least 4/5’s of packages are putting <4 here.

There are other reasons for this (adding dependency caps is actually useful), and other issues with this - see my much longer winded post in the original thread. But this really is a huge overkill for Python capping only. As I’ve stated, you can’t really fix solves knowing the Python limits, it is only for error messages. You’d get 0 solve changes by going back over and editing all previous versions to add Python caps. If you see a cap, there’s no point looking for older versions without the cap or with a looser cap already. People don’t drop newer versions of Python.

33

There’s also just picking exact versions, like ==3.6.*, ==3.7.*, etc.

34

Is there a way to see what build system they use? I’d guess quite a few Poetry ones for the <4.

I don’t think that’s in the data: https://pypi.org/pypi/sampleproject/json

Actually, how did you check that?

Like this:

git clone https://github.com/hugovk/pypi-tools
cd pypi-tools
python3 -m pip install -r requirements.txt # Don't actually need all of these

wget https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.json -O \
    data/top-pypi-packages.json

# Show all requires_python in a table
python3 pypi_fields.py --field requires_python -n 5000

# Show all requires_python with an upper bound
python3 pypi_fields.py --field requires_python -n 5000 --format list | grep "<"
# (Then a bit of wrangling to remove the lower limit and make the table here)

It’ll download and cache the JSON for each (e.g. https://pypi.org/pypi/sampleproject/json) but I’ve just uploaded a zip to https://file.io/xKhCUCq8f3Tu which you can put in your user cache dir (e.g. on Mac: ~/Library/Caches/source-finder/).

35

Yep, that was already “any character is <”, but I edited the lower bounds out for clarity. Here they are unedited:

python3 pypi_fields.py --field requires_python -n 5000 | grep "<"
...
| >=3.6,<4.0                                                                          |    49 |
| >=3.6,<4                                                                            |    18 |
| >=3.6, <4                                                                           |    16 |
| >=3.7,<4.0                                                                          |    14 |
| >=3.6.2,<4.0.0                                                                      |    13 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4                                       |     9 |
| >=3.5, <4                                                                           |     8 |
| >=3.6.1,<4.0.0                                                                      |     8 |
| >=3.6.1,<4.0                                                                        |     6 |
| >=3.7,<4                                                                            |     6 |
| >=2.6, !=3.0.*, !=3.1.*, !=3.2.*, <4                                                |     4 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4                              |     4 |
| >=3.7, <3.11                                                                        |     4 |
| >=3.7, <4                                                                           |     4 |
| >=3.8,<4.0                                                                          |     4 |
| <4,>=3.6                                                                            |     3 |
| >=2.7, <4                                                                           |     3 |
| >=3.6,<3.9                                                                          |     3 |
| >=3.6.2,<4                                                                          |     3 |
| >=3.6.2,<4.0                                                                        |     3 |
| >=3.7,<3.10                                                                         |     3 |
| >=3.7,<3.9                                                                          |     3 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <3.10                           |     2 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, <4                                                |     2 |
| >=3.5,<4.0                                                                          |     2 |
| >=3.6, <3.10                                                                        |     2 |
| >=3.6, <3.11                                                                        |     2 |
| >=3.6, <3.9                                                                         |     2 |
| >=3.6,<3.8                                                                          |     2 |
| >=3.7,<3.11                                                                         |     2 |
| !=2.*, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4                              |     1 |
| !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<4,>=2.6                                    |     1 |
| <3.10,>=3.6                                                                         |     1 |
| <3.11,>=3.6                                                                         |     1 |
| <3.9,>=3.6                                                                          |     1 |
| >2.6, !=3.3.*, <4                                                                   |     1 |
| >2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4                                        |     1 |
| >3.4.*, <4                                                                          |     1 |
| >3.5.*, <3.9.*                                                                      |     1 |
| >3.5.*, <4                                                                          |     1 |
| >3.6.*, <4                                                                          |     1 |
| >=2.4, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, <4                     |     1 |
| >=2.6, <3                                                                           |     1 |
| >=2.7, !=3.0, !=3.1, !=3.2, !=3.3, !=3.4, <4                                        |     1 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3*, <4                                        |     1 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <3.11                           |     1 |
| >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.5                                     |     1 |
| >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,<3.10                                         |     1 |
| >=2.7,!=3.0.*,!=3.1.*,<3.4.*                                                        |     1 |
| >=2.7,<3.0                                                                          |     1 |
| >=2.7,<3.11,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*                                 |     1 |
| >=2.7.9, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.0, !=3.5.1, !=3.5.2, <4 |     1 |
| >=2.7.9, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4                                     |     1 |
| >=3, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4                                         |     1 |
| >=3.3,<4.0                                                                          |     1 |
| >=3.4, <4                                                                           |     1 |
| >=3.4,<3.9                                                                          |     1 |
| >=3.4,<4                                                                            |     1 |
| >=3.4.*, <4                                                                         |     1 |
| >=3.5, <3.8                                                                         |     1 |
| >=3.5,<3.10                                                                         |     1 |
| >=3.5.*, <4                                                                         |     1 |
| >=3.5.2,<4.0.0                                                                      |     1 |
| >=3.6, <3.7                                                                         |     1 |
| >=3.6, <4.0                                                                         |     1 |
| >=3.6, <=4.0, !=4.0                                                                 |     1 |
| >=3.6,< 4.0                                                                         |     1 |
| >=3.6,<3.10                                                                         |     1 |
| >=3.6.0,<4                                                                          |     1 |
| >=3.6.0,<4.0.0                                                                      |     1 |
| >=3.6.2, <4                                                                         |     1 |
| >=3.6.2,<3.11                                                                       |     1 |
| >=3.6.2,<=3.10.0                                                                    |     1 |
| >=3.6.5,<4.0.0                                                                      |     1 |
| >=3.7, <3.9                                                                         |     1 |
| >=3.7.1,<4.0                                                                        |     1 |
| >=3.8,<3.11                                                                         |     1 |
1 Like
36

Here are the wiggle additions:

Upper limit Count
~=3.6 59
~=3.5 5
~=3.4 3
~=3.7 3
~=3.8 1
~=3.3 1

So this adds a few, but not too many, and again all of these are randomly guessing that they will support every version of Python 3 but not 4. I’d say this means at least 4/5 of all users are misusing upper caps to add guesses. If they actually break on Python 3.19 or 3.12 or any other future version, the guesses are wrong and completely unhelpful.

Code 
import json
from setuptools._vendor.packaging.specifiers import SpecifierSet
from setuptools._vendor.packaging.version import Version, InvalidVersion
from pathlib import Path
from collections import Counter

def is_valid_version(v: str) -> bool:
    try:
        ver = Version(v)
        return not ver.is_devrelease
    except InvalidVersion:
        return False

def requires_python(r: dict) -> str:
    versions = [v for v in r["releases"] if is_valid_version(v) and r["releases"][v]]
    version = sorted(versions, key = Version)[-1]
    release = r["releases"][version][0]
    return release["requires_python"]

def get_requires_python(fname: str | Path) -> str:
    with open(fname) as f:
        r = json.load(f)
    return requires_python(r)

rp_strings = [get_requires_python(f) for f in Path(".").glob("*.json")]
rp_strings = list(filter(None, rp_strings))
wiggles = Counter([r for r in rp_strings if "~" in r])

for expr, w in sorted(wiggles.items(), key=lambda x: -x[1]):
    print(f"| {expr} | {w} |")

Please, never do this, it makes your upper limit “infectious” for locking solvers (Poetry, PDM), and is meaningless if you end up breaking, for example, on Python 3.19 when Optional is proposed to be removed. Feel free to read Pinning the Python version is special :slight_smile:

37

I understand that it does not fix all use case, but it’s at least an improvement for most end users, and the case where this behaves wrong will I believe mostly impact advanced users that install/upgrade often.

I avoided going down this rabbit-hole as the threads seem to be only about Python-Requires, but for fully-backward compatibility my idea would to allow publishing of <packagename>-<version>-.metadata.patch files, which basically contain the new metadata for the given files.

I disagree with you interpretation, it’s not because one swan is black that all swans are black. It may work if you ignore the upper bound for numba, and the problem is not that you don’t hit setup.py to get the error, the problem is that you can’t update metadata of past releases. The solver does not have enough data.

Would 0.51 and all previous versions had a <3.10, the resolver would clearly state it can’t find matching versions.

Yes I agree.

Incorrect, IPython is a counter example, you miss the case where a software can have an LTS version.

IPython 5.x (LTS) was/is maintained (last commit Aug 28, 2020) past the 6.0 end of life (1. Nov 3, 2018), and if I get a PR agains 5.x I would merge it and not 6.0.
So 5.x is likely compatible with newer version of Python than 6.x. There is no reason if you see maximum pinning on IPython 6.x to propagate it to 5.x.

And take every other version of ubuntu as a on python example, where older versions will get fixes for longer than their younger siblings. Time and Software version ordering are not the same.

If you switch version number for time, I’m tempted to agree with you, though I would not bet on it. But the resolver does not resolve version number based on time.

2 Likes
38

This is still requiring all previous releases be updated, and again, is fixing nothing, just producing a nicer error. Lower bounds on Requires-Python does fix things, it allows a working package to be used instead of a non-working one. This is why it was added. If you don’t correctly fix all older versions, locking solvers like Poetry/PDM will still missolve this. Do you trust all package authors to do this?

If there was something to be gained over a better error message - one that you can still get using other techniques! - then this might be something to push for. But for this, there is not.

I described that “patch” idea in detail in the previous SciPy thread at BUG: PyPI requires on 1.7.1 should allow Python 3.10+ · Issue #14738 · scipy/scipy · GitHub, I’m happy to have someone run with that. But I don’t think it has much impact on Requires-Python - it’s much more important for adding dependency upper bounds. I’m not interested enough in that at the moment to take the lead on that.

If you have a LTS version and are also keeping it up to date with Python and are not keeping your next version up to date with Python and have a pin such that you can get either the next version or the LTS version but not the current version, then yes. But I think we are in a very, very small subset of phase space here - and not a social pattern common in Python packaging. So if I request IPython<7, and IPython 6 doesn’t support Python 3.10, but IPython 5 has been updated to support Python 3.10 (even though it exists in order to support Python 2), then yes. But if I don’t pin <7, the current version of IPython also supports 3.10, so that’s fine. If I pin the LTS IPython, =5.*, that’s also fine - the limit on 6 doesn’t affect me.

The current behavior is affecting multiple packages today. Numba and SciPy are getting incorrect solves due to this and SciPy is trying to apply a workaround (that doesn’t support locking solvers). I have the example of pycln; not supporting Python 3.10 always brought down the CI systems for pybind11, cibuildwheel, and multiple other packages that I worked on. And it worked perfectly on 3.10, it was just a false failure. In fact, since some systems left the same package installs, it actually was working on 3.10, it just couldn’t initially install on 3.10! And so forth.

Once discussion is slowing down, should I start a poll? I’d put 5ish options:

0: Leave it as is.
1: Disallow upper bounds by producing warnings in development backends. Basically pushing for “good” behavior. Failing via other methods can still produce a good error message.
1b: Disallow upper bounds, also ignore them in solvers.
2: Implement upper bounds as an instant fail. Closest to what people think they want, but encourages bad behavior and interferes with Python upgrade migrations.
3: Ignore upper bounds in solvers. Leave the current wording of “guaranteed”, encouraging upper bounds.

I think “editable metadata” doesn’t actually “fix” the current behavior - unless every single package tests everything and updates correctly, then we’d still need one of these options - I don’t think option 0 is right even for this. Locking packages solvers simply don’t support upper caps correctly unless every one is correct, and Pip does not either. And Poetry/PDM force a library’s Requires-Python to be based on the lock file, but they might be able to fix that.

I can’t really tell what people are favoring from the discussion above, other than an initial vote for option 1. I’ll also see if I can ping a few people working on locking solvers.

39

@frostming Ahh, you are here. Was looking for you on the PyPA discord server. You might be interested in this discussion in terms of PDM. @sdispater and @finswimmer for Poetry.

40

Ahh but I think that’s not what you described, or you need to be much more careful about how you define the Cap.

For me the intrinsic cap that the resolver see on a package is independent of what version the user asks. If you start adding the requirements that the intrinsic cap depend on user-version-requirements, the user-requirement filtering and python-requirement filtering are not commutative, which I think is a code smell, and pretty much prevent any caching of which package version is compatible with which python.

I think I would avoid this behavior (but that’s maybe going too far in implementation ? )

Yes, if a package is already maintained and kept up to date, there is no reason a simple enough process would not be followed.

I see requires-python and other dependencies as the same problem. I don’t know why Requires-Python is any different that depending on scipy or something else beyond having access to it before downloading the sdist/tar, and the fact that pip can’t change current Python. But that looks like an implementation details as conda is happy to downgrade your Python, so really Requires-Python is just another dependency field.

You also get the same issues with pinning max version of dependencies as pinning max Python version:

  • It’s the same problem that I currently have with IPython/Ipykernel where ipykernel maintainers did pin IPython to <8.0 while they don’t need to, and sometime pip refuses to update IPython if users don’t update to a dummy ipykernel that just unpin IPython.
  • Or when jedi did a new release that broke API compatibility, and I was asked to pin jedi in a new release that did not help because pip is happy with a newer jedi and older IPython that don’t work together.

You also don’t need every packages to fix everything, you only need problematic packages to be able to “update metadata” when an issue is discovered.

I would also love to see how other languages solves that, because we can’t be the only language that has this issue ?

([rust only have min-rust version](The Manifest Format - The Cargo Book, I wasn’t able to find quick answer for Julia or R),

And how this affects linux distributions maintainers/conda (even if the relationship with linux distribution maintainer can be tense). I feel like the implicit “ignore max pinning” is less useful that “pypi provides correct(ed) Python version requirements”.

(Note that Python-version is just another dependency for Apt/emerge/packman… which may be indicative of something here).