If commercial redistributors that are getting paid for Python 2.7 support don’t step up to pay the pip developers to continue doing that work, then why should the pip developers continue donating that labour to them for free? (and the same question applies to cryptography et al)
I’ve been on the other side of that fence: if the volunteers are still doing the work, the answer to internal funding requests is “Nah, the community is already handling that just fine”, so redistributors only step in when it’s abundantly clear that if they don’t do something, then they’re not going to be able to meet the support commitments they’ve already made to their customers.
(Tangent: being able to state this openly is one of the nicest aspects of not working for a redistributor any more)
We’re all human beings with our own time and interest constraints. I wouldn’t fault any project leader for either continuing Python 2 support, or dropping it. E.g. If you believe that Python 2 support is strategic for your project (helps people transition to Python 3), and you are willing to do the work, mindful of getting burned out or overly stressed, then great! If you want to simplify your life, and have more limited resources, drop it! Let’s be compassionate and grateful for our universe of package maintainers, large and small.
I’ve personally made the decision that I won’t support Python 2 any more, for the majority of my packages. A couple (e.g. importlib_resources and importlib_metadata) will continue to do so for strategic reasons, at least for now. At some point, we’ll drop it, and it’ll be okay!
BTW, I expect that Python 2 packaging will run into another problem in a year or two. At some point, package maintainers will start taking for granted that python 3 is the only python. Right now, Python-Requires: gives a way for packages to opt-out of python 2 support, but we might need to switch to an opt-in model, or else have PyPI become gradually unusable for the py2 laggards.
(I also wonder how Redhat is planning to handle all the python 2 packages their users need. Will they fork requests, numpy, django, …?)
[with my RH hat on]
That’s how we always did it*. Red Hat generally takes software, integrates and tests it, and then only applies security fixes and patches necessary to keep it going. Forking is what we do.
* but I can’t reveal any future plans – that could be construed as a promise to deliver.
Anyway, you can reasonably expect Red Hat to maintain Python 2 stuff for Linux for some time. If anyone wants to {maintain CI and try keeping it green} for other systems, I’d be very happy to maintain shared repos. But if not, I won’t bother – setting up “ghost town” infrastructure is not a good use of my paid time, and I won’t spend my unpaid time on Python 2 support.
Nick, thank you for sharing your thoughts in this conversation.
You’re allowed to say much more than I am, and you still probably know more about Python on enterprise Linux than I do :)
It’s probably better to solve this at the build tool level. Tools like setuptools, flit, etc could start defaulting to having a 3.x only Requires-Python and require someone to explicitly set Requires-Python to regain 2.x support. With an appropriate deprecation period and backwards compatibility management for that particular project.
I think PyPI works best remaining relatively neutral about the files it’s hosting.
This is one of the reasons I suggested using a separate domain name for the Python 2 PyPI servers. Again, I don’t understand the details of the PyPI infrastructure but switching to two separate domains would seem to me to solve some problems. On some flag day, you could make the Python 2 PyPI stuff completely separate. Essentially, fork the PyPI package archive. Then, when people upload Python 2 packages using “setup.py sdist” or whatever, those files get sent to the Python 2 specific archive.
Trying to allow PyPI to support Python 2 packages indefinately seems a mistake to me. Once you fork the archives, if someone steps up to maintain the Python 2 version, it can live for as long as people care about it.
I’m not entirely convinced that PyPI download metrics are the best gauge of language version popularity. Commercial users typically download from internal mirrors, Linux distros download once from PyPI, and I suspect that other redistributors have similar download patterns. Those all skew usage metrics since PyPI knows nothing about redistributor popularity. Maybe it’s the best we have, but I would caution to put too much faith in those numbers.
We’re not really looking at a general “language popularity” here. If a million people are using Python 3 on Debian and installing those modules using apt-get and a 1000 people are using Python 2.7 with pip to talk to PyPI (but zero people are using Python 3 for that). Then the fact that Python 3 is more “popular” in that hypothetical doesn’t matter at all to pip, because those Python 3 users are not pip’s users. Likewise with other redistributors like Conda or the like.
Those numbers do matter for the build backends, like setuptools.
The internal mirror thing does effect it, but I suspect not to the degree that people expect. I have had numerous commercial companies (large and small, including F100s) more or less admit to me they’re pulling straight from PyPI. I doubt that it’s going to be changing the resulting numbers by huge amounts.
Agreed. However, there are significant factors that probably skew PyPI download numbers - pip’s cache, universal wheels, etc. So while I agree that the specific factors @barry mentioned may be less important for determining pip’s usage profile, I still maintain that even though download numbers are the best we have, they should be treated with caution.
We do ultimately have to make a decision, though - even if we had no usable data to base it on, that would remain true.
What actual force any decision we make actually has remains a question, too. We can say we support Python 2, but without volunteer effort to deliver that support, it makes little difference in practice…
FWIW, I started working on recording metrics for requests to /simple/*/ as well as file downloads. That should change a little bit of the “issues” (I hesitate to call them full fledged issues since they’re just constraints not real issues).
Pip’s caching is less of an issue. By that, I mean some versions of pip do cache access to /simple/*/, but not the latest version, and even the ones that did were only for 10 minutes so tox usage is maybe effected, in some cases, but the general case it isn’t, and not at all in the latest pip.
This is true for python2 -m pip install ... && python3 -m pip install ... usage on the latest pip.
pip install --upgrade will be tracked, even if the latest version is installed.
Mirrors, local or otherwise are still completely ignored.
Because we’re tracking hits to /simple/*/ we’re not tracking actual users but API requests. It’s entirely possible that 100% of the usage is one person sitting there spamming pip install --upgrade in a loop.
I believe PyPy is intending to support 2.7 for a very long time? Probably Jython as well? Questions around when and how packaging related tooling could drop 2.7 support should at least look beyond CPython for transition planning purposes when the tooling gets used with those VMs.
I’ve only just found this discussion, but yes, PyPy has no Py2 EOL plans currently (and I can say for us $WORK-wise, we have no real plans to move to PyPy3 until it is a bit more mature, and until we run extensive benchmarking showing performance is close to equal, so we’re certainly at least a year or two away).
Obviously those of us who use it are still using pip today, so continuing py2 support is certainly in my interests, but I obviously understand the desire to drop it from a maintenance perspective…
!