Hey everyone!
I hope 2019 is coming along well for you.
Python 2.7 will reach EOL in 2020, as per PEP 373 – at least from Python Core Devs directly.
I think now would be a good time to start figuring out how to handle the EOL of 2.7, in the packaging ecosystem tooling. The sooner we document these expectations, the sooner will we be able to inform users to help them prepare better.
Looking at PyPI usage, Python 3 installs still only account for ~30% of installs on PyPI. I expect the trend to increase moving forward in 2019 but as much as I’d like to see it, I’m skeptical there would be a big sway suddenly (like > 80% by 2020).
One of the ideas along the lines of better informing users and pushing them to switch, is to make pip log a warning message informing users about PEP 373 and the January 2020 deadline if they’re on Python 2.7.
Finally, personally, I don’t want to keep support for too long (beyond early 2020) and want to unlock being able to use newer Python 3 syntax / stdlib modules in these codebases sooner than later though I wouldn’t want us to be too aggressive when going about this.
virtualenv will probably keep support for the foreseeable future (another 2 year at least). tox will drop support in June this year as host interpreter (but will support creating test environments as long as virtualenv does).
As for wheel, I am eager to move to Python 3 syntax some time 2020. I’ll try to do this in a manner which does not cause too much trouble to those still left behind with CPython 2.7 or PyPy.
I think there’s 2 aspects here:
Item (1) is the “formal support” question. I’d personally like to stop having to deal with Python 2’s Unicode model, and get access to Python 3 features and libraries, when maintaining pip, so I’m in favour of sooner rather than later (people have had a long time to migrate off Python 2 now, so I don’t have that much sympathy for anyone who expects to still use Python 2 and yet demands to be able to use latest releases of other projects like pip and whatever libraries they are installing).
But regardless of the formal answer to (1), the problem with (2) is one of support resources and interest. If the project maintainers want to get off Python 2, they are going to be a lot less interested in fixing Python 2 problems. That’s more of a people problem - we’re entirely volunteer resourced, and regardless of any support statement, no-one has any right to require that we look at bug reports. So there’s a real possibility of saying we support 2.7, but not actually offering practical support.
I don’t know if that’s something we can formalise (I know paid software does - “we’ll keep the code working, but won’t fix newly reported bugs”) but it may be worth thinking about.
I can imagine a statement along the lines of:
Pip will continue to ensure that it runs on Python 2.7 after the 2.7 EOL date, unless bugs in Python 2.7 itself prevent this (which is unlikely). Issues reported with pip which only occur on Python 2.7 will be considered, but may not get as much attention from the developers. Equally, PRs to fix 2.7 only bugs will be considered, and may be applied (subject to normal review processes) but there may be delays dur to lack of developer resource for reviews.
Bandersnatch is already Python 3 (>= 3.6.1) only. I see no reason to not continue to mirror Python 2 only packages etc. Bandersnatch is very different as it’s hidden away on a server somewhere and not user facing. It will still cater for Python 2 users (of pip etc.) for the foreseeable future.
On my phone so excuse my brevity.
My general opinion is that the packaging toolchain needs to be conservative in dropping support for packaging and installing packages on Python 2 and use a usage based methodology for deciding when to drop support for that.
To unpack that statement first let’s look at the why.
The main underlying reason is that I think it will hamstring our ability to continue to improve Python’s packaging. Obviously someone has to upgrade the versions of the relevant tools in order to get support for new features. However if a project is supporting 2.7 for one reason or another then they are going to be stuck using only the features that existed in the 2019 era toolchain unless we go to great lengths to make sure that all new standards degrade gracefully. This is similar to what happened in the Python itself with the Python 3 introduction where for most users of Python, new improvements basically didn’t exist. I feel like it’s widely regarded that splitting the user base like that was a mistake (at least in terms of strategy) and I feel that we’d essentislly be doing the same thing.
Now, I worded that statement to focus on the ability to continue to support Python 2 projects for installation and packaging but only that. So for instance I think that it is fine to only support running a particular project under Python 3 as long as they can still package/install a Python 2 project.
For projects like bandersnatch (and PyPI) that requirement is easy to satisfy because we’re treating the files as largely opaque blobs. For projects that need to more directly interact with the packages themself (either producing them or consuming them) it is a lot trickier. Generally I think that until usage numbers go down, those projects need to do one of two things. Either they can continue to support Python 2 wholly OR they can add a (or utilize an existing) feature that allows them to target an environment other than the one they’re being run under. So example I think it’s fine for tox to only support running itself under Python 3 as long as a Python 2 environment is still reasonably possible.
There’s also the question of what it means to “support” Python 2. Generally I would say that it means that we continue to keep CI for Python 2 and ensure that the tool generally works on Python 2 (including new features). However I don’t think that it means that we, as maintainers, need to focus on Python 2 only bugs.
I’m struggling to come up with a proper term for it, but generally I’m thinking of a sort of life support where we keep the lights on but not much else. I think as part of that it would be fine to start raising warnings with some links and description of this kind of life support support. I even think it would be fine to say that any bug that gets reported that can’t be reproduced under Python 3 either gets closed or tagged for “community support” basically saying PRs fixing it would be accepted, but the project itself isn’t going to be looking at it.
Largely though, I don’t think we should pick hard dates right now. I think fracturing our user base is going to make things harder, not easier for us as a whole. However I think we can start taking steps to reduce that amount of Python 2 support work that we need to do.
The 70% python2 installs may include a large percentage of people providing backward-compatibility. To take myself as an example: I’ve pretty much moved to python 3 (finally:-) but I do keep all my packages python2 compatible, for now. So all my Travis builds and all that will install all dependencies both for py2 and py3.
Is it possible to see how many of the PyPi downloads are done by humans, and how many by automatic agents like Travis and such? Because just looking at the former numbers would give a much better indicator…
Unfortunately no. The PyPI numbers are (obviously) not an end-be-all of anything, since it’s tracking actual downloads, a number of factors influence them:
pip install --upgrade ... where the latest version is already installed won’t be counted since no file will be downloaded.python2 -m pip install ... && python3 -m pip install ... where the download is the same file (universal wheel, sdist, whatever) it’ll count as a Python 2 download but not a Python 3 download, and if they do the reverse the reverse is true.At the end of the day, we can really only use the PyPI metrics (and really, any metric IMO) as a rough gauge. It’s a particularly easy metric for us to get at currently, but I don’t think it’s the only metric either.
Since we’re talking metrics, I can’t provide specifics, but I recently had it confirmed that the vast majority of Python usage on Windows is on 3.6 and later.
A lot of caveats apply, given people who opt-out of reporting usage and crash data, temporary VMs (in build systems) likely don’t run long enough to report anything at all, and that Python 3.4 and earlier don’t report their version number properly, but I’d be massively surprised if it was even close to 50/50 anymore.
I can also say without sharing exact numbers that we’re past a super-majority of users of the Python extension for VS Code using Python 3 (and that’s conservative).
Is it based on installer stats? Something else?
The installer download stats show it as well, but this is other data. Still validating a few things, but the ratio is pretty clear.
The relevance here is how many people really are still using 2.7, and what do they need in terms of packaging support. I think saying “best effort support for an LTS release” is going to be just fine.
These two points make me think that the PyPI download stats could be very misleading. Consider that CI builds tend to be done from a clean VM, so the pip cache won’t apply, whereas manual installs will likely almost always be on a machine with a filled cache. As a result, the cache is essentially biasing the results in favour of CI builds over manual installs.
Add to this the fact that CI builds (where the various Python versions run in the same CI instance - which may be the minority as a lot of projects seem to use a separate CI run per Python version) likely run in version order, which will mean universal wheels and sdists will be counted against Python 2 more often.
I’m not saying the download stats are useless, but these points make me very much more inclined to discount them when trying to measure actual Python 2 usage. Of course, I’m on record as wanting to desupport Python 2, so there’s bound to be a high level of bias in my interpretation, so take the above analysis with a large pinch of salt.
Possibly it should be in a separate thread, but I think the packaging/install chain should definitely lag behind the main EOL date because we should be responsive to the needs of major libraries that want to drop support for Python 2 without leaving their Python 2 users in a lurch.
We are only now starting to get a decent test of heavy reliance on python_requires, and I frequently run into issues with it when packages I use drop support for Python 3.3 - and I’m not even actively using Python 3.3. I suspect that we will only start getting a real sense of the ways that python_requires fails after some tipping point where major libraries are relying on it as their “2/3” support mechanism. As much as I’m the first person to recommend abandoning Python 2, I think the PyPI probably needs to be working to fight the “rearguard” action.
Of course, this level of support could be in the form of some kind of “LTS” branch that is focused almost entirely on migration path issues.
I agree with Donald that pip and setuptools probably will end up being at the tail end of packages dropping support for py2. What I wonder though is whether there’s anything we can do ahead of that to avoid simply being held hostage by enterprise users.
Do we think we could work out some kind of coordination with the enterprise distros too share the load? Or in my fantasy world, we’d announce that community support for py2 will continue so long as users collectively donate ($200,000 * ({year} - 2020))/year to making it happen (though that runs into the legal delicacies around the PSF’s non-profit status).
I think the PyPA client tools are a case where a “long term maintenance branch” model would be appropriate, and we’d look for ways to kick the task of actually maintaining those branches in the direction of the Python maintenance team at Red Hat (cc @encukou). Backporting new work to old stable versions is a good part of what Red Hat does in general, and with RHEL 7 & 8 both supporting Python 2.7 out to 2024, it’s in Red Hat’s interest to ensure there’s a PyPI client that aligns with that.
So I think PyPA client projects will be able to follow the example of the Scientific Python ecosystem: use Python-Requires to restrict Py2 users to still-compatible maintenance branches, and then open up their main dev branches to Py3-only contributions.
As far as timelines go, I think it would be reassuring to users if that branch point for pip was in 2020 sometime, rather than being in 2019.
That would also allow the final Python 2.7 release to include a fully up to date copy of pip for bootstrapping with ensurepip.
For wheel and setuptools, I don’t think it matters as much exactly when they branch, as long as they start setting Python-Requires appropriately first.
This is a good point. We need @benjamin to comment, but I believe the likely plan was for the final Python 2.7 release to only include critical fixes from the previous one. Updating the bundled packaging tools at this time seems appropriate, and then there’s no need to rev the major version for 2.7 (without an application of cash/effort from an interested organisation).
It would potentially be interesting to make a change in tox so that it runs Py2 CI after other Python versions, and see if that makes an observable difference in the PyPI Python version download metrics for sdists and universal wheels.
There are also a few open discussions against linehaul and pip regarding improving the UserAgent metrics in Big Query to better distinguish different download scenarios:
There may also be value in teaching pip to identify well-known CI execution contexts, and report that info as an extra string field. (e.g. set “travis-ci” based on seeing “TRAVIS_BUILD” from Environment Variables - Travis CI environment variables, or “jenkins” based on seeing “JENKINS_URL” from Jenkins : Building a software project)
There will likely be one in Jan 2020. I agree that it would be a good idea to branch at that point, as well as include this release in Py 2.7.
Yeah, this is my main concern with having an LTS release, how long we’d have to support it and who would do the work to support it.
I personally have little interest in maintaining Python 2 compatible code beyond late 2020/early 2021, since I’ve basically not used Python 2 in my personal projects for a fair amount of time now.
Aside; let’s split out the discussion on the metrics to a different thread? I’m pretty sure we collectively have a lot of thoughts there and it’ll be nice to keep a thread focused on what our plans/options are.