PEP 602 and maintenance releases

PEPs
#1

PEP 602 states under 1½ year of full support, 3½ more years of security fixes,

For the next forty two months (3½ years) it receives security updates and source-only releases are made on an as-needed basis (no fixed cadence).

That means installers for the latest 3.7.10 release are not available. Please reconsider this policy in order to improve the security profile of python. From the latest survey [0] 72% of users are using 3.7 or 3.8, and about half of users are using Windows or macOS. Under “Python installation and upgrade” section of the survey, something like 34% [1] are getting python from python.org with the note “Windows users tend to install Python from Python.org”, and typically that is the hardest platform to compile for.

My take, and feedback I hear from users, is that a source downloads are not very helpful for the very people most exposed to security flaws: users on Windows with no devops support.

Is there a large maintenaner burden to upload installers for security updates?
Matti

[0] https://www.jetbrains.com/lp/python-developers-survey-2020
[1] not clear what 34% means since the total for that category is > %100

#2

There is a non-trivial maintenance burden with sustaining build environment, building releases, and testing binaries before they are released. Binary builds depend on external components (TCL/TK, OpenSSL, bz2, gzip, …), build environments (specific version of VS or XCode), and OS deployment targets.

Aren’t there any companies that provide security releases of and support for older versions for Windows and macOS, e.g. Anaconda?

1 Like
#3

There is another big issue for releases in the security-fix-only phase of their life cycle that won’t be fixed just by providing binary installers. That is: once past the bugfix (fully supported) phase, releases no longer receive changes to support new operating system or third-party library releases. A good example of that right now is with macOS 11 Big Sur where some significant changes to how the operating system works affect Python (and other software) and, while Apple attempted to mitigate the effects of those changes by providing some assistance with porting, we and they did not try to backport those changes to older releases. So even if we manufactured binary installers for the latest 3.7 and 3.6 security-fix releases, they still would not work entirely correctly on the latest macOS releases.

From my perspective as a release manager, I think we don’t want to encourage most users to install security-fix releases; we want them to use a fully supported release. I think of security-fix releases as aimed at sophisticated users who are used to and willing to maintain their own environments and also aimed at downstream distributors of Python who provide additional support to their users.

Now, of course, this is not an ideal world and not everyone can upgrade to a newer release easily. But since we have limited resources, I think it is important to draw the line somewhere and this has been our policy for a long time, long predating PEP 602. If we were to change the policy, that would not only put more work on the release team with manufacturing releases but, more importantly, on the whole CPython development project as people would be encouraged to use older releases longer and have more expectations for platform support and bug fixes. That would be setting the wrong expectations with users. Better to be upfront about it, I think.

5 Likes
#4

I agree entirely with Ned’s post. And while I don’t think the actual building of Python for Windows is particularly complicated, I recognise that needing to spend money to get the environment set up (or registering for free GitHub Actions CI) is more of a burden for people who have not already done it.

The primary cost is in making sure that the thing we build and release works, and as Christian mentioned, making sure it works with OpenSSL, SQLite3, Tcl/Tk, and other libraries that we do not control.

Personally, I’d be totally okay with fewer major releases and longer support cycles, but when it was discussed with the community the preference was for faster major releases. The tradeoff has to be that those releases do not live for as long (even though we are currently absorbing the extra effort for “free”), and so I don’t see any way we would increase support lifetimes without a serious investment in general maintenance and testing.

4 Likes
#5

Rather than using survey data, look at the actual patch levels people on binary installer dominated OSes (Windows and Mac) are running from non CI system PyPI pip statistics. (Someone else will need to figure out the query to show this)

We’ve never had an auto upgrading binary installation. So I hypothesize that people running older versions from binary our installers largely don’t even have the latest binary installed.

2 Likes
#6

Here’s some of the information I can get for Windows usage. (This is via the PSF account for publishing to the Store, using the PSF certificate to identify the binaries, so it isn’t internal Microsoft data. Unfortunately, because it isn’t, I don’t know exactly how the numbers are obtained/calculated, but I expect there’s some serious extrapolation involved.)

image
image821×702 29.8 KB

Note: This is definitely desktop (regular) installs, and not Store installs. There are separate stats for those, but they also auto-update, so I’d expect everyone to be on the latest patch release for whatever versions they have.

1 Like
#7

You can see similar stats for Chocolatey installs:

https://chocolatey.org/packages/python#versionhistory

image
image1100×584 33.1 KB

I wonder why most people have 3.7.4 installed. My guess is that 3.8.0 got released a day before 3.7.5 and they simply moved on to the next stable release.

#8

I reformatted the 3.7 data.

I assume the numbers in the image are sorted by hits, so the missing data is all below 300,000 hits. That is a bit strange since it means 3.8, released Oct 2019, is much less popular than 3.7. As a data science practitioner, maybe that is due to the lack of availability of binary installations for common data science packages like tensorflow and pytorch for 3.8 and up? But I digress.

I wonder how the data cluster around the release dates. Any peak in the data might give an indication of how many people are conscientiously updating due to a new release vs. how many new users are downloading to get started.

Matti

version hits installer? release date days as “latest release” note
3.7.2 1,361,660 yes Dec 24, 2018 91
3.7.3 1,264,835 yes March 25, 2019 105
3.7.4 12,929,343 yes July 8, 2019 99 outlier
3.7.5 619,479 yes Oct. 15, 2019 64 Python 3.8.0 also released
3.7.6 453,454 yes Dec. 18, 2019 83
3.7.7 4,534,523 yes March 10, 2020 109 outlier
3.7.8 ? yes June 27, 2020 51
3.7.9 1,324,718 yes Aug. 17, 2020 182
3.7.10 ? no Feb. 15, 2021 38 so far
1 Like