[ACCEPTED] PEP 602: Annual Release Cycle for Python

ambv · September 10, 2019, 9:36am

I’ve been asked by the Steering Council to move parts of PEP 596 that discuss the release cycle changes to its separate PEP. Here it is. I’ve also incorporated feedback from previous discussion.

Abstract

This document describes a change in the release calendar for Python starting with Python 3.9. This change accelerates the release cadence such that major versions are released predictably every twelve months, in October every year.

Implementation

Seventeen months to develop a major version

This PEP proposes that Python 3.X.0 will be developed for around 17 months:

The first five months overlap with Python 3.(X-1).0’s beta and release candidate stages and are thus unversioned.
The next seven months are spent on versioned alpha releases where both new features are incrementally added and bug fixes are included.
The following four months are spent on versioned beta releases where no new features can be added but bug fixes are still included.
The final month is spent on a release candidate (or more, if necessary) and concludes with the release of the final release of Python 3.X.0.

One year of full support, four more years of security fixes

After the release of Python 3.X.0, the 3.X series is maintained for five years:

During the first eighteen months (1½ year) it receives bugfix updates and full releases (sources and installers for Windows and macOS) are made approximately every other month.
For the next forty two months (3½ years) it receives security updates and source-only releases are made on an as-needed basis (no fixed cadence).
The final source-only release is made five years after 3.X.0.

Annual release cadence

Feature development of Python 3.(X+1).0 starts as soon as Python 3.X.0 Beta 1 is released. This creates a twelve month delta between major Python versions.

Example

3.9 development begins: Tuesday, 2019-06-04
3.9.0 alpha 1: Monday, 2019-10-14
3.9.0 alpha 2: Monday, 2019-11-18
3.9.0 alpha 3: Monday, 2019-12-16
3.9.0 alpha 4: Monday, 2020-01-13
3.9.0 alpha 5: Monday, 2020-02-17
3.9.0 alpha 6: Monday, 2020-03-16
3.9.0 alpha 7: Monday, 2020-04-13
3.9.0 beta 1: Monday, 2020-05-18 (No new features beyond this point.)
3.9.0 beta 2: Monday, 2020-06-15
3.9.0 beta 3: Monday, 2020-07-13
3.9.0 beta 4: Monday, 2020-08-17
3.9.0 candidate 1: Monday, 2020-09-14
3.9.0 candidate 2: Monday, 2020-09-21 (if necessary)
3.9.0 final: Monday, 2020-10-05

Figure 1. Consequences of the annual release cycle on the calendar.

In comparison, if this PEP is rejected and Python keeps the current release schedule:

3.9 development begins: Tuesday, 2019-06-04
3.9.0 alpha 1: Monday, 2020-08-03 (10 months later)
3.9.0 alpha 2: Monday, 2020-09-07
3.9.0 alpha 3: Monday, 2020-10-05
3.9.0 alpha 4: Monday, 2020-11-02
3.9.0 beta 1: Monday, 2020-11-30 (6 months later)
3.9.0 beta 2: Monday, 2021-01-04
3.9.0 beta 3: Monday, 2021-02-01
3.9.0 beta 4: Monday, 2021-03-01
3.9.0 candidate 1: Monday, 2021-03-29
3.9.0 candidate 2: Monday, 2021-04-05 (if necessary)
3.9.0 final: Monday, 2021-04-19 (6 months later)

Dependent Policies

Deprecations

The current policy around breaking changes assumes at least two releases before a deprecated feature is removed from Python or a __future__ behavior is enabled by default. This is documented in PEP 387.

This PEP proposes to keep this policy of at least two releases before making a breaking change.

The term of the Steering Council

The current wording of PEP 13 states that “a new council is elected after each feature release”. This PEP proposes to keep this policy as it will lead to a consistent election schedule.

The term of the Release Manager

The current undocumented convention is for a single Release Manager to handle two feature releases of Python. This PEP proposes to keep this policy, allowing for the term to be extended to more releases with approval from the Steering Council and the Cabal of Release Managers.

In particular, since this PEP is authored by the active Release Manager and its effect would shorten the term of the Release Manager, the author is open to managing the release of a third feature release to compensate for the disruption.

Rationale and Goals

This change provides the following advantages:

makes releases smaller: since doubling the cadence doesn’t double our available development resources, consecutive releases are going to be smaller in terms of features;
puts features and bug fixes in hands of users sooner;
creates a more gradual upgrade path for users, by decreasing the surface of change in any single release;
creates a predictable calendar for releases where the final release is always in October (so after the annual core sprint), and the beta phase starts in late May (so after PyCon US sprints), which is especially important for core developers who need to plan to include Python involvement in their calendar;
decreases the urge to rush features shortly before “Beta 1” due to the risk of them “slipping for 18 months”;
increases the explicit alpha release phase, which provides meaningful snapshots of progress on new features;
significantly cuts the implicit “alpha 0” release phase which provides limited use for new development anyway (it overlaps with the beta of the currently developed, still unreleased, version).

Non-goals

Adopting an annual release calendar allows for natural switching to calendar versioning, for example by calling Python 3.9 “Python 3.20” since it’s released in October '20 and so on (“Python 3.23” would be the one released in October '23).

While the ease of switching to calendar versioning can be treated as an advantage of an annual release cycle, this PEP does not advocate for or against a change in how Python is versioned. Should the annual release cycle be adopted, the versioning question will be dealt with in a separate PEP.

Non-risks

This change does not shorten the currently documented support calendar for a Python release, both in terms of bugfix releases and security fixes.

This change does not accelerate the velocity of development. Python is not going to become incompatible faster or accrue new features faster. It’s just that features are going to be released more gradually as they are developed.

Consequently, while this change introduces the ability for users to upgrade much faster, it does not require them to do so. Say, if they upgrade every second release, their experience with Python is going to be similar to the current situation.

Risks

Python redistribution

This requires changes to how integrators, like Linux distributions, release Python within their systems.

The testing matrix

This eventually increases the testing matrix for library and application maintainers that want to support all actively supported Python versions by one or two:

Figure 2. Testing matrix in the 18-month cadence vs. the 12-month

The “extended bugfix support at the discretion of the Release Manager” stage of the current release cycle is not codified. If fact, PEP 101 currently states that after the release of Python 3.(X+1).0 only one last bugfix release is made for Python 3.X.0. However, in practice at least the last four versions of Python 3 overlapped with stable releases of the next version for around six months. Figure 2 is including this information to demonstrate that overlap between stable version releases with the 12-month release cadence will be nothing new.

Other policies may depend on the release cadence

Although identified dependent policies were addressed in a previous section, it is entirely possible there are some other areas which implicitly rely on the timing of Python releases.

Rejected Ideas

Keep the current 18 month release cadence

This is undesirable both for core developers and end users. From the perspective of the core developer:

it makes contribution scheduling harder due to irregular release dates every year;
it creates a surge of rushed commits before (and even after!) Beta 1 due to the stress involved with “missing a release”;
ironically, after Beta 1 it creates a false sense of having “plenty of time” before the next release, time that passes quickly regardless;
it causes certain elements of the workflow to be executed so rarely that they are not explicitly documented, let alone automated.

More importantly, from the perspective of the user:

it creates releases with many new features, some being explicitly incompatible and some being accidentally incompatible, which makes the upgrade cost relatively high every time;
it sits on features and incompatible bug fixes for over a year before becoming available to the user; and more specifically
it causes every “point zero” release to be extra risky for users. While we provide and recommend testing with alphas and betas, “point zero” is the first release of a given Python version for many users. The bigger a release is feature-wise, the more potential problems are hiding in “point zero releases”.

Double the release cadence to achieve 9 months between major versions

This was originally proposed in PEP 596 and rejected as both too irregular and too short. One consequence of a 9 month release cadence was shortening of the beta phase and this was considered dangerous.

Slow down releases but don’t freeze feature development with Beta 1

This is described in PEP 598. This proposal includes non-standard concepts like the “incremental feature release” which makes it hard to understand. The presented advantages are unclear while the unfamiliarity of the scheme poses a real risk of user and integrator confusion.

Long-Term Support Releases

Each version of Python is effectively long-term support: it’s supported for five years, with the first eighteen months allowing regular bug fixes and security updates. For the remaining time security updates are accepted and promptly released.

No extended support in the vein of Python 2.7 is planned going forward.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

pitrou · September 10, 2019, 10:06am

Some comments:

It seems the PEP creates additional workload for our own packagers (Windows and macOS packages): two releases will have to be packaged each month (a bugfix release for 3.X and an alpha / beta / rc release for 3.X+1). Was this discussed with them?
The PEP argues that 18 months is “too long” and 9 months is “too short”, but 12 months is some kind of sweet spot. Was some poll conducted with the user base? With distribution packagers? With library maintainers? Or is it just a personal guesstimate?
The PEP argues that a 18 month release cycle “creates releases with many new features, some being explicitly incompatible and some being accidentally incompatible, which makes the upgrade cost relatively high every time”. How would a 12 month release cycle be fundamentally different? There’ll still be several new features (approximately 33% less of them, but still), some compatible, some explicitly incompatible and some accidentally incompatible…
I’m curious about those “explicitly incompatible or accidentally incompatible” new features. Can you give examples? I don’t think we have many of these, since we strive to be backwards-compatible…
I’m skeptical about the argument “too many features” as it seems that our new features are more and more specific, less and less game-changing.

I have no personal disagreement with the proposal of switching to a 12-month release cycle, but I also don’t find the overall argumentation very convincing.

brettcannon · September 10, 2019, 10:06am

Might be helpful to mention where things correspond in another release, e.g. “3.9 development begins: Tuesday, 2019-06-04 (corresponds to 3.8.0b1)”

This is only true if we change our deprecation policy to be more conservative and to last two feature releases at least instead of one. Otherwise deprecations will publicly last 12 months instead of 18 months. Probably formalizing this in PEP 387 and accepting that PEP would take care of this.

The preferred copyright notice has changed. Please see the PEP 12 section template for the preferred wording.

ambv · September 10, 2019, 11:09am

@steve.dower and @nad can correct me if I’m wrong. My understanding is that none of them are concerned about the additional workload. Steve doesn’t like the idea of 12-month releases for unrelated reasons, he can elaborate on why.

Annual releases that pin to the calendar are easier to reason about and thus are the natural sweet spot. Additionally, several reviewers in the previous discussion, stated that an accelerated cycle would be better (for example @guido saying: “I agree a somewhat faster release cycle would be good for us.”)

You responded to yourself in your parenthesized comment. 1/3 fewer features per release is a lot. Look at “What’s New” of Python 3.6, Python 3.7, and now Python 3.8. Imagine they’re 1/3 shorter.

What you’re asking about is spelled out explicitly in the PEP:

In other words, fewer changes correlates with fewer compatibility pitfalls for upgrading to a newer Python version. @dstufft put it very nicely in the previous thread:

There’s always some. In my time at FB I was responsible for upgrading Python versions across the company and there were plenty incompatibilities big and small. I no longer have access to the internal posts I made about rolling out the new releases and what I found while doing them. Off the top of my head, some of the head scratchers were:

changes in required system libraries to build Python, like unbundling libffi;
async and await as proper keywords;
implementation details breaking test mocks and/or monkey patches (something in logging, cannot find it now);
configparser.read_file() started iterating over f, it used to call f.readline() (yeah, I’m guilty of this one);
ipaddress disallowing __contains__ between Network objects;
quite a few asyncio and typing changes.

There was one that we almost shipped in 3.7 that was making docstrings an attribute on AST objects which unexpectedly broke parsing of single strings using the default mode of ast.parse().

Some were bugs and could be fixed later, like:

concurrent.futures.as_completed() no longer accepting arbitrary iterables;
weakref spewing exceptions during finalization when combined with multiprocessing;
logging.LoggerAdapter objects cannot be nested on Python 3.

True. This is mentioned in the list of policies that will have to be updated. I agree that we cannot allow for the faster release cadence implicitly shortening deprecation periods.

Updated, thanks.

pitrou · September 10, 2019, 11:26am

I agree with this.

Do they represent all demographics? Third-party library maintainers and distribution packagers? These are the categories most likely to voice opposition to an acceleration of the release cadence.

Well, it seems we have different viewpoints on this. If I try to imagine a 1/3 shorter “What’s new” document, I don’t picture something radically different. The probability of breaking changes might be 33% smaller. But that’s not a fundamental change. A fundamental change would be if it were an order of magnitude smaller.

Ok… I just disagree that making the surface of change 33% smaller will really make things fondamentally different in this regard.

ambv · September 10, 2019, 1:09pm

Well, I could be snarky and quote you saying that it’s not a fundamental change. “A fundamental change would be if it were an order of magnitude faster.”

But really, it’s not a fundamental change. We’re not changing how long a single release is supported with security fixes, it’s still five years.

If today, as a library maintainer, you want to cover everything that python-dev still publishes versioned releases for, you have to support three or four releases in your testing matrix. People could cheat there not long ago because adoption of Python 3.3 and 3.4 was very low. But that’s besides the point. You have to support three or four releases.

With the annual releases, you’d have to support five. This is a small change, especially given that consecutive releases themselves are going to be 33% more similar to each other.

See below, an 18-month cadence vs. a 12-month cadence:

(I put this image in the PEP body since I find it useful in demonstrating that the increase is not dramatic.)

pitrou · September 10, 2019, 1:39pm

Really, I’d rather have the input of people concerned, rather than some rationalization about what their concerns may be.

ncoghlan · September 10, 2019, 2:10pm

Thanks for breaking this out from the 3.9 release schedule PEP.

The big pieces of data I would like to see when considering both this PEP and PEP 598 are what we can glean from PyPI metrics regarding adoption rates:

on the publishing side, how long it takes for a given release to catch up with its predecessor in terms of the percentage of projects offering pre-built wheel archives, and in terms of compatibility claims in project metadata
on the consumption side, how long it takes for a new release to overtake its predecessor in terms of number of downloads

After all, having a new binary available from python.org is only step 1 in a new CPython release becoming as usable as its predecessor, and if it’s taking 6-12 months for that to happen, then a 12 monthly cadence could have the unfortunate result of folks choosing to delay adoption until the point where releases are already in security-fix only mode (in which case, they would have been much better served by an incremental feature release model that separates the “stable platform for library authors to target” date from the “locked feature set for application authors to adopt” date.

ambv · September 10, 2019, 2:17pm

You’re not wrong. So I asked them: https://twitter.com/llanga/status/1171426661893435397

I’ll send this form out also via python-users and python-dev.

steve.dower · September 10, 2019, 2:38pm

I’ve automated my publishing workflow to the point where I’ll happily do (>=3.8) releases monthly. Other people may have concerns about the reduced security implied in this (i.e. much easier for someone else to publish a release), but the greatest risk is in my GPG key (which I don’t recommend trusting anyway - the PSF’s signing certificate is handled properly, even in this system).

However, my personal preference is to slow down the release cadence, probably to 2 years, and to have a much longer beta release. This will also have the effect of reducing average churn (per month, for example), and condensing it all at once on a predictable schedule. It will also encourage making early releases of functionality on PyPI where possible, which allows those who can to opt in to using it earlier than the next stable release.

I’d be quite happy to make beta continuous as well - even switch to a CalVer release cycle for “straight from master” with a stabilisation period leading up to every stable “3.x” release (with “3.x.y” releases used for bug fixes as scheduled and approved by the RM).

That way there are essentially two releases to worry about - either the latest beta (because users have more explicitly opted into keeping themselves on the latest) or the current latest stable (because we’re making sure it’s safe to upgrade from 3.x.y to 3.x.(y+1) ). And when we make a new 3.x release the previous one goes into security fix/source-only mode immediately.

But I’m not motivated enough to write a competing PEP and participate in all the arguments, so if someone wants to take this idea and run with it I’m happy to support them. But if I’m on my own here then so be it. I’ll go with whatever is decided by those who care enough to argue about it

ambv · September 10, 2019, 2:41pm

Nitpicking but I think you mean “continuous alpha”? Beta implies feature freeze.

steve.dower · September 10, 2019, 2:43pm

Sure, but if we have continuous alpha, nobody will test it

I’m 100% okay with changing our definition without telling the people who assume it means “broken”.

pf_moore · September 10, 2019, 3:16pm

As a pip developer, I doubt more frequent releases would be a big burden for us. But we’re a pure Python project, so we just deliver universal wheels.

The people who are more likely to be affected are people with C extensions, who will need to ship wheels for each version/platform combination they support. They may not have an issue (depends on how automated their processes are, I suspect) but they are the people whose feedback you need.

The scientific stack (numpy, scipy, pandas) would be a good place to start. Maybe database and GUI interfaces as well. Plus some smaller projects (as they are less likely to have automated processes).

Also, Christoph Gohlke distributes Windows wheels for a lot of projects. He’s only one person, but his input would be useful even if only as feedback on how much effort a new release is to a library maintainer.

encukou · September 11, 2019, 10:20am

Forwarding a reply from Miro Hrončok, the main Fedora packager (who can’t post here yet).

The annual cadence would actually make us update Python every two releases of
Fedora, which is manageable. What I worry about is the release date in October.

Odd Fedora’s final freezes are now scheduled annually to -10-08 and Python
final releases are scheduled annually to -10-05.

That brings us to an unfortunate position. With Python 3.8 and Fedora 31 we’ve
learned this is (close to) impossible. Arguably, if we focus on this more, and
if we continually check that everything builds with Python 3.N+1 as soon as 3.N
has landed, the situation will be easier, but I don’t trust all the package
maintainers in this. Also, it is quite exhausting.

Given the above, we would probably only update Python in Fedora in even Fedora
releases. The release date of even Fedoras is scheduled to -04-28:

https://fedorapeople.org/groups/schedule/f-32/f-32-key-tasks.html

That is, we’d always ship the new Python version half a year after it was
released. I don’t think this corresponds with the Fedora’s “leading edge” stuff,
but that’s what we are left with if we want to avoid “bleeding edge”.

Is there any chance the annual release date could be shifted by at least a month
sooner, or this is what we get no matter what?

(Fedora won’t shift one month later because of holiday season.)

ambv · September 11, 2019, 1:12pm

Hi @hroncok, I moved the discussion to Users so you can respond.

Thanks for your feedback! Having an annual release cycle has the nice property of aligning with Python community events, the two most important being PyCon US (+ the following sprints) and the annual weekly core print.

Moving the annual release calendar forward such that the final freeze happens in September would misalign it with those two events. Due to the fluidity of the calendar, for example in terms of the number of release candidates, this change wouldn’t actually guarantee Python would align with Fedora every year.

Your Plan B of shipping Python always six months after release to me sounds like a big improvement over the current situation. You gave the example of Python 3.8 above. With annual Python releases, shipping 3.X.5 or 3.X.6 with Fedora will make it easier for you to ship related third-party projects that are already upgraded by the community to work with this version of Python. That should help with the exhaustion you mentioned. What do you think?

hroncok · September 11, 2019, 1:40pm

Shipping 3.X.5 or 3.X.6 would certainly make things much easier and help with exhaustion. However Fedora would no longer be driving early adaption of new Python 3.X versions in various upstream projects as it currently does and I’ve always consider that as a benefit for the larger Python ecosystem, not to Fedora itself. This actually sounds like a big drawback over the current situation.

Some datapoints:

Fedora 29 Final was shipped with 3.7.0.
Fedora 26 Final was shipped with 3.6.1.
Fedora 24 Final was shipped with 3.5.1.
Fedora 21 Final was shipped with 3.4.1.
Fedora 18 Final was shipped with 3.3.0.
Fedora 15 Final was shipped with 3.2(.0).

We wanted to ship 3.8.0 in Fedora 31 but didn’t make it, due to the closely misaligned schedules. I want to avoid this in the future, but the current annual release cycle proposal makes it permanent.

When we switch packages to new Python version, we often discover bugs. I’d rather help them discover during betas than during 3.X.5.

Having Fedora and Python release schedules closely aligned (as it happens e.g. with Fedora and GNOME) would IMHO be hugely beneficial to both projects. Having Python releases scheduled as proposed is the worst possible combination with Fedora’s schedule – it either creates an impossible race we cannot win, or it drops the benefits that were present with the “adapt soon” Fedora’s Python relationship so far.

What is the maximum amount of days this can be moved? Is it 0?

We can certainly live on the edge. I just don’t want to satisfy with permanent misalignment.

aeros · September 11, 2019, 2:03pm

Since this has been moved over to the users section, I think it would be a good idea to add a poll to the topic with something along the lines of “Which release major period do you prefer?”, including “9 months”, “12 months”, “18 months”, “24 months”, and “other” as options. Even if some of these are not open for direct consideration, it would be an easy way to get a rough idea for preferences.

The input form that @ambv created will be a great way to get more detailed feedback, but I suspect that it will end up leaving out quite a few users. The results of a single question poll give less information overall, but tend to be more widely appealing since they require far less time investment.

Also, could the post be edited to include a link to the input form at the bottom? Not everyone actively uses twitter.

sildar · September 11, 2019, 2:15pm

Hi!

I’m just a python user, but one advantage of longer release cycles which is not mentioned in the PEP is that it is easier to advocate for a version upgrade when there is a descent number of new features to enjoy.

I can always find one or two reasons to upgrade (dataclasses for 3.7, positional only arguments for 3.8, …), rarely more, and I feel I might lose interest in following new releases when they’re smaller. This goes along with @pitrou’s comment saying that

it seems that our new features are more and more specific, less and less game-changing

aeros · September 11, 2019, 2:21pm

Also, nice work on the graphical models @ambv. I found the first graphic with the visual representations of each release phase and minor version to be especially helpful.

Could we potentially upload it somewhere on python.org when/if this PEP is finalized? I think it would provide a particularly helpful and simple method of communicating the release phases to both users and package maintainers.

steve.dower · September 13, 2019, 3:57pm

I’ve been talking about a possible alternative with a few people and wanted to summarise it here, and I give permission to merge it into the PEP as a [rejected] alternative (or we can move the current proposal into rejected ones I guess )

Fast and Stable releases

The main goal is to minimize the number of releases in bugfix mode (that period between the x.y.0 release and entering security fix-only mode) and also minimize the time between new code/features being merged and becoming available to users.

In short, we have a stable release series that looks basically the same as our current releases (3.x.y). The main bugfix period becomes 2 years, and security-fix period remains as it is. The next release’s beta is 6 months (or 3, I don’t really mind) overlapping with the end of the previous version’s bugfix, so that we only ever have one stable release taking bugfixes.

We also have a fast release series that is calendar versioned (“2019-09”, “2019-10”, etc.) and releases from master every month (or I could see every three months being reasonable). This becomes like a continuous alpha release, and of course us core developers have to take compatibility seriously all the time to avoid breaking users unnecessarily.

In pictures (version numbers and months selected arbitrarily - can realign however we want):

One fast release each month, six-month beta periods and 24-month bugfix:

Alternatively, one fast release every three months and stable releases as above:

For users, the implications are:

if you’re on fast track release X-20YY, you won’t get bugfixes unless you update to (X+1)-20YY
if you’d prefer a slower release, there is exactly one current release, and potentially one beta release (for those whose job it is to make sure it works before upgrading their user’s CPython version)

For library developers:

your own stable releases should work with stable track CPython, and may work with fast track (if desired)
your own prereleases should probably work with fast track, if you want early-adopting users to be able to give you feedback
your minimal test matrix is “current fast track release” and “current stable/bugfix release”, optionally also testing the beta stable release when available

For core developers:

fast track and stable must install side-by-side
we have to become more careful about making any breaking changes (or face broken users within 1-3 months), and probably think about better use of runtime or per-module feature flags