[ACCEPTED] PEP 602: Annual Release Cycle for Python

PEPs
#69

Request for final comments

As discussion on this particular proposal died out and I’m happy with its current form, I’d like to file a motion for final comments on this PEP. Given no additional issues are identified, I’d like to put the PEP under the Steering Council’s consideration on October 17th (two weeks from now).

The Steering Council will likely want to wait with deciding on the PEP until the alternative PEP 605 takes final form and is sufficiently discussed. That’s expected, I don’t mean to rush things. However, release cadence PEPs are special in the sense that they are literally tied to the calendar. I would like to signal that the proposal described in PEP 602 is ready.

One consequence of this is that on Monday, October 14th I will be releasing the first alpha of Python 3.9.0 as described in the PEP above. I thought about this for a while and I hope you won’t be seeing this as forcing anybody’s hand. In fact, should PEP 602 be ultimately rejected, this is a harmless event, we will just have one additional alpha tagged unusually early. However, should PEP 602 be accepted in the future, the calendar described in the PEP will be upheld.

3 Likes
#70

We actually discussed this at the last steering council meeting and we decided to let PEP 602 and PEP 605 try and finish their discussions naturally before we exerted any time pressure to try and resolve this before 3.9 scheduling would be impacted and before the election starts since that will basically freeze up PEP decision making until mid-December.

2 Likes
#71

I hadn’t raised it before, as I thought you were already planning to address it, but the questions in https://www.python.org/dev/peps/pep-0602/#some-policies-depend-on-the-release-cadence need concrete proposed answers in the PEP itself.

It would be fine to make that a simple adjustment that assumes a non-trivial proportion of users will only be running every other release:

  • Deprecation cycles: 2 releases (up from 1)
  • __future__ imports: 2 releases (up from 1)
  • Release manager term: keep at 2 releases (~7 year commitment vs current ~8)
  • Steering Council: 2 releases (up from 1, pending a core dev vote for a PEP 13 update)

That way, these aspects would be essentially the same between PEP 602 and PEP 605, and the core process decision to be made would be between:

  • 3.9.0 in late 2020 (support cycle through to late 2025) + 3.10.0 in late 2021 (support cycle through to late 2026); and
  • rolling releases from late 2019 through to mid 2021 (each only supported until the next one comes out) + 3.9.0 in late 2021 (supported through to late 2026)
2 Likes
#72

source-only releases never fit the brain (nor the need) for Windows users. Any maintenance release shall have installable binaries (and continous integration environements)

#73

Matplotlib, scikit-learn, IPython, Jupyter, yt, SciPy, NumPy, and scikit-image have proposed:

Where:

This NEP recommends that all projects across the Scientific Python ecosystem adopt a common “time window-based” policy for support of Python and NumPy versions. Standardizing a recommendation for project support of minimum Python and NumPy versions will improve downstream project planning.

For example, they suggest:

This project supports:

  • All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions.
  • All minor versions of numpy released in the 24 months prior to the project, and at minimum the last thee minor versions.

And they also note:

Given the current release cadence of the Python, the proposed time (42 months) is roughly equivalent to “the last two” Python minor versions. However, if Python changes their release cadence substantially, any rule based solely on the number of minor releases may need to be changed to remain sensible.

It looks set to be accepted soon.

1 Like
#74

The intended audiences for source-only releases are redistributors and folks that were building their own Python from source anyway (and in many of those cases, it’s going to be the tag in the git repo that matters more than the published tarball).

Folks relying on the python.org binaries are going to want to move on to new releases at the end of the regular maintenance period, since they don’t have any way to effectively consume the source-only security releases. (Neither PEP 602 nor PEP 605 explicitly discuss this, since it isn’t a change from the status quo, but I assume it was one of the factors in PEP 602 reinstating 18 months of regular maintenance for each release).

#75

The Steering Council asked me to address in PEP 602 how policies dependent on the release cadence should be updated. I did that and my answer to those that were identified in the PEP is pretty much: “the policies don’t need to be updated”.

The PEP body in this Discourse topic has been updated with this change. The diff on the peps repository is linked below.

1 Like
#76

I’d like to bump this, because I never got an answer. I personally like 602 way more than 605 (I consider 605 too complicated and not actually solving the problem). However, 605 has a section called Release cycle alignment for prominent Linux distributions. That topic is very important to me and we stopped discussing this entirely. Does it mean it is out of the question? If the release date is synchronized with Fedora and Ubuntu releases, it would really bring benefits, not just to Fedora and Ubuntu, but to the entire Python ecosystem.

I understand that there are PyCons and sprints. But how much are their dates non-negotiable? Can we plan for the release 1 day sooner than currently planned? Can it be 1 week sooner? Can it be 2 weeks sooner? Can it be in September? What are the implications? For example, what precondition other than the release date actually fixes the date of the core sprint to September?

#77

Another argument to align the Python 3.9 schedule with Fedora schedule is that Red Hat has a team working on Fedora to ensure that all Python packages are working well with the next Python (Python 3.9 here). We do bug triage for each broken package and attempt to report issues to upstream (broken projects and Python upstream) as much as possible. We are also actively working on fixing these issues (with our limited resources).

It is related to the PEP 608 “Coordinated Python release” that I just published: https://www.python.org/dev/peps/pep-0608/

1 Like
#78

Core developers have no say in when PyCon is being held. They also have little say in when the annual sprint is happening because finding a date that is sufficiently far apart from PyCon and fits everybody turns out to be pretty impossible. Finding the “US Labor Week” was a fluke on my part and I doubt we will be moving that for future years.

The plan in PEP 602 talks specifically in months and not in concrete dates because all events: PyCon, the annual sprint, committers’ vacation and sick days, are different each year. We also sometimes identify issues that push us back a week or three. So we’re talking in months and not in concrete days. But we’re talking about leeway within the months specified. Sadly I don’t think there is any leeway forward.

Note: with the current status quo the dates are just as non-synchronized. The release schedule of 3.8.0 turned out to be misaligned for Fedora and if we keep the status quo, 3.9.0 will also be misaligned (because we only ever release later than planned, never sooner that planned).

That being said, it would of course be cool to have the release schedules aligned for mutual benefit. But I don’t understand what your needs are. I mean, can you specifically tell us which date you need to move and by how much for this to be helpful to you? For example, @brettcannon suggests that maybe we could call “Beta 4” our “Release Candidate 1”. No dates would change but the stability expectation would be different. You’d have your RC1 in August. Would that help?

1 Like
#79

Looking at current proposal in PEP 596 and the expected Fedora 33 schedule.

At 2020-08-25, Fedora 33 Beta Freeze starts. At that date, Python 3.9.0 is still at b4. Having it as rc1 would be much better. When Python slips, we would satisfy with b4. We want to avoid b3, because there are dangerous changes happening between betas, such as changing the bytecode magic number (and we need to rebuild the whole distro when this happens which is hard after beta freeze, but not impossible).

At 2020-10-06, Fedora 33 Final Freeze starts. Python 3.9.0 final is scheduled 1 day before that. It is not very uncommon that such a release slips by a day or two and what’s worse is that Fedora freezes at 00:00 UTC and Python schedules deadlines as AoE, adding additional 36h difference. That’s why I’d like to aim for at least 1 week earlier. 3.9.0rc2 is currently scheduled at 2020-09-14 which is quite good, but moving it sooner would make me sleep better.

So, technically, this boils down to:

  • move by a week might work if Python doesn’t slip too much (it often does)
  • move by 2 weeks would probably cover for the delays
  • move by 3 weeks would be pretty awesome
  • move by 4 weeks is probably paranoid

It doesn’t really matter that much what is it called, but rather what are the guarantees behind that. (OTOH shipping a Fedora release with an RC version of Python definitively sounds better than shipping it with Beta.) Doing a real RC at the time of current Beta 4 would be indeed nice, however I’m not that sure that it would not require further adjustments of the rest of the milestones. IMHO shifting the schedule by 2-3 weeks instead would make it more likely that we don’t end up with major differences between rc1 and rc2.

#80

The proposal I floated by @ambv was to drop b4 and spread out rc1 and rc2 so they were a month each. So it’s not just a name change but an actual shift in the length of the RCs.

#81

That would probably work for Fedora. But it also makes the beta period shorter by one release. I don’t think that’s wise, but it might work, as the expected amount of changes is lower with the shorter release cycle.

#82

I’m thinking maybe we could eat the cake and have it too by compressing the betas a bit and releasing one every 21 days instead of one every 28 days. After four releases we’d save four weeks et voila.

3 Likes
#83

Then pick your poison: longer beta or longer RC?

#84

From the distributor point of view? Definitively RC. I’ve just assumed that from the core dev PoV it might be Betas.

1 Like
#85

This would work nicely.

1 Like
#86

In the past, when the releases only slightly misaligned, Ubuntu would ship a late beta or rc in the final release and then update it to the final release in a subsequent point-release of Ubuntu. Not ideal, but given that all distros will have to synchronize with Python point releases anyway, is that too onerous?

#87

That would be a thing to plan as a contingency plan if Python slips. I was just trying to avoid this being the default.

1 Like
#88

Final update

As requested by @brettcannon on python-dev and discussed with @hroncok here, I submitted a final update to the PEP. Commit: 493dc56.

I don’t have time now to update the PEP body in the original post here but here’s the diff:

diff --git a/pep-0602.rst b/pep-0602.rst
index 268996ca..efb3c43d 100644
--- a/pep-0602.rst
+++ b/pep-0602.rst
@@ -36,15 +36,16 @@ This PEP proposes that Python 3.X.0 will be developed for around
 - The next *seven months* are spent on versioned alpha releases where
   both new features are incrementally added and bug fixes are included.

-- The following *four months* are spent on versioned beta releases where
-  **no new features** can be added but bug fixes are still included.
+- The following *three months* are spent on four versioned beta releases
+  where **no new features** can be added but bug fixes are still
+  included.

-- The *final month* is spent on a release candidate (or more, if
-  necessary) and concludes with the release of the final release of
+- The final *two months* are spent on two release candidates (or more,
+  if necessary) and conclude with the release of the final release of
   Python 3.X.0.

-One year of full support, four more years of security fixes
------------------------------------------------------------
+1½ year of full support, 3½ more years of security fixes
+--------------------------------------------------------

 After the release of Python 3.X.0, the 3.X series is maintained for
 five years:
@@ -81,11 +82,11 @@ Example
 - 3.9.0 beta 1: Monday, 2020-05-18
   (No new features beyond this point.)

-- 3.9.0 beta 2: Monday, 2020-06-15
-- 3.9.0 beta 3: Monday, 2020-07-13
-- 3.9.0 beta 4: Monday, 2020-08-17
-- 3.9.0 candidate 1: Monday, 2020-09-14
-- 3.9.0 candidate 2: Monday, 2020-09-21 (if necessary)
+- 3.9.0 beta 2: Monday, 2020-06-08
+- 3.9.0 beta 3: Monday, 2020-06-29
+- 3.9.0 beta 4: Monday, 2020-07-20
+- 3.9.0 candidate 1: Monday, 2020-08-10
+- 3.9.0 candidate 2: Monday, 2020-09-14
 - 3.9.0 final: Monday, 2020-10-05

 .. figure:: pep-0602-example-release-calendar.png
@@ -168,6 +169,12 @@ This change provides the following advantages:
 - decreases the urge to rush features shortly before "Beta 1" due to
   the risk of them "slipping for 18 months";

+- allows for synchronizing the schedule of Python release management
+  with external distributors like Fedora who've been historically very
+  helpful in finding regressions early not only in core Python but also
+  in third-party libraries, helping moving the community forward to
+  support the latest version of Python from Day 1;
+
 - increases the explicit alpha release phase, which provides meaningful
   snapshots of progress on new features;

@@ -285,8 +292,21 @@ Double the release cadence to achieve 9 months between major versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 This was originally proposed in PEP 596 and rejected as both too
-irregular and too short.  One consequence of a 9 month release cadence
-was shortening of the beta phase and this was considered dangerous.
+irregular and too short.  This would not give any of the benefits of
+a regular release calendar but it would shorten all development phases,
+especially the beta + RC phases.  This was considered dangerous.
+
+Keep "4 betas over 4 months and a final month for the release candidate"
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While this would make the release calendar a bit cleaner, `it would make
+it very hard for external distributors like Fedora
+<https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/79?u=ambv>`_
+to release the newest version of Python as soon as possible.  We are
+adjusting Python's calendar here in the hope that this will enable
+Fedora to integrate the newest version of Python with the newest version
+of Fedora *as both are being developed* which makes both projects
+better.

 Slow down releases but don't freeze feature development with Beta 1
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5 Likes