Hi folks,
Prompted by Łukasz’s work on the annual cadence proposal in PEP 602, and Steve Dower’s initial write-up of a fast track/slow track proposal in the related discussion thread, Steve and I are happy to present PEP 605, a proposal to produce a more regular series of production ready releases from the master git branch: https://www.python.org/dev/peps/pep-0605/
I’ve withdrawn my previous proposal in PEP 598 in favour of this one.
The major difference between this PEP and its predecessors in PEP 598 and PEP 602 is that it focuses on the question of “What would we need to change in order to make our beta releases production ready for at least some environments, and reduce our feature delivery latency that way?”, whereas the other two PEPs both have a much larger potential impact on users that are actually happy with the way our final releases are currently managed.
(Edit: comments prior to [REJECTED] PEP 605: A rolling feature release stream for CPython mostly relate to an earlier draft of the proposal)
PEP: 605
Title: A rolling feature release stream for CPython
Author: Steve Dower steve.dower@python.org, Nick Coghlan ncoghlan@gmail.com
Abstract
Rather than proposing more frequent full CPython releases (as PEP 602 does),
or a policy change to allow backwards compatible feature additions later in a
release series (as PEP 598 does), this PEP instead proposes that we create a
rolling stream of production-ready beta releases, together with alpha releases
that are specifically designed to be suitable as platforms for building extension modules
and wheel archives that are compatible with the subsequent beta releases.
The key desired outcome of this proposal is that the usage guidance given for
beta releases would become “suitable for production use only in environments
with sufficiently robust compatibility testing and operational monitoring
capabilities”, rather than current unqualified “not for production use”.
Similarly, the guidance given for alpha releases would be amended to state
“intended for library compatibility testing and the creation of ABI compatible
binary artifacts”, rather than simply saying “not for production use”.
The PEP authors believe these outcomes can be achieved by amending CPython’s
pre-release management process as described in the Proposal section below.
This PEP also proposes that the frequency of X.Y.0 releases be adjusted to
begin each new release series in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).
Example Future Release Schedules
Under this proposal, Python 3.9.0a1 would be released in December 2019, two
months after the Python 3.8.0 baseline feature release in October 2019.
Assuming no further breaking changes were made to the full CPython ABI, the
3.9.0b2 release would then follow 2 months later in February 2020, continuing
through to 3.9.0b9 in April 2021.
Any time a breaking change to the full CPython ABI was introduced, the first
pre-release that included it would be marked as an alpha release.
3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then
the full release published as 3.9.0 in August 2021.
The cycle would start over again in October 2021, with the publication
of 3.10.0a1 (4 months after the creation of the 3.9.x maintenance branch).
The exact schedule of maintenance releases would be up to the release team,
but assuming maintenance releases of 3.9.x were also to occur every other month
(offset from the 3.10.0 beta releases), the overall release timeline
would look like:
- 2019-12: 3.9.0a1
- 2020-02: 3.9.0b2
- … beta (or alpha) releases every other month
- 2021-04: 3.9.0b9
- 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
- 2021-07: 3.9.0rc2
- 2021-08: 3.9.0
- 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
- 2021-10: 3.10.0a1
- 2021-11: 3.9.2
- 2021-12: 3.10.0b2
- … beta (or alpha) and maintenance releases continue in alternate months
- 2023-04: 3.10.0b10
- 2023-05: 3.9.11
- 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
- 2023-07: 3.10.0rc2, 3.9.12
- 2023-08: 3.10.0
- 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
- 2023-10: 3.11.0a1
- 2023-12: 3.11.0b2
- … etc
If we assume two additional pre-releases were made that introduced breaking
changes to the full CPython ABI in the 3.9.0a5 and 3.9.0a7 releases, then the
overall calendar would look like:
Figure 1. Impact of the pre-release process changes on the calendar.
There are always two or three active maintenance branches in this model,
which preserves the status quo in that respect. The major difference is that
we would start encouraging publishers to provide pre-built binaries for the
pre-freeze rolling releases in addition to providing them for the stable
maintenance branches.
Figure 2. Testing matrix in the 18-month cadence vs. the 24-month
Package publishers targeting the full CPython ABI that choose to provide
pre-built binaries for the rolling pre-freeze releases would at least need
to build new wheel archives following the 3.9.0a1 release. Whether they needed
to publish updated binaries after subsequent alpha releases (e.g. 3.9.0a5 or
3.9.0a7 releases in the example timeline) would depend on whether or not they
were actually affected by the ABI breaks in those later releases.
As with the status quo, all package publishers wishing to provide pre-built
binaries for the final release will need to build new wheel archives following
the ABI freeze date. Unlike the status quo, this date will be clearly marked
by the publication of the first release candidate, and it will occur early
enough to give publishers a couple of months to get ready for the final release.
Motivation
The current CPython pre-release and release management processes were developed
in an era where automated continuous integration and operational monitoring
systems were still relatively immature. Since that time, many organisations
have adopted deployment models that allow them to incorporate new CPython
feature releases without adding substantially more risk than they incur for any
other code change. Newer deployment models, such as lightweight task specific
application containers, also make it easier to combine an application with a
language runtime in a CI pipeline, and then keep them together until the entire
container image is later replaced by an updated one.
In light of those changes in the wider environment, PEP 602 has proposed
reducing the feature delivery latency for the Python standard library and
CPython reference interpreter by increasing the frequency of CPython feature
releases from every 18-24 months to instead occur every 12 months.
Unfortunately, for many organisations, the cost of adopting a new Python release
doesn’t automatically scale down with a reduced number of changes in the release,
as the primary costs aren’t associated with resolving any discovered issues;
the primary costs are associated with the search for issues. This search may
involve manual testing of software systems, human review of written materials,
and other activities where the time required scales with the size of the
existing system, rather than with the number of changes between the versions of
Python.
For third party library developers, costs are primarily associated with the
number of distinct Python versions in widespread usage. This currently tends
to be influenced by a combination of which releases are still actively
maintained by python-dev, and which releases are the latest versions offered
by particular redistributors (with the Debian, Ubuntu LTS, and RHEL/CentOS
system Python versions being particularly popular development targets). In
addition to the basic CI cost of testing against more Python versions, having
more variants in widespread use can make it more difficult to determine when a
fault report is an actual error in the project, or an issue in the reporting
user’s environment.
PEP 602 proposes that affected organisations and projects simply switch to
adopting every second or third CPython release, rather than attempting to adopt
every release, but that creates its own set of new problems to be resolved, both
practical (e.g. deprecations would need to cover more than one release if we’re
expecting users to routinely skip releases) and cultural (e.g. with a larger
number of versions in active use, there is a much higher chance that open source
library maintainers will receive bug reports that only occur on Python versions
that they’re not using themselves).
PEP 598 was an initial attempt by one of the authors of this PEP to propose
an alternative scheme to reduce feature delivery latency by adopting a
semantic versioning style policy that allowed for the incremental delivery of
backwards compatible features within a release series, until that series
reached feature complete status. That variant still had the undesirable
consequence of imposing visible changes on end users that are happy enough
with the current release management model.
This PEP takes the view that both PEP 598 and PEP 602 share a common flaw: they
are attempting to satisfy the needs of two quite distinct audiences within the
constraints of a single release model, which results in conflicting design
requirements, and the need for awkward trade-offs between those conflicting
requirements. The proposal in this PEP aims to avoid that flaw by proposing the
creation of two distinct production-ready release streams, with the existing
release stream being largely left alone, while the new release stream is
tailored towards the audience that would most benefit from a reduction in
feature delivery latency.
Aims of this Proposal
The core of the proposal in this PEP is changing the CPython pre-release process
to produce a rolling stream of incremental feature releases at a regular
cadence, and to ensure that most of those builds offer a sufficient level of
stability as to be suitable for use in appropriately managed production systems.
By adopting this approach, the proposal aims to provide an improved outcome
for almost all Python users and contributors:
- for users of the new incremental feature release stream, targeting the
pre-release phase allows for even lower feature delivery latency than the
annual cadence proposed in PEP 602; - for core developers working on new features, increased frequency and adoption
of pre-releases should improve pre-release feedback cycles; - for users of the established release stream, the increased adoption and
improved feedback cycles during the pre-release period should result in
increased feature maturity at the time of its first X.Y.0 release, as well
as higher levels of ecosystem readiness; - for Python library maintainers, the rolling stream of pre-releases will
hopefully provide more opportunities to identify and resolve design issues
before they make it into a full stable release than is offered by the current
pre-release management process; and - for developers of alternative Python implementations, the rolling stream of
pre-releases may provide an additional incentive for extension module authors
to migrate from the full CPython ABI to the Python stable ABI, which would
also serve to make more of the ecosystem compatible with implementations that
don’t emulate the full CPython C API.
That said, it is acknowledged that not all the outcomes of this proposal will be
beneficial for all members of the wider Python ecosystem:
- for Python library maintainers, both this PEP and PEP 602 would likely
result in user pressure to support the faster release cadence. While this PEP
attempts to mitigate that by clearly marking which pre-releases include
potentially breaking changes to the full CPython C ABI, and PEP 602 attempts
to mitigate it by keeping the minimum time between full releases at
12 months, it isn’t possible to eliminate this downside completely; - for third party extension module maintainers, both this PEP and PEP 602 would
likely result in user pressure to start supporting the stable ABI in order to
provide wheel archives that work on the new version as soon as it is
available. Whether that’s a net negative or not will depend on how the request
is presented to them (it could be a positive if the request comes in the form
of a courteous contribution to their project from a developer interested in
supporting the rolling pre-freeze releases); - for some users of the established release stream that rely on the
availability of pre-built wheel archives, switching to adopting a new release
every 12 months may be an acceptable rate increase, while moving consistently
to the 24 month end of the historical 18-24 month cadence would be an
undesirable rate reduction relative to the 18 month cycle used for recent
releases. Whether this proposal would be a net negative for these users will
depend on whether or not we’re able to persuade library maintainers that
it’s worth their while to support the upcoming stable release throughout its
pre-freeze period, rather than waiting until its API and ABI have been
frozen.
Proposal
The majority of the proposed changes in this PEP only affect the handling of
pre-release versions. The one change affecting full release versions is a
suggested change to their cadence.
Two year cadence for stable releases
With the rolling pre-freeze releases available to users that are looking to
use leading edge versions of the reference interpreter and standard library,
this PEP proposes that the frequency of X.Y.0 releases be adjusted to publish
a new stable release in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).
This change is arguably orthogonal to the proposed changes to the handling of
the pre-freeze release period, but the connection is that without those
pre-release management changes, the downsides of a two year full release cadence
would probably outweigh the upsides, whereas the opposite is true for a 12
month release cadence (i.e. with the pre-release management changes proposed
in this PEP in place, the downsides of a 12 month full release cadence would
outweigh the upsides).
Merging of the alpha and beta phases into a “pre-freeze” phase
Rather than continuing the status quo where the pre-release alpha and beta
phases are distinct and sequential, this PEP proposes that they instead be
combined into a single “pre-freeze” phase with a monotonically increasing serial
number on the releases.
Rather than denoting distinct phases, the “alpha” and “beta” names would
instead indicate whether or not the release contains breaking changes to the
full CPython C ABI:
-
“alpha” releases would be “ABI breaking” releases where extension modules
built against the full CPython ABI in the preceding pre-release are not
necessarily going to load correctly -
“beta” releases would be “binary compatible” releases, where extension modules
built against the full CPython ABI in the preceding pre-release are expected
to load correctly, as long as those modules abide by the following additional
criteria:- the module must not be using any provisional or private C APIs (either from
the previous stable release series, or the in development pre-release series)
that were removed in this beta release, or were changed in an ABI incompatible
way - the module must not be using any C APIs that were deprecated in the previous
stable release series, and removed in this beta release
- the module must not be using any provisional or private C APIs (either from
Pre-freeze phase duration and cadence
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Rather than being released monthly for a period of a few months while preparing
a new X.Y.0 release, pre-freeze releases would instead be consistently published
every two months.
The only time this would not be the case is during the two month release
candidate period for an upcoming X.Y.0 release (see the release candidate
section below for more details). This means two otherwise scheduled releases
would be skipped (one corresponding with the first release candidate date, one
with the final release date).
The pre-freeze phase would typically be expected to start 2 months after the
preceding stable X.Y.0 release.
The first pre-freeze release for any new release series will always be X.Y.0a1
(as there is no preceding release with the same ABI version markers to judge
binary compatibility against).
Pre-freeze releases would gain an additional flag in their C ABI compatibility
markers to avoid binary compatibility issues with the eventual stable release.
Release policy for beta releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This PEP proposes that the policy for beta releases be set as follows:
- as with current beta releases, the stable BuildBot fleet is expected to be
green prior to preparation and publication of the beta release - as with current beta releases, the release manager is expected to review
open release blocker issues prior to preparation and publication of the beta
release - as with current beta releases, any additions to the
abi3stable C ABI would
be expected to become a permanent part of that ABI unless and until that
stable ABI version is retired completely (Note: there are no current plans
to increment the stable ABI version) - unlike current beta releases, beta releases under this PEP would not be
considered feature complete for the next X.Y.0 release - unlike current beta releases, all APIs added since the last CPython feature
release (other than additions to the stable C ABI) would be considered
provisional - unlike current beta releases, beta releases under this PEP would be prepared
and published from the master development branch - unlike current alpha or beta releases, beta releases under this PEP would be
required to be fully ABI compatible with the immediately preceding pre-release
in the series (excluding any changes to provisional APIs, or the removal of
APIs that were deprecated in the previous release series)
Release policy for alpha releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This PEP proposes that the policy for alpha releases be set as follows:
- as with current alpha releases, the stable BuildBot fleet is expected to be
green prior to preparation and publication of the alpha release - as with current alpha releases, the release manager is expected to review
open release blocker issues prior to preparation and publication of the beta
release - unlike current alpha release, the release manager would be expected to
target a similar level of stability to the current beta releases, even
for the alpha releases
Under this PEP, an alpha release would be published whenever it isn’t possible
to publish a release that satisfies the criteria for a beta release, and
allowing some additional time before making the release won’t resolve the issue.
It is expected that the full CPython API changing in a way that breaks ABI
compatibility (for example, a field may have been added to or removed from a
public struct definition) will be the most likely reason for publishing
additional alpha releases beyond the initial compatibility tag defining
X.Y.0a1 release, but the decision for any particular release rests with the
release manager.
Release candidate policy, phase duration, and cadence
Given the proposed changes to the alpha and beta release phases, the release
candidate phase would see the following related adjustments:
- Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
creation would all correspond with the creation of X.Y.0rc1 (currently these
occur across a mixture of X.Y.0b1, the last beta release, and X.Y.0rc1) - The X.Y.0 release candidate period would be extended from 3 weeks to 2 months
- There would normally be two release candidates issued a month apart, but
additional candidates may be published at the release manager’s discretion - The final X.Y.0 release would occur between 1 and 4 weeks after the final
release candidate (depending if additional release candidates were needed
after the second) - If the final X.Y.0 release is delayed beyond the August target date, the
subsequent release series is not affected, and will still be scheduled for
August (now slightly less than two years later).
In addition to allowing more time for end user feedback on the release
candidate, this adjusted policy also provides additional time for maintainers
of Python projects to build and publish pre-built wheel archives for the new
stable release series, significantly improving the initial user experience of
the X.Y.0 release.
Changes to management of the CPython stable C ABI
The CPython stable ABI [5_] makes the commitment that binary extension modules
built against any particular CPython release will continue to work on future
CPython releases that support the same stable ABI version (this version is
currently abi3).
Under the proposed rolling pre-freeze release model, this commitment would be
extended to also apply to the beta releases: once an intentional addition to the
abi3 stable ABI for the upcoming Python version has been shipped in a beta
release, then it will not be removed from future releases for as long as the
abi3 stable ABI remains supported.
Two main mechanisms will be available for obtaining community feedback on
additions to the stable ABI:
- the preferred mechanism will be to add new APIs to the full CPython API first,
and only promote them to the stable ABI after they have been included in at
least one published beta release and received relevant user feedback - for APIs where that approach is unavailable for some reason (e.g. some API
additions may serve no useful purpose when the full CPython API is available),
then developers may request that the release manager mark the next release
as an alpha release (even in the absence of an ABI break in the full CPython
API), and attempt to obtain further feedback that way
As a slight readability and usability improvement, this PEP also proposes the
introduction of aliases for each major stable ABI version::
#define Py_LIMITED_API_3_3 0x03030000
#define Py_LIMITED_API_3_4 0x03040000
#define Py_LIMITED_API_3_5 0x03050000
#define Py_LIMITED_API_3_6 0x03060000
#define Py_LIMITED_API_3_7 0x03070000
#define Py_LIMITED_API_3_8 0x03080000
#define Py_LIMITED_API_3_9 0x03090000
// etc...
These would be used both in extension module code to set the target ABI
version::
#define Py_LIMITED_API Py_LIMITED_API_3_8
And also in the CPython interpreter implementation to check which symbols should
be made available::
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9
// A Python 3.9+ addition to the stable ABI would appear here
#endif
The documentation for the rolling pre-freeze releases and the stable C ABI would
make it clear that extension modules built against the stable ABI in a later
pre-freeze release may not load correctly on an earlier pre-freeze release.
The documentation for alpha releases and the stable C ABI would make it clear
that even extension modules built against the stable ABI in an alpha release
release may not load correctly on the next release if two alpha releases are
published in a row (this situation would ideally be rare).
Changes to management of the full CPython ABI
This PEP proposes two changes to the management of the full CPython ABI.
An explicit NEWS file convention to mark ABI breaking changes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The proposal in this PEP requires that release managers be able to appropriately
mark a pre-freeze release as either an alpha or a beta release based on whether
or not it includes an ABI breaking change.
To assist in that process, core developers would be requested to include a
“(CPython ABI break)” marker at the beginning of all NEWS file snippets for
changes that introduce a breaking change in the full CPython C ABI.
The “CPython” marker is included to make it clear that these annotations relate
to the full CPython ABI, not the stable ABI.
In addition to being useful for release managers, these markers should also be
useful for developers investigating unexpected segfaults when testing against
the affected release.
Explicitly marking builds against the pre-freeze ABI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The full CPython ABI has long operated under a policy where binary
compatibility only applies within a release series after the ABI has been
declared frozen, and only source compatibility applies between different
release series.
This policy means that extension modules built against CPython pre-releases
prior to the ABI freeze for that release series may not actually load correctly
on the final release.
This is due to the fact that the extension module may be relying on provisional
or previously deprecated interfaces that were changed or removed in a later
alpha or beta release, or it may be due to public structures used by the
extension module changing size due to the addition of new fields.
Historically, adoption of alpha and beta releases has been low enough that this
hasn’t really been a problem in practice. However, this PEP proposes to actively
encourage widespread operational use of beta releases, which makes it desirable
to ensure that users of those releases won’t inadvertently publish binary
extension modules that cause segfaults for users running the release candidates
and final releases.
To that end, this PEP proposes amending the extension module SOABI marker
on non-Windows systems to include a new “p” flag for CPython pre-releases, and
only switch back to omitting that flag once the ABI for that particular X.Y.0
version has been frozen on entry to the release candidate stage.
With this change, alpha and beta releases of 3.9.0 would get an SOABI tag of
cpython-39p, while all release candidates and final builds (for both 3.9.0
and later 3.9.x releases) would get an unqualified SOABI tag of cpython-39
Debug builds would still add the “d” to the end of the tag, giving
cpython-39pd for debug builds of pre-releases.
On Windows systems, the suffix for tagged pyd files in pre-release builds
would include “p” as a pre-release marker immediately after the version number,
giving markers like “cp39p-win_amd64”.
A proposed reference implementation for this change is available at [4_] (Note:
at time of writing, that implementation had not yet been tested on Windows).
Updating Python-Requires for projects affected by full C ABI changes
When a project first opts in to providing pre-built binary wheels for the
rolling pre-freeze release series, they don’t need to do anything special: they
would add the rolling release series to their build and test matrices and
publish binary archives that are flagged as being compatible with that release
series, just as they would if providing pre-built binary wheels after the
full CPython ABI freeze for that release series.
However, if the project is affected by a CPython ABI compatibility break in the
rolling release stream, then they will need to issue a version update that
includes both the new binary build, and a new environment constrained
Python-Requires marker.
For example, if a project supporting the rolling release stream was affected by
a CPython ABI compatibility break in the 3.9.0a5 release, then they would add
the following metadata entry on the version that published the updated binary
build::
Python-Requires: >= "3.9.0a5"; python_version == "3.9"
What this does is add an additional compatibility constraint as part of the
published packages, so Python 3.9.x versions older than 3.9.0a5 won’t consider
the updated package as a candidate for installation.
Caveats and Limitations
Actual release dates may be scheduled up to a month earlier or later at
the discretion of the release manager, based on release team availability, and
the timing of other events (e.g. PyCon US, or the annual core developer
sprints). However, as one goal of the proposal is to provide a consistent
release cadence, adjustments should ideally be rare.
Within a release series, the exact frequency of maintenance releases would
still be up to the release manager and the binary release team; this PEP
only proposes an expected cadence for pre-releases and X.Y.0 releases.
However, for the sake of the example timelines, the PEP assumes maintenance
releases every other month, allowing them to alternate months with the rolling
pre-freeze releases.
