Right, which we don’t want to do.
Or you get provisional packages, which we no longer want to do.
So we can try “provisional” releases (PEP 605), or more frequent stable releases (PEP 602). But the latter doesn’t look at all like the other ideas we’ve had to solve this problem, so is it really even trying to?
They should, but that section is based on their current plan and they haven’t been consulted specifically on this proposal yet.
Note that unlike real releases, the beta stream are all fundamentally compatible (where the core team gets to determine how compatible, and we argue that we should aim to maintain compatibility better).
So while 3.8 requires new wheels (and as a result, doesn’t have them all yet), each beta release can use the same one. Bugs/changes found between releases would also impact the next release, just as today, but they would be found far sooner.
Right, and I think this is the aspect where the initially published version of PEP 605 painted a picture that’s far more dire than I believe would actually occur in reality. So Paul’s impression of the current text is correct, but the current text is excessively pessimistic regarding the viability of publishing rolling release wheels that target the full CPython ABI rather than the stable ABI.
I’ve put a PR up that I believe should fix that problem: PEP 605: Change ABI management to encourage pre-built wheels by ncoghlan · Pull Request #1190 · python/peps · GitHub
The major change is that it brings back routine alpha releases, but with a very specific purpose: marking ABI breaks in the full CPython ABI.
Thus, there would always be at least one alpha release for each release series: the very first pre-release, which defines the new ABI compatibility tags. Whether there are any more alpha releases between X.Y.0a1 and X.Y.0rc1 would then depend on whether or not we introduced any actual compatibility breaks in the full CPython ABI.
I think you’re ignoring the social issue here, which is that we have never had any real success getting projects to publish binaries for anything other than full releases. We can talk all we like about “fast access streams” or “suitable for limited production use”, but ultimately I expect people to see “beta” and decide not to support that release or provide binaries.
Users have a lot of faith in Python’s compatibility story, but it’s intimately tied to the “full release” cycle. This proposal is, IMO, taking quite a risk by tampering with that perception. And yes, I know that “X.Y.Z is stable and follows the same backward compatibility as always”, but by encouraging people to use the betas because they are “stable enough” we’re creating a confusion where none existed before.
We’re in a better position than ever before to change this, though. We know the people involved in the core projects, often personally, and modern infrastructure is considerably better than the last time our release cycle was updated (witness Miro yesterday trying to get 3.8rc while it was still being pushed).
Projecting the past onto a redesigned future is being unduly pessimistic, IMHO.
(Edit - I’m good with changing the name from beta though, provided the version numbers match our current beta scheme)
The open PR suggests alpha → ABI breaking and beta → binary compatible as a possibility.
However, I do think it would make sense to stick with alpha/beta the first time we do this (assuming we do it at all), as that’s the iteration where the early adopters (on both the end user and library publisher side) are also going to be codesigners and developers of an updated way of working.
If we can’t figure out a way to get the two year cadence with rolling pre-freeze releases to work by 2021 (probably judged by comparing the number of nominally compatible wheels available on PyPI for 3.8 and the 3.9 beta stream), then we’d revisit the notion of switching to an annual cadence for 3.10+.
Note that the status quo is that we actively discourage the publication of binaries prior to ABI freeze, as pre-freeze builds currently use the same compatibility tags as post-freeze ones, but may not be providing the same ABI.
So we actually have a good track record of getting publishers to abide by our requests in this area (there aren’t any technical barriers against publishing pre-freeze binaries, folks just generally know not to do it).
Letting us instead switch to actively encouraging pre-freeze binary builds is the reason PEP 605 proposes the new “pre-release” ABI flag: it means the post-freeze interpreter builds will automatically ignore any pre-freeze wheel builds, so there’s no longer any risk of stable release users hitting a segfault due to an extension module that was built against the pre-freeze ABI.
FWIW, I disagree with the pre-release ABI flag (and I’m not sure whether Nick and I are arguing about it here or on the PR 
).
(Based somewhat on updated text in the PR) If we are guaranteeing that there are no ABI changes since the most recent alpha release, then the only transition where existing builds may stop working is the transition into that alpha release. Everything after that should be fine, and so the only thing gained by adding an ABI flag is to invalidate all the otherwise valid builds at RC time.
(In case it isn’t obvious, any alpha release will be a different X.Y/cpXY from the current stable release, so it won’t get extensions built for the stable release anyway.)
The best I think we could do here is have a version check at initialization to provide quick feedback that the extension was compiled before the most recent alpha. Then it doesn’t matter whether it’s running against a beta or final release - the ABI is compatible, or else we’d have done another alpha and it wouldn’t be the most recent one anymore.
If that were to happen, that would certainly be problematic. Let’s say you rely on a package which skips every odd Python version, and another which skips every even Python version, and both those packages have non-trivial build procedures. Now you must compile at least one of both instead of relying on published binary packages?
I think you should remove the bullet list from the Abstract, because it seems to mirror the ones in the Proposal section, but it’s not clear which one is authoritative and if they’re entirely saying the same thing. I don’t want to have to understand two slightly different summaries of the proposal.
Another piece of feedback is that the part where it’s possible to demote a beta into an alpha release is complicated and will confuse users. I’d say just yank it.
Also an important question is whether this creates significantly more work for the release manager.
Note for readers: the ABI marker discussion between Steve and I will likely only make sense if you’ve read PEP 605: Change ABI management to encourage pre-built wheels by ncoghlan · Pull Request #1190 · python/peps · GitHub, which changes the ABI management proposal to actively encourage the publication of binary wheels during the pre-freeze phase.
Say we end up having three releases that include ABI breaks: X.Y.0a1, X.Y.0a4, X.Y.0a7. The X.Y.0a7 ABI is then the ABI that carries through all the subsequent beta releases and into X.Y.0rc1.
Forcing everyone to rebuild the world every time there’s an alpha release in the rolling release stream would almost certainly lead to publishers deciding supporting the rolling releases was more trouble than it was worth, so we want to allow modules built against X.Y.0a1 to be loaded against X.Y.0a7, as they’re probably going to be compatible. If a module does break though, but works after recompiling, then users of the rolling stream need to ask the affected project to push a new build against the latest alpha ABI, rather than bringing it up with us.
Once we hit X.Y.0rc1 though, we want any binaries that were built against X.Y.0a1 and X.Y.0a4 to be completely gone from the end user experience. It would be nice to be able to keep the builds against X.Y.0a7 and any subsequent beta releases (since it turned out those actually were built against the post-freeze ABI, even if we didn’t know that at the time), but losing them isn’t any worse than the status quo. So the pre-freeze flag is “the simplest thing that could possibly work” - it’s just a new ABI flag, and we already have the tools available to deal with those.
A cleverer scheme that was able to retroactively accept everything built against the last alpha or later would likely be possible, but I don’t think it’s necessary, and I don’t think starting out with a simple pre-release flag would rule out migrating to a cleverer scheme later.
So to be clear, that would mean there is a window for breaking changes from June through Oct every two years (X.Y-1.0rc1 to X.Y.0b1)? PEP 605 – A rolling feature release stream for CPython | peps.python.org doesn’t list alpha timelines.
Alphas can be scheduled whenever necessary by the release manager (so yes, it’s more responsibility for core devs to identify breaking changes and the RM to evaluate them).
So more than just the initial release period, but not as open-season as it is right now. I’d expect the window for breaking changes to narrow, so that we make less of them over time and find more creative ways of enabling new functionality, which will help reduce churn for users.
(Emphasis mine.) Maybe we could acknowledge this in the text, then? That way future readers can look at it and go “okay, the authors weren’t 110% sure this idea was perfect, so it’s okay to try something else” without having to track us down in our nursing homes 
I’ve hit the merge button on that PR now. If the rendered version of the PEP has pictures showing example timelines and support period overlaps, then it’s showing the version of the proposal I’m referring to in the quote above
Relative to the version originally posted at the start of this thread, the substantive change is the one to actively encourage the publication of pre-built binary wheels for a release series as soon as the first pre-release is made available. That change in turn led to the reintroduction of routine alpha releases, specifically to mark pre-releases that include breaking changes in the full CPython C ABI.
The rest of the changes in the update are editorial ones that aim to make the PEP easier to read (adding pictures similar to those in PEP 602, presenting the example timeline earlier in the text, reducing the length of the abstract, etc).
(Note: the in-thread proposal has been updated to match the most recently published version, with a pointer down to this comment to indicate which parts of the thread related to the older version. With a mailing list, I would have just started a new thread for the new version, but that doesn’t seem likely to work as well on a web forum, since old topics don’t decay as quickly).
On re-reading the design discussion section on the alpha/beta naming scheme, I realised I didn’t have this metadata declaration quite right yet, due to the fact that all alpha releases sort as earlier than beta releases.
Instead, I think the required metadata to express the desired constraint would be:
Python-Requires: >= "3.9.0b6"; python_version == "3.9" and python_full_version != "3.9.0a5"
That carves out an exception to allow installation on the alpha build that defines the relevant target ABI, rather than incorrectly allowing installation on all beta builds (even those that preceded the relevant alpha build).
Once the 3.9 series was stable, then those qualifiers could be dropped from the metadata.
It’s weird that a new version stream always starts with a “alpha 1” release and then switches to “beta 2”. While I understand the motivation, it breaks users’ expectation wrt. to alpha and beta releases (usually you get all alpha releases clustered together at one point of the development process, then a sequence of beta releases start from “beta 1”).
I would still recommend that this proposal drops alpha releases altogether, for clarity and ease of adoption.
Otherwise, I’m impressed that this seems to be the first release model proposal that successfully addresses the needs of various groups in the ecosystem without presenting blocking barriers for any of these groups.
I think there’s agreement that noting the abi breaks in some way is important so that the wider ecosystem does not have to repackage the world every two months. IMO this should be obvious straight from the version numbers/tags.
However, I keep circling back that the names “alpha” and “beta” have a lot of historical baggage (and strong connotations from other software projects). In particular, I don’t think the following will hold true in practice:
IMO, these conventions are very hard to replace, and trying to assign a different meaning to the suffix “b” will not stop >95% people reading a version label 3.9.0b3 as a beta version.
@ncoghlan, @steve.dower, why not extend the the pre-release tag “p” (already proposed as a SOABI-marker in this PEP) to be a fully-fledged pre-release marker (cf. PEP 440)? Replacing the usages of “beta” in this PEP with “pre-release” and the “b”-suffix with “p” would IMO solve several problems, the most crucial of which I regard to be that there’s no previous expectations what 3.9.0p3 means (re: adoption of beta releases), and that the naming would explicitly signal that the “p”-releases are considered “above-beta-quality”.
As such, it would even be possible to also “upgrade” the alphas in this PEP to betas (if the idea is to keep the number of possible tags in the rolling stream as low as possible), and denote the ABI-breaks by a “b” (“breaking?”).