[REJECTED] PEP 605: A rolling feature release stream for CPython

The Windows analogy works well for me, and yes, I avoid the insider builds. So I would probably stick to the Python stable releases, for similar reasons.

As a user, yes, I guess so. Although it’s the 2 year release cycle, rather than the 3.9 release date that matters to me, so shifting the start doesn’t help that much. The actual schedule you quoted (Apr 2021, Aug 2022, Aug 2024) seems a lot more palatable, but it feels like you’re just easing people into the new schedule, and I feel that I should take a view on the actual proposal, not the transition (even though the transition takes 5 years, by which time anything could have happened!)

Ultimately I look at it from a few perspectives:

  1. As a user, faster releases is good so my preference is PEP 602 (Annual), the status quo (18 month) then PEP 605 (biannual, but with complications ;-)) in that order.
  2. As a library maintainer (of libraries that don’t have C code), mostly neutral as we’re more driven by the oldest version we need to support and until Python 2 is finally gone, new features are irrelevant. Faster Python releases might add extra work, but I doubt it’s a lot.
  3. As a consumer of other people’s code, simpler and “not too fast to deter shipping binaries” is better. But as I can only speculate on how the projects I use would be affected, it’s mostly guesswork. The change-averse part of me says that PEP 605 is the loser here simply because the whole beta/ABI compatibility stuff is fundamentally more complex than the status quo, whereas a frequency change is easy to reason about.
  4. As a core developer, I’m not sufficiently involved in release management to feel like I should offer an opinion. If the RMs are happy, so am I.

But overall, I’ve spent way more time considering the implications of this proposal than I ever have on any other aspect of Python’s release schedule. And certainly more than any benefit I’d get would justify.

Agreed, and I hope I’m relatively accurately representing the portion of the userbase that works like I do. But I equally think that people like me are probably the most flexible group, in practice. Getting feedback from organisations that build products in Python is probably way more important here, but they are the group that it’s generally hardest to get input from (because they are often least engaged with open source communities).

Have we had any feedback from organisations like Heroku or Python Anywhere?

1 Like

hi, my own experience trying to build a full WinPython along the Python-3.8 development process:

  • playing well the “python next” game were cpython maintainer and cgohlke,
  • playing not so well was the lack of continuous integration options for package maintainers entirely relying on it,
  • sad experience was the very late community reaction on breaking change to the ProActorEventLoop switch on Windows that occured in beta1 https://bugs.python.org/issue37373 .

So, I would suggest:

  • ensure that there are “continuous integration” branches following each alpha/beta cycle, available to fundation packages at least,
  • plan a small “war group” on each breaking change that is intended, to identify what must be critically adapted downward and help on it, so that the change is a joy at RC time, rather than a continuous pain, for maybe until the next Python cycle.

Having a full Distro released between each alpha/beta cycle is a third step, maybe a too far objective at this point, and too much diluting of force.

1 Like

What do you mean by that? Every release is tagged so it can be tested on its own. But if this is more about getting CI providers to give users access to beta builds then I’m afraid that’s outside our control.

However, it is a relevant point in terms of the practicality of the proposal here. If CI providers like Travis, Appveyor, Azure, etc, don’t provide easy access to the beta stream of releases, it will be quite a lot harder for 3rd party packages to support those releases (and maybe even provide binaries, if they use CI to create binaries).

As you say, this may be out of our control, but I find it hard to accept the arguments in the proposal that betas will be a usable option for people when so much of the ecosystem that makes up a typical Python working environment is dependent on things like this that are out of our control.

2 Likes

as an example, this quotes from Pandas team:

may 8th: : no support at all

august 26 : can’t do anything because of CI

Agreed, though at the same time, I’ve heard that they are hesitant to provide releases that are not endorsed by us. So if we explicitly endorse the rolling release, they’re more likely to pick it up.

Another tactical idea that would help them is if we have a single “latest” tag/build/container that is a stable URL to the current version. Then either their build process or possible even an install-on-demand process doesn’t require a lot of manual effort on their side.

But these ideas don’t belong in the PEP, they’re just things we can more easily explore when we make more robust promises about what it means to update your Python version.

It’s also worth noting that there are major differences between the status quo and the rolling release stream proposed in PEP 605:

  • in the status quo, green buildbots are a “nice to have” for alpha and beta release. In PEP 605, they’re a release blocker, even for alpha releases
  • while it’s not mandatory, I’d expect RMs to be more reluctant to use the “deferred blocker” status in a rolling release world
  • in the status quo, binary builds prior to ABI freeze get the same ABI tags as post-freeze builds, so publishing binaries from CI is a bad idea, even if you get your tests running. PEP 605 strictly separates pre-freeze and post-freeze builds.

However, even with all the current barriers to doing so, there are folks successfully testing against the master branch.

PEP 605 is about asking the question “What might happen if we actively set out to make it easier for projects to do that?”

As far as pre-release availability goes, Travis CI shouldn’t have much trouble offering a new rolling stream alongside their nightly stream, I think we’d have a strong chance to persuade the Azure DevOps folks to get on board, and at least Fedora will make the pre-releases available to any CI pipeline that supports custom Docker images.

It also helps that the pay-off from doing that is much higher if we’re talking 20 months or so out of every 24, vs the status quo, whereas the current beta releases are only around for 4-6 months out of every 18, so it’s tempting to wait the extra few months until the first RC.

It turned out my belief on this point was correct, so there’s now a shared PEP 607 that covers common background points, and provides a neutral overview of the key differences between PEP 602 and PEP 605: PEP 607: Shared background for the release cadence PEPs

I’m also making some minor updates to the text of PEP 605, but nothing that fundamentally changes the proposal (just rewording a few sections based on the creation of PEP 607, updating it to account for NEP 29 updates, and making it more explicit that the Steering Council and release managers aren’t locked in to abide by every single aspect of the concrete proposal, even if they accept the PEP as a general way forward): https://github.com/python/peps/pull/1212/files

Do I understand this correctly that the releases might end up being released for example like this?:

  • 3.9.0a1
  • 3.9.0b2
  • 3.9.0a3
  • 3.9.0a4
  • 3.9.0b5
  • 3.9.0b6
  • 3.9.0rc1

How does that technically work? It seems very unorthodox. As a Fedora Python maintainers we would need to hack around this a lot to make the versions sort properly. (However it’s fair to say that prerelases already need a certain level of hacks to make 3.9.0b5 < 3.9.0 work in RPM).

1 Like

It works because that’s the chronological order, but we expect most people to stick to a “b” version even if there’s a newer “a”. So the semantic ordering doesn’t change for the most part unless you very explicitly opt in to an alpha release.

I wouldn’t expect any kind of distro to make an alpha release available as “current”.

This goes back to what I mentioned above:

I see it as an important point of the adoption rate for both PEPs (in the broader view @pf_moore’s comment above, conda is a CI provider as well).

The conda(-forge) folks haven’t responded in this thread yet (@msarahan @jjhelmus @scopatz @jakirkham @ocefpaf @CJ-Wright @soapy1 @pzwang; not everyone is taggable on discuss), but - for example - each binary-incompatible release also needs to be accompanied with a quite time- and resource-intensive “migration” for conda packages. I’m sure some of them would have an opinion on the frequency of such releases (among other aspects), and thus PEPs 602/605/607.

1 Like

One of the important things to keep in mind is that most ABI breaks don’t break the world - they break specific structures, and only require rebuilds of extension modules that are accessing affected fields.

And if folks want to support the rolling release stream without needing frequent rebuilds, then their best option is to migrate to the stable ABI, which we want to encourage anyway.

Also keep in mind the baseline that we’re comparing to: essentially no pre-built wheels available prior to X.Y.0rc1. That didn’t matter so much 10 years ago, when the only binary format was eggs and most people either weren’t using them or were building their own, but it’s now one of the factors contributing to low levels of pre-release feedback.

That said, if PEP 605 is accepted, I’m sure there are going to be lots of practical details to be worked through to make the rolling release experience an enjoyable one.

1 Like

Fedora’s non-system Python stacks are a likely exception to that - they’re mostly there as a dependency of tox for use in Python CI pipelines, and don’t otherwise have system packages depending on them. We want there to be a Fedora package that tracks the rolling release stream, including the alpha releases.

My own suggestion will be that we deal with it using the existing epoch numbering system so “2:3.9.0a5” would sort after “1:3.9.0b4”.

Yes, in Fedora we’re definitely planning for a python39 package starting with early alphas.
The interleaved alpha/beta versioning would be inconvenient, but solvable (with epoch as a last resort, but let’s not derail the discussion here with specifics).

2 Likes

The packaging metadata Requires-Python assumes PEP 440 version numbers, ordered in the “natural” way defined in that PEP (alphas before betas). In practice, I doubt having a “mixed” alpha and beta ordering will actually affect anyone in that context, but it is technically a violation of that spec. I could imagine, if the rolling release PEP gets accepted and projects supporting the beta releases becomes common, Requires-Python: >=3.10.0b2 could be a spec we’d see in the wild. And if 3.10.0a3 doesn’t conform to that spec, even though it technically should, someone may potentially have a problem. (Yes, I know, “you’re using an alpha, so expect breakage”…)

Overall, mixing alphas and betas like this seems like a fairly gratuitous violation of an extremely common convention, and while we can argue all we like that it won’t affect anyone in practice, it’s still going to break a lot of assumptions.

My understanding of the rolling release stream is that it isn’t just one set of releases but a group of severals streams under one banner.

Given that, Requires-Python: >=3.10.0b2 would only ever see subsequent beta (x.y.zbn) releases (Slow Ring), not the alphas. Whereas, Requires-Python: >=3.10.0a3 would get all the releases (Fast Ring).

Using @hroncok’s list above:

  • Requires-Python: >=3.9.0a3 (Fast Ring)
    • matches: 3.9.0a3, 3.9.0a4, 3.9.0b5, 3.9.0b6, 3.9.0rc1, 3.9.0
  • Requires-Python: >=3.9.0b2 (Slow Ring)
    • matches: 3.9.0b2, 3.9.0b5, 3.9.0b6, 3.9.0rc1, 3.9.0
  • Requires-Python: >=3.9.0rc1 (Release Preview)
    • matches: 3.9.0rc1, 3.9.0

I know that the simple increasing numerical component of the pre-releases look weird, but it still follows PEP 440 § Pre-releases:

Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase.

I could not find where it states that pre-releases must be chronological by phase or the numerical component must be in series.

Update: What I stated wrt Requires-Python holds true within a single release version, once a new version (say, 3.9.1a1) is released, the >=3.9.0b2 selector would pick it up, at least from a Version standpoint, thus breaking the nice aN -> Fast Ring / bN -> Slow Ring correlation.

From my (limited) reading of PEP 440, without some extension (clarification?), the whole Insider Rings comparison cannot hold. Although I do find that association works well for describing these release streams.

1 Like

We don’t. But we make alpha release available as “other” (“alternative”?). We’ve packaged Python 3.8 since 3.8.0a1. And we’d like to continue doing that with 3.9+. Mixed alphas and betas will certainly make it hack-ish, but solvable.

What is more important is that mixed alphas and betas break users’ expectations. And it’s very hard to hack around those.

If my understanding of this model (the PEP, not Windows Insider) is correct, that the release stream can be thought of several independent but increasing stable streams, how would one specify a Requires-Python indicating such (wrt @pf_moore’s concerns)? Not just within one version (say, 3.9.0), but multiple (3.{9,10}.{0,1,2}{a|b|rc}N).

  • Fast Ring (aN)
  • Slow Ring (bN)
  • Release Preview (rcN)
  • Final

From a Version (PEP 440) standpoint, the Fast Ring would be >=3.9a0. That seems straight forward. Say you opt-in to using/testing against the beta channel (Requires-Python: >=3.9b0), what should it be changed to to prevent 3.9.1a5 or 3.10a1 from being matched once those are published?

My understanding, from PEP 607, is that this is basically about providing earlier access to new features (for people who want the early access and are willing to pay a price in terms of stability). Based on that, I think that a time-based understanding of the release numbering is reasonable, and trying to split the releases into “rings” misses the point. If my code uses feature X, then I want to say something like Requires-Python >=3.10.0b5, with the understanding that, stability considerations to one side, I can reasonably assume that any version released after 3.10.0b5 will have feature X. But the mixed beta/alpha approach invalidates that assumption - 3.10.0a6 will be seen as not satisfying that requirement, but will have feature X (barring that feature being withdrawn or massively rewritten, as is possible in an alpha).

I’ll repeat - this is likely to be a rare situation, and will only be encountered by people opting in to a known unstable version. But it is how people view release numbering for pre-releases, and no amount of handwaving the discrepancy away can alter that. I think that the issue should be acknowledged as a downside of the proposal and we just move on. It’s not a showstopper in itself, and a long debate gives the problem more weight than it warrants. My concern is less about any one issue like this, and more about the gradual accumulation of such issues, and how they contribute to an overall feeling that this proposal is “a bit weird” compared to typical release numbering approaches.

2 Likes

Maybe shall the PEP (or a side-PEP) look again at that risk of having two semi-incompatible CI package systems, conda and pip, the community one being not meta-data-ed enough to ease the incompatibility pain.