[REJECTED] PEP 605: A rolling feature release stream for CPython

What do you mean by that? Every release is tagged so it can be tested on its own. But if this is more about getting CI providers to give users access to beta builds then I’m afraid that’s outside our control.

However, it is a relevant point in terms of the practicality of the proposal here. If CI providers like Travis, Appveyor, Azure, etc, don’t provide easy access to the beta stream of releases, it will be quite a lot harder for 3rd party packages to support those releases (and maybe even provide binaries, if they use CI to create binaries).

As you say, this may be out of our control, but I find it hard to accept the arguments in the proposal that betas will be a usable option for people when so much of the ecosystem that makes up a typical Python working environment is dependent on things like this that are out of our control.

2 Likes

as an example, this quotes from Pandas team:

may 8th: : no support at all

august 26 : can’t do anything because of CI

Agreed, though at the same time, I’ve heard that they are hesitant to provide releases that are not endorsed by us. So if we explicitly endorse the rolling release, they’re more likely to pick it up.

Another tactical idea that would help them is if we have a single “latest” tag/build/container that is a stable URL to the current version. Then either their build process or possible even an install-on-demand process doesn’t require a lot of manual effort on their side.

But these ideas don’t belong in the PEP, they’re just things we can more easily explore when we make more robust promises about what it means to update your Python version.

It’s also worth noting that there are major differences between the status quo and the rolling release stream proposed in PEP 605:

  • in the status quo, green buildbots are a “nice to have” for alpha and beta release. In PEP 605, they’re a release blocker, even for alpha releases
  • while it’s not mandatory, I’d expect RMs to be more reluctant to use the “deferred blocker” status in a rolling release world
  • in the status quo, binary builds prior to ABI freeze get the same ABI tags as post-freeze builds, so publishing binaries from CI is a bad idea, even if you get your tests running. PEP 605 strictly separates pre-freeze and post-freeze builds.

However, even with all the current barriers to doing so, there are folks successfully testing against the master branch.

PEP 605 is about asking the question “What might happen if we actively set out to make it easier for projects to do that?”

As far as pre-release availability goes, Travis CI shouldn’t have much trouble offering a new rolling stream alongside their nightly stream, I think we’d have a strong chance to persuade the Azure DevOps folks to get on board, and at least Fedora will make the pre-releases available to any CI pipeline that supports custom Docker images.

It also helps that the pay-off from doing that is much higher if we’re talking 20 months or so out of every 24, vs the status quo, whereas the current beta releases are only around for 4-6 months out of every 18, so it’s tempting to wait the extra few months until the first RC.

It turned out my belief on this point was correct, so there’s now a shared PEP 607 that covers common background points, and provides a neutral overview of the key differences between PEP 602 and PEP 605: PEP 607: Shared background for the release cadence PEPs

I’m also making some minor updates to the text of PEP 605, but nothing that fundamentally changes the proposal (just rewording a few sections based on the creation of PEP 607, updating it to account for NEP 29 updates, and making it more explicit that the Steering Council and release managers aren’t locked in to abide by every single aspect of the concrete proposal, even if they accept the PEP as a general way forward): PEP 605: Update for PEP 607 publication by ncoghlan · Pull Request #1212 · python/peps · GitHub

Do I understand this correctly that the releases might end up being released for example like this?:

  • 3.9.0a1
  • 3.9.0b2
  • 3.9.0a3
  • 3.9.0a4
  • 3.9.0b5
  • 3.9.0b6
  • 3.9.0rc1

How does that technically work? It seems very unorthodox. As a Fedora Python maintainers we would need to hack around this a lot to make the versions sort properly. (However it’s fair to say that prerelases already need a certain level of hacks to make 3.9.0b5 < 3.9.0 work in RPM).

1 Like

It works because that’s the chronological order, but we expect most people to stick to a “b” version even if there’s a newer “a”. So the semantic ordering doesn’t change for the most part unless you very explicitly opt in to an alpha release.

I wouldn’t expect any kind of distro to make an alpha release available as “current”.

This goes back to what I mentioned above:

I see it as an important point of the adoption rate for both PEPs (in the broader view @pf_moore’s comment above, conda is a CI provider as well).

The conda(-forge) folks haven’t responded in this thread yet (@msarahan @jjhelmus @scopatz @jakirkham @ocefpaf @CJ-Wright @soapy1 @pzwang; not everyone is taggable on discuss), but - for example - each binary-incompatible release also needs to be accompanied with a quite time- and resource-intensive “migration” for conda packages. I’m sure some of them would have an opinion on the frequency of such releases (among other aspects), and thus PEPs 602/605/607.

1 Like

One of the important things to keep in mind is that most ABI breaks don’t break the world - they break specific structures, and only require rebuilds of extension modules that are accessing affected fields.

And if folks want to support the rolling release stream without needing frequent rebuilds, then their best option is to migrate to the stable ABI, which we want to encourage anyway.

Also keep in mind the baseline that we’re comparing to: essentially no pre-built wheels available prior to X.Y.0rc1. That didn’t matter so much 10 years ago, when the only binary format was eggs and most people either weren’t using them or were building their own, but it’s now one of the factors contributing to low levels of pre-release feedback.

That said, if PEP 605 is accepted, I’m sure there are going to be lots of practical details to be worked through to make the rolling release experience an enjoyable one.

1 Like

Fedora’s non-system Python stacks are a likely exception to that - they’re mostly there as a dependency of tox for use in Python CI pipelines, and don’t otherwise have system packages depending on them. We want there to be a Fedora package that tracks the rolling release stream, including the alpha releases.

My own suggestion will be that we deal with it using the existing epoch numbering system so “2:3.9.0a5” would sort after “1:3.9.0b4”.

Yes, in Fedora we’re definitely planning for a python39 package starting with early alphas.
The interleaved alpha/beta versioning would be inconvenient, but solvable (with epoch as a last resort, but let’s not derail the discussion here with specifics).

2 Likes

The packaging metadata Requires-Python assumes PEP 440 version numbers, ordered in the “natural” way defined in that PEP (alphas before betas). In practice, I doubt having a “mixed” alpha and beta ordering will actually affect anyone in that context, but it is technically a violation of that spec. I could imagine, if the rolling release PEP gets accepted and projects supporting the beta releases becomes common, Requires-Python: >=3.10.0b2 could be a spec we’d see in the wild. And if 3.10.0a3 doesn’t conform to that spec, even though it technically should, someone may potentially have a problem. (Yes, I know, “you’re using an alpha, so expect breakage”…)

Overall, mixing alphas and betas like this seems like a fairly gratuitous violation of an extremely common convention, and while we can argue all we like that it won’t affect anyone in practice, it’s still going to break a lot of assumptions.

My understanding of the rolling release stream is that it isn’t just one set of releases but a group of severals streams under one banner.

Given that, Requires-Python: >=3.10.0b2 would only ever see subsequent beta (x.y.zbn) releases (Slow Ring), not the alphas. Whereas, Requires-Python: >=3.10.0a3 would get all the releases (Fast Ring).

Using @hroncok’s list above:

  • Requires-Python: >=3.9.0a3 (Fast Ring)
    • matches: 3.9.0a3, 3.9.0a4, 3.9.0b5, 3.9.0b6, 3.9.0rc1, 3.9.0
  • Requires-Python: >=3.9.0b2 (Slow Ring)
    • matches: 3.9.0b2, 3.9.0b5, 3.9.0b6, 3.9.0rc1, 3.9.0
  • Requires-Python: >=3.9.0rc1 (Release Preview)
    • matches: 3.9.0rc1, 3.9.0

I know that the simple increasing numerical component of the pre-releases look weird, but it still follows PEP 440 § Pre-releases:

Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase.

I could not find where it states that pre-releases must be chronological by phase or the numerical component must be in series.

Update: What I stated wrt Requires-Python holds true within a single release version, once a new version (say, 3.9.1a1) is released, the >=3.9.0b2 selector would pick it up, at least from a Version standpoint, thus breaking the nice aN → Fast Ring / bN → Slow Ring correlation.

From my (limited) reading of PEP 440, without some extension (clarification?), the whole Insider Rings comparison cannot hold. Although I do find that association works well for describing these release streams.

1 Like

We don’t. But we make alpha release available as “other” (“alternative”?). We’ve packaged Python 3.8 since 3.8.0a1. And we’d like to continue doing that with 3.9+. Mixed alphas and betas will certainly make it hack-ish, but solvable.

What is more important is that mixed alphas and betas break users’ expectations. And it’s very hard to hack around those.

If my understanding of this model (the PEP, not Windows Insider) is correct, that the release stream can be thought of several independent but increasing stable streams, how would one specify a Requires-Python indicating such (wrt @pf_moore’s concerns)? Not just within one version (say, 3.9.0), but multiple (3.{9,10}.{0,1,2}{a|b|rc}N).

  • Fast Ring (aN)
  • Slow Ring (bN)
  • Release Preview (rcN)
  • Final

From a Version (PEP 440) standpoint, the Fast Ring would be >=3.9a0. That seems straight forward. Say you opt-in to using/testing against the beta channel (Requires-Python: >=3.9b0), what should it be changed to to prevent 3.9.1a5 or 3.10a1 from being matched once those are published?

My understanding, from PEP 607, is that this is basically about providing earlier access to new features (for people who want the early access and are willing to pay a price in terms of stability). Based on that, I think that a time-based understanding of the release numbering is reasonable, and trying to split the releases into “rings” misses the point. If my code uses feature X, then I want to say something like Requires-Python >=3.10.0b5, with the understanding that, stability considerations to one side, I can reasonably assume that any version released after 3.10.0b5 will have feature X. But the mixed beta/alpha approach invalidates that assumption - 3.10.0a6 will be seen as not satisfying that requirement, but will have feature X (barring that feature being withdrawn or massively rewritten, as is possible in an alpha).

I’ll repeat - this is likely to be a rare situation, and will only be encountered by people opting in to a known unstable version. But it is how people view release numbering for pre-releases, and no amount of handwaving the discrepancy away can alter that. I think that the issue should be acknowledged as a downside of the proposal and we just move on. It’s not a showstopper in itself, and a long debate gives the problem more weight than it warrants. My concern is less about any one issue like this, and more about the gradual accumulation of such issues, and how they contribute to an overall feeling that this proposal is “a bit weird” compared to typical release numbering approaches.

2 Likes

Maybe shall the PEP (or a side-PEP) look again at that risk of having two semi-incompatible CI package systems, conda and pip, the community one being not meta-data-ed enough to ease the incompatibility pain.

https://www.python.org/dev/peps/pep-0605/#updating-python-requires-for-projects-affected-by-full-c-abi-changes specifically addresses how to set Python-Requires when a project is affected by an ABI break in the rolling release stream: the version gets specified based on the beta stream, and then an environment marker is used to carve out an exception for the alpha release that defines that particular ABI baseline.

It’s absolutely a different way of doing things, but without it, there wouldn’t be a sufficiently clear indication of the cases where we think the rolling stream is going to get a little rough for a month or two.

conda isn’t substantially different from a Linux distro in that regard.

If the mixing of alpha & beta releases proves unworkable in practice, the obvious fallback is to call everything betas, the same as we do today. It would just be unfortunate to perpetuate the situation where upgrading between two consecutive beta releases can cause segfaults due to an extension module being affected by an ABI break. Thus the PEP’s proposal to insert an intervening alpha release as a marker instead.