Disagree. We’re going to want a huge break at some point, and if we’ve already gotten people used to “major version number changes happen regularly but aren’t that bad” (that is, what is currently a 3.x → 3.y) then we’ll lose the ability to have a very clear mark.
We’ll also lose the likelihood of grouping those changes together, and will end up with “6.x is when half the C API changed and 8.x is when the stdlib became opt-in and 7.x is when arithmetic became faster than C extensions…”. Because any major version number will become sufficient for such major changes when there is enough support.
Keeping the “3.” (what I call the “series”) and changing it no more than once a decade when things change drastically is far more important for our users than the other end of the version number.
As the PEP draft puts it, ‘There are complex pros and cons on both sides of that future choice, and this PEP would merely add one more potential pro in favour of choosing the “Python 4.0” option, with the caveat that we would also need to amend the affected installation layout and compatibility markers to only consider the major version number (rather than both the major and minor version).’
So while I disagree that we’re going to want a huge break again (and if we did, I think we could handle it by changing the reference interpreter name from CPython to SomeOtherName), I am even more firmly convinced that we need to keep that can of worms clearly separated from the question of “Should we give ourselves the option of shipping less disruptive minor feature releases in addition to our current inherently disruptive major feature releases?”
I am tired of discussing the minutiae of version numbering schemes, so I’ll leave that to the experts. However, I am still lurking, and this remark stood out for me. Why do you think that we’re going to want a “huge” break? What do you anticipate happening that cannot be introduced gradually (over a few releases) or in a backwards compatible way? My experience with the 2->3 transition makes me very wary of “huge breaks”.
The examples I gave in my post were the actual ones that came to mind. Changing the C API to allow things like GC or tagged pointer optimizations is the big one (I’m very skeptical we can achieve it in a source-compatible way), and we currently have a pretty well-kept guarantee of source compatibility within the “3” series, which is I think why we’re able to release 3.x updates as frequently as we do.
Of course, if we move to SemVer, then arguably we could break C source compatibility with every major version change. But I think that’s a horrible thing for our users - batching up incompatible changes into a single, big, parallel (for a period) release series is far more manageable than increasing the porting effort required on each release.
(The opposite approach with switching to SemVer is to keep the limits on compatibility that we currently have, despite major version number changes, but I think that will eventually drive Python into obsolescence, or are least seriously weaken its definition as alternative implementations are forced into incompatibility. If we were starting from less leaky abstractions, such as a minimal C API and stdlib, I’d be less concerned, but we’re very constrained by our public API surface and there’s no way to clean that up slowly.)
(Addendum: My main argument is that we should not change version numbers from 3.X.Y to just X.Y. I’m not trying to argue for having a 4.X.Y yet, just that I think we - and our users - are better off if we don’t prevent ourselves from having the option.)
Or getting rid of the GIL, which would also require C API changes. (When we were more hopeful about Larry Hastings’ Gilectomy I had assumed that that would be the major driver for moving to 4.)
BTW I presume you meant “C source compatibility” – of course we have Python source compatibility!
And the C source compatibility is not uniformly strict. There seems to be a hierarchy, where some APIs are allowed to change incompatibly (e.g. the recent PyCode_New() saga).
Yeah, that would break the entire numpy/scipy ecosystem. In fact, even though alternative implementations may not like this, I think we’re in practice constrained to maintain a similar compatibility model for C as well as for Python.
To be explicit about this, are you saying that in this alternative model, C source compatibility would be more strict than Python source compatibility? That doesn’t make much sense to me either.
Sorry for all the questions asking for disambiguation – this thread is too long to read back. All in all it seems you and I actually largely agree that the current versioning scheme is just about optimal in terms of semantics, and that there is no benefit to switching to SemVer.
Yes, I think we agree. It’s just Nick who I’m worried about
(All the clarifying questions you asked are for things we haven’t even discussed in this thread, so they were just poorly stated by me. But yes, Python source compatibility is absolutely critical, C and semantic compatibility slightly less so, but I don’t think we need to derail this discussion further with this topic right now.)
Please read the PEP, as it specifically says we should NOT change the version numbering now. I personally also think we should release 3.10 in 2022, not 4.0.
However, Brett, Guido, and Paul all objected up thread to using the 3rd digit to cover minor feature releases in addition to bug fix releases and security fix only releases, so the PEP concedes that it is indeed odd to use the concepts of semantic versioning without also using that numbering scheme.
I just don’t want perfect to be the enemy of better, so I’d like the folks that care to grit their teeth for now, and if they still care in a couple of years time, try to make their case then.
My objection was to difficult-to-understand rules on what was in a given release. Numbering was part of that (and the most obvious part in your proposal) but there’s also the question of what “minor feature releases” even are, as opposed to our current feature releases.
The PEP distinguishes between “minor feature releases” and “full feature releases”, but while the practical effects to me as an end user are obvious (I can’t install 2 different “minor feature” versions alongside each other) the defining criteria aren’t. You define major releases as “releases that change the filesystem layout and CPython ABI” - but as an end user that doesn’t seem to me like a usable definition, in the sense that I can’t look at the description of a feature and know whether it would be “minor” or not.
To take some concrete examples, I can’t tell which of the release highlights for Python 3.7 would have been “major” and which would have been “minor”. I’m not asking for a detailed breakdown, just trying to illustrate the point that the definitions in the PEP could add to confusion over what’s eligible for a given release (heck, sometimes even the dividing line between bugfix and feature is arguable )
Anyway, that was just a brief clarification. Ultimately, I still don’t like this proposal, but that’s just me, so let’s see what others think.
Yes. What the PEP says on this is what I meant by “the information implied by the first few version fields is more important than the last few”.
I think your definitions of what should/should not change are spot on. But I just can’t shake the sense that what we’re really looking at is having 12 beta releases, where the feature freeze “bar” is raised in certain ways at various points.
Now we know that very few people use our beta releases, but maybe that would change if we extended them to be long enough that they couldn’t be ignored? And presumably the people who want more frequent releases will be quite happy to use one labelled “beta” that fits their desired release schedule, right?
Still haven’t read Nick’s PEP, but I have some recent experience with adding new features to bugfix releases, in the context of the typing module (can’t remember if we did this for asyncio too). (So now I am interested in the minutiae of version numbers again – alas, I have been nerd-sniped. )
It’s not been a great experience – we saw a fair amount of bug reports for mypy from people who had not upgraded to the latest bugfix release of 3.6 or 3.7. The failure modes were mystifying to users and sometimes even to mypy team members (mypy’s release cycle is so much faster than CPython’s that we sometimes are baffled by old bugs).
In a somewhat different context, mypy (and PEP 484) also has Python version checking functionality that is strictly limited to the (major, minor) part of sys.version_info. This has ocassionally confronted us with situations where a certain API was not implemented in e.g. 3.5.0, but did exist in 3.5.1, so were were forced to lie about it. (I can’t remember the specific API, but I recall clearly that there were 1-2 cases like that in typeshed.)
This experience points to two problems with putting features in bugfix releases: you can’t easily have, say. 3.9.0 and 3.9.1 co-exist, and people don’t necessarily upgrade to the latest bugfix release. (I can think of several reasons why this is a costly operation for a large organization.)
Our position has always been that versions that only differ in the 3rd part of the version number should be compatible both ways, and (unless you hit an actual implementation bug) it should be safe to develop on, say, 3.9.5 and expect your code to run on 3.9.0 (as well as on 3.9.6, when it comes out).
I don’t think we should lightly change this. Which leaves us with a faster release cadence. I am so far not objecting to a yearly cadence; but 9 months feels too fast to me.
FWIW I disagree with this, smaller breakages over time are significantly easier to deal with than one big lump breakage. Smaller more incremental breakages over time makes it far easier to break up the work into smaller chunks. When you have a big delta of changes required, particularly if they’re unrelated changes it means you have to fix 10 different things before you can even see if your code runs on the new version, much less works correctly. On the flipside, if each version breaks one thing, and you spread that over 10 versions, then each release has a small impact, and you can fix up that one thing and immediately see if things are running and working correctly now.
The Python 2 → 3 transition is not something to model future transitions after, and it’s somewhat distressing to see it apparently being argued that it should be.
I guess the other thing I’d say is that I don’t think (C)Python is special here and we probably shouldn’t invent our own complex versioning scheme. That just ends up with confusion for everyone, particularly if it looks similar to some existing scheme, but isn’t quite that existing scheme. I don’t even really think it matters that much what the versioning scheme is, as long as whatever it is either standard or REALLY simple (or has no changes, since everyone already knows the current versioning scheme).
Speaking as someone who has been a part of a project (pip) that has in recent history gone through several versioning schemes (Python like, SemVer, and now CalVer), I think all of them are reasonable, though I’m particularly fond of CalVer because it more or less forces you to stop thinking in terms of “breaking release” vs “non breaking release” and instead forces you to think of things in terms of phasing them in over time, because there is no built in escape hatch for making a big Python3-esque change. That doesn’t mean that I think Python should switch to CalVer (I think it could, and it’d be fine) but rather that I don’t think we should be picking versioning schemes on our ability to repeat what happened in the 2 to 3 transition.
I’m not surprised. Similar attempts in CPython were not great either ; even backported performance improvements could sometimes cause unforeseen regressions. There’s a reason why so many software projects have developed a strong distinction between feature releases and bugfix-only releases. By ignoring that accumulated experience we’re just bound to repeat past errors.
I think one occurrence where a feature backport didn’t cause too much trouble is the large SSL backport to 2.7, but that’s probably because it was done so cautiously and minutiously.
I agree with Donald on versioning. For the benefit of our users, I would go even farther to highly encourage us to avoid introducing a versioning change while sunsetting Python 2. It would be far less disruptive to users to follow 3.10, 3.11, 3.12… and publish the cadence that corresponds to them in their PEP.
If there were to be a major architecture change for performance, a PEP at that time could address any name change, version scheme change, etc. Without a more concrete plan for this, we’re merely speculating on what “might” be best.
Why is any library doing version based detection instead of actual feature detection? There might be specific reasons for type checkers to work that way, but most runtime checks should be of the form “Does the API I need exist?”, not “What Python version is nominally running?” (the latter kind of check won’t work properly on Linux LTS releases, since those may backport features without bumping the version number).
That said, I think the most important thing for us to ensure is that the costs of changing the release process are primarily incurred by the core development team - it is not OK for us to make cavalier process changes that significantly increase the support burden for other projects when those projects experience no benefit from the process change (and doubling the release cadence would be such a change - libraries are constrained by the oldest version they support, and that’s dictated by the 5 year branch lifetime - doubling the cadence means doubling the number of releases to support, vs only actively supporting the latest maintenance release in a series).
I think there’s potential merit in the idea of “production ready beta releases” though, such that the minor feature releases carried beta labels, and the dot-zero release continued to mark the transition to feature stability (that point would just occur 12 months later in the release series lifetime than it does now).
I doubt I’d prefer it to just making minor feature releases a thing (since we already effectively make those anyway), but it’s worth considering in the design discussion section.
While in general I agree that if you can get away with a simple “hasattr(, )” that’s preferable over version checking (who can remember when a certain API was added anyway?) I have a few notes here.
As you note, static type checkers can’t ask questions of the runtime, so this is their only recourse. They also can’t know about the antics of Linux LTS releases.
For others, sometimes it’s kind of hard when the API exists but you need to know whether a particular keyword argument is supported, for example. (You can find out for pure Python APIs using inspect, but that’s cumbersome, and who knows how to do this for C extensions.)
I also doubt the wisdom of LTS releases backporting features, unless it’s about essential security APIs. Again, it breaks the property that you can develop on a later bugfix release and deploy to anyone on an earlier bugfix release, modulo actual bugs.
Ubuntu’s approach to the version number problem is simply to append the letters “LTS” to the version of LTS releases. So their versions go like: 17.10, 18.04 LTS, 18.10, 19.04, 19.10, 20.04 LTS, 20.10 …
It’s almost too simple but it seems to work, and does avoid a lot of the issues raised above. And it lets you communicate separately “how big a change is this” (the number) vs “how long will this be supported” (the LTS tag). Which is nice because imo our current model for what we change in each 3.x release is good and should continue, and keeping the same version number system is probably the single thing we can do that will most reassure users that we aren’t going to go break everything.
I think one remaining question is: is there enough williness among core developers to support a LTS version in addition to the usual bugfix branch? Perhaps a poll could give an indication.
Antoine’s question is why I want to avoid the “LTS” phrase: for a community project, I think our current support periods are long enough, and any extension should only occur as part of a separate commercially backed (but PSF coordinated) activity (like the EL Python concept).
I’d be fine with the notion of explicitly labelling the new incremental feature releases, though (probably under that name rather than the “minor feature releases” phrase PEP 598 currently uses - major and minor sharing their first letter is a problem).
So instead of an unqualified 3.8.0, 3.8.1, etc, we’d have 3.8.0 (IFR), 3.8.1 (IFR), etc until reaching feature freeze, at which point we would drop the IFR qualifier.
We couldn’t add a marker for that to the version_info tuple directly, but we could add a “feature_complete” attribute to indicate programmatically that the branch has entered maintenance only mode.
I also think it would be reasonable to require that all features added outside X.Y.0 releases be detectable at runtime without relying on either a version number check or code object introspection. For new attributes, the hasattr check would qualify, but new parameters and behaviours would either need an explicit flag attribute somewhere, or else a fast check API like os.supports_dir_fd and friends (see https://docs.python.org/3/library/os.html#os.supports_dir_fd ).
Static type checkers would need to be enhanced with additional lookup tables if they wanted to support typechecking an earlier IFR while running on a later one, but those could be programmatically generated from the versionadded and versionchanged notes in the documentation.
)
)