I expect that unsophisticated users will just ignore any letters after the version number that they don’t understand. Not sure whether that’s a good or a bad thing. 
My hope/aim is that going between incremental feature releases would be as uneventful as going between maintenance releases is today, so if we made the change and most folks ended up being able to ignore the distinction, I’d consider that a win
This is where I’m mentally landing as well. Hitting beta the same month every year, releasing the final version the same time every year, and just having that kind of regularity while letting us slightly smooth out the differential between X.Y.0 and X.Y+1.0 is a good enough benefit to switch off of an 18 month cycle to a 12 month one.
For me the only questions regarding this approach are:
- How long does the beta stage last?
- In what month do we enter beta? (I.e. do we want to do it shortly after the core dev sprints like we did in 2016?)
- How much does this change the burden on those cutting the releases?
For the last 10 years, we’ve been supporting 2 bugfix branches (2.7 + 3.$LATEST), and ~2-3 security fix branches.
If we switch to a 1 year cadence but keep everything else the same, then we’ll be supporting 1 bugfix branch + 5 security fix branches.
Ubuntu has a 6 month cadence, while supporting 2 bugfix branches plus ~1-3 security fix branches – so actually less support burden than we’ve had for the last 10 years, while shipping features 3x faster.
I’m not sure how many security branches it takes to equal 1 bugfix branch in terms of dev effort, and there are some other subtleties (we probably wouldn’t copy Ubuntu exactly, LTS also interacts with deprecation cycles, the python 2 situation is weird in lots of ways). But hopefully this illustrates the basic point: the reason LTS releases are worth considering is that they might let us reduce our support burden compared to other schemes.
“LTS” is a relative term: it just means that some releases have longer support than others. I don’t think anyone is proposing going beyond our current 5 years of support. So basically right now all our releases are LTS releases, and the question is: if we’re adding more releases, should they also be LTS releases, or should we add some short-term-support releases instead.
I think your proposal is basically an LTS proposal from my perspective… you’re suggesting some feature releases have longer support periods than others. I think using the micro version for feature releases is probably not the clearest way to communicate this, because it conflicts with how other projects use the micro version. But “what should we do” and “how should we communicate it” are different questions.
I think it’s a good idea to consider how we can make feature detection more useful, and it’s worth considering independently of the release schedule discussion. But, I don’t think it’s realistic to expect it to replace all version checks. Making it 100% reliable would be a substantial extra burden on devs. And lots of our users are going to insist on using version comparison checks anyway, because that’s simple and obvious and they don’t know why they shouldn’t. And speaking as someone who does try to do feature detection when possible, I feel like I’ve hit several cases where the “feature” being detected is too subtle for this approach to help. (E.g., subtle behavioral differences that might not even have been intended at the time. Here’s an example where the stdlib does version-based detection, that’s trying to guess whether we’re using a glibc where “enough” bugs were fixed to make posix_spawn useful.)
I guess all numbering schemes have this problem in some form :-). But LTS versions are designed for folks running complex infrastructure where transition costs are high, but support is still important, so it’s reasonable to expect them to have some sophistication. Plus those users are probably already using other projects with LTS releases, so they probably know what those letters mean already.
I think unsophisticated users either run the latest version, or else they just keep running unsupported versions forever… so if they ignore those letters, that’s fine?
Soon we’ll be supporting only 1 bugfix branch.
As for security fix branches, the burden is mostly on their release manager, and even there there’s not a ton of work. There’s probably 50x as many changes in a bugfix branch per year as there’s on a security fix branch.
Actually switching to “no bug fixes after 12 months” is brutally difficult for users to keep up with though (many of them won’t even have started testing it by then), and we have no real justification for it, since “no new features in minor releases, even if fully backwards compatible” is an entirely self-imposed rule based on experiences with the Python 2.2 series almost 20 years ago.
By contrast, a 2 year cadence can provide breathing space between each period of churn, and having a lower latency delivery channel for backwards compatible API enhancements helps create a healthy incentive for folks to consider the question “Is breaking backwards compatibility the only way to achieve my goal, or is there a lower impact way to obtain a comparable end result sooner?”.
Anyway, I’ve posted an update to my proposal here: PEP 598: Adjust terminology and other updates by ncoghlan · Pull Request #1109 · python/peps · GitHub
I strongly disagree with the concept of “semantic versioning”. The idea that you can solve the difficult problem of introducing incompatible changes in software by using a special version number scheme is flawed. The reality is that you have to provide some way for dependent software to smoothly upgrade to new versions. Slapping a new major version number on something and calling the job done is not actually solving the problem.
We experienced that with the change from Python 2 to 3. As a community, we should be well aware of the issues with the SemVer concept. Libraries like ‘six’ were very important to making the transition less painful. As CPython core developers, we made a mistake in not thinking more about the transition process and trying to make it easy. I hope we have learned a lesson from it. Using different version numbers doesn’t solve any of the hard problems.
Yes. I think SemVer is bad because it suggests that as long as the change is in a major version, it is okay. Making incompatible changes is okay (if you don’t do that, your software is going to eventually die), but you need to think carefully how dependent software can transition. Flag days are bad.
Certainly this is going on. The PyCode_New change is a recent example. Cython checks the Python hex version to decide how to call PyCode_New. It should be doing feature detection. However, we didn’t introduce a useful CPP symbol so that Cython can do that. Now that we have decided to change PyCode_New back and use a new API, we are avoiding merging it because we don’t want to change the Python hex version. It is a mess and not the way things should be done. Feature detection is better than version numbers.
I’d just point out that SemVer itself does not mean that the only thing you need to do is bump the version number and you’re done. There are a fair number of projects who interpret it that way, but at the core all SemVer states is that when you do break compatibility, you need to bump the major version number. It’s perfectly within the spirit of SemVer to say, require 2 major versions of deprecation warnings before you drop support for some feature.
Another way to think of it is that the standard practice for safely breaking compatibility tends to have two components, communicating that something is going to break in the future, and communicating that something has broken now. You need both parts for a “safe” backwards compatibility break, SemVer provides the latter but the former is still on the project to ensure (and obviously some projects simply don’t bother).
2 Likes
The big difference is that Ubuntu is time-based releases, while Python is effectively release-when-ready. When I was working on Ubuntu, I did propose to switch Python to a time-based release, and IIRC Shuttleworth even offered to align Ubuntu releases with Python releases if we did that. Of course, that never happened.
In some sense, Python’s 18 month cycle actually worked out pretty well for Ubuntu. I haven’t been involved in Ubuntu development for about 2 years now, so I can’t say what effect if any changing Python’s schedule would have on downstream consumers like Ubuntu.
1 Like
I’m not so sure about that. Even something as simple as updating a tox.ini file for a new Python version can be overwhelming. And you can’t claim you support a new Python version unless it’s actually part of your support matrix.
That’s aside from the very real burden of adopting new Python versions in some corporate environments. I think it’s over simplifying the problem to say that those users can just skip a release. There will be lots of internal pressure to adopt (or at least support).
1 Like
Or we could switch to some kind of calendar based versioning. I.e. for a yearly release cycle, Python 20 would come out in 2020, Python 21 in 2021, etc. We wouldn’t need the minor version numbers that Ubuntu uses unless we switch to a 6 or 9 month schedule.
Sorry, I totally disagree with this. Imagine the scenario in a corporate environment where different entities control rollout of new Python minor releases on their own cadence. Maybe some data centers get X.Y.Z+1 this week and another DC gets X.Y.Z+1 next week. Some upgrades fail so they get stuck on X.Y.Z for longer.
New backward compatible feature gets released in X.Y.Z+1, and some developers start using it. Now their code runs in some DCs but not others, and that can cause major breakages. You can argue that automation and other constraints would limit the use of new X.Y.Z+1 feature until X.Y.Z+1 has been rolled out everywhere, but I claim that’s a nearly impossible state to guarantee.
Even in the open source world, I think introducing new features in a point-release is asking for a world of hurt. Let’s remember the lesson we learned with adding True and False in 2.2.1.
1 Like
On a historical note, Guido and I talked about this recently. We used to just deprecate the last version when the new one was released. E.g. 1.4 support stopped when 1.5 was released. I think this was mostly because we had a much less sophisticated VCS and CI, but also because we could get away with it; the user base was much smaller then.
Obviously we can’t do that today.
That argument doesn’t hold Barry - if you have that kind of environment, just decline to deploy (or strictly limit the use of) new Python versions until they hit FCR, or else set Python-Requires appropriately on all your deployments, so DCs running old versions of Python, will also keep running the compatible versions of everything else. An organization’s CI pipelines should also be restricted to the oldest still deployed IFR, rather than the newest.
2.2 was a long time ago, and Python app deployments have changed massively since then - we’re not talking about humans trying to cobble together a working system by downloading tarballs from an assortment of different websites anymore - we actually have machine readable metadata, and tools to automate putting together a consistent system.
(Plus the usual corporate argument applies: I put zero weight on proposals that ask volunteer publishers to do more work for the sake of enormous organizations that aren’t managing their development processes properly)
1 Like
A more recent painful example was the introduction of typing.NewType in Python 3.5.2. I can forgive that one because typing is officially provisional.
The other problem with adding new features in point releases is that it conflates bug fixes and security fixes with new features. If we have to adopt X.Y.Z+1 to address a critical bug or security issue, that means that we’re also adopting all the new features in X.Y.Z+1. It will be difficult to know what new features have been added and to add blockers that prevent developers from using those new Z+1 features.
Take the NewType issue mentioned above. I’m not even sure how that would be workable. It would mean that we would either have to have a feature matrix (i.e. “You can’t use NewType because 3.5.1 is still deployed somewhere”) or you have to keep all old versions live in your CI and run your CI/CD against all those versions. How else would you catch that a developer snuck in a reference to NewType that works almost works everywhere?
1 Like
OK, I think I figured out a way to better express why the “But compatibility with older point releases…” argument confuses me. Here’s Barry’s hypothetical example on that front:
Now consider that same hypothetical scenario, but with the second paragraph adjusted to be about a bug fix rather than a new backwards compatible feature:
A bug gets fixed in X.Y.Z+1, and some developers start omitting their previous workaround for the standard API being broken (or perhaps are using the API in new code and don’t even realise that a workaround used to be required). Now their code runs in some DCs but not others, and that can cause major breakages. You can argue that automation and other constraints would limit the reliance on X.Y.Z+1 bug fixes until X.Y.Z+1 has been rolled out everywhere, but I claim that’s a nearly impossible state to guarantee.
Basically, I don’t understand how “New APIs that existing code necessarily isn’t using (as the API didn’t previously exist), and is marked in the documentation as added or changed in a particular point release” can possibly be more of a threat to the reliability of staged rollouts than bug fixes in existing APIs. And if measures have been put in place to handle the “Don’t rely on bug fixes that may not have been rolled out universally yet” scenario than those same measures (which are already needed today) will be just as effective in addressing the “Don’t rely on feature additions that may not have been rolled out universally yet” scenario that the incremental feature release proposal in PEP 598 introduces.
(I’ll also note that this particular argument concedes the point that PEP 598 would work in getting features into the hands of developers sooner, since it only causes new problems if developers are actually using the new features we’d be delivering to them. By contrast, PEP 596 won’t do anything to help a large proportion of developers, since it requires that a new major Python version be made available in their target environments, and that they switch their project to target it, rather than taking the much lower risk step of updating to a new point release of an existing major version)
1 Like
You pin your CI to run the oldest still deployed Python in your production environments. If that’s Python 3.5.1, then that’s what you run in CI - you’ll pick up both reliance on missing features, and you’ll pick up missing workarounds for bugs that are still present in 3.5.1.
For generic publishers that aren’t targeting any particular environment, you’d test against the latest point release, and drop official support for older point releases as soon as a new one comes out (this is effectively what already happens today when public CI providers update their Python runtimes to a new point release).
The difference is that those kinds of bug workarounds are less frequent and can be worked around in local code unconditionally. It does mean you tend to carry around those workarounds for a long time.
But it means that if you’re pulling in Z+1 to fix a bug, you can only defend against incremental new features by backporting or reimplementing every one of them. That’s essentially what we did with NewType. But with this proposal, I now have to track every single new feature from .0 to .Z in additional to tracking bug fixes in those versions.
I don’t want to treat new features as something I have to defend against. 
I think that’s a wildly optimistic view of corporate infrastructure, especially when there are 100ks or more machines. It effectively means you’d be pinning to point releases instead of major releases. Do you really want to do that?
1 Like
But is that benefit to you in a well-funded corporate environment worth the cost of gating every feature addition to Python behind the backwards incompatible filesystem layout changes that mean major feature release adoption cycles are measured in months and years, rather than the weeks and months of minor point releases?
It’s also the case that the versionadded/versionchanged notes in Sphinx generate a machine-readable inventory of when new APIs were added, which means that a static analyser could be built that caught the use of APIs that didn’t exist in the older version.