Please be careful with backports

As release manager, I have to say, the 3.13.4 release went a little less than optimal. It contained several changes that had unintended consequences, causing problems for people who tried to upgrade. Some were more obvious problems than others. That’s extra unfortunately considering we had a bunch of CVEs attached to the release, which should, generally speaking, make upgrading more important. We’re rolling out 13.5 to fix the issues (hopefully) today.

I hope everyone already knows this, but even so I want to stress how important it is that we try to follow the backward compatibility policy in PEP 387 for major versions to the letter. People rely on us doing so. Of course, mistakes happen. Unintended consequences are sometimes hard to predict. Sometimes fixing a bug is worth potentially breaking weird edge cases. But we have to be deliberate and intentional about these things. We cannot just go “oh, this doesn’t make sense, we should fix it” as a reason to change some behaviour. These things have to be discussed and evaluated. We should put some effort into evaluating the effect on real user code. People have to have the opportunity to provide alternate ideas, or conflicting evidence.

For bug fix releases, we have an additional policy. (That policy is based on PEP 6, which is old, terse and not very explicit [1], but it’s not ambiguous.) Users need to be able to upgrade from release 3.Y.z to 3.Y.z+1 without having to worry it will break things. That means:

  • Be careful with dubious behaviour, which people might rely on.
  • Fixing outright crashes, unexpected exceptions or unreliable behaviour (e.g. randomness in which exception is raised or which type is returned) is fine. Probably.
  • Don’t backport outright behaviour changes.
  • If you had to adapt tests to accommodate your change, that’s a strong signal you shouldn’t backport it, and maybe consider PEP 387 for main.
  • If you had to remove or change a test that explicitly covered the behaviour you’re changing, definitely don’t backport. (I would say at this point you should definitely follow PEP 387 before making the change on main, as well.)
  • In general, avoiding new bugs or changes in behaviour is more important than fixing existing bugs. Existing bugs are bad, but they won’t prevent people from upgrading from the previous bug fix version to the next one. (They might be preventing people from upgrading from 3.X to 3.Y, but at least they still get security fixes in 3.X.)
  • Sometimes serious bugs warrant changes in behaviour. That’s fine, as long as we take all the care we can with these changes, discuss them on discuss.python.org (where people have a chance to see them), and we document them appropriately in What’s New, not just in a blurb item.
  • Sometimes bugs can’t be fixed without changing behaviour in incompatible ways – sometimes the behaviour is the bug – and that means we have to decide to not backport a fix. That’s okay, too.
  • Security fixes are a little special, in that their impact is generally much bigger than regular bugs, and we do sometimes have to backport features or additional bug fixes to make security fixes possible. This is a necessary trade-off, but not a cart blanche to do whatever we want for security’s sake. If we take too many risks or break too many things in security releases, people won’t be able to upgrade and we all lose. (I think we’ve done fine in this respect.)

And again, mistakes can happen. That’s fine! We’re all human. All I’m asking is that we all try, own up to our mistakes, try to fix them and learn from them.

As release manager I can’t review all the changes that go into bug fix releases. I simply don’t have enough time. I rely on everyone involved with reviewing, merging and backporting changes to try and make careful choices here. But if you’re unsure about a change, unsure if you’re interpreting the guidelines correctly, have any doubt what so ever about any change you see, you don’t have to say no. You just have to toss it my way and let it be my decision, and my fault if it goes wrong :slight_smile:


  1. which is honestly a little surprising given who wrote it :joy:. :heart: Anthony ↩︎

38 Likes

I know I asked everyone to find more fixes so I wouldn’t be the only one, but please, you can all stop now! :laughing:

11 Likes

There was also an issue with asyncio task factories that also hit several 3rd party task factories. Introduced in 3.13.4 and fixed in 3.13.5, but already too late, most folks already released work-arounds…

2 Likes