As requested by @hugovk, I started a new topic yesterday to discuss ideas for rehabilitating inc gc:
He wants to keep this topic focused on 3.14/3.15. So please continue this in the new topic (or create your own).
@nas already posted a patch there that that does that (plus some more)..
It’s nevertheless a major improvement (slashing peak memory use) on the stress tests we have so far. A promising start .
@nas;a original 3-gen design was focused on timely collection of shorter-lived cycles, and I’m confident that moving, as you suggest, back closer to that will pay off. It’s the gen2 collections that caused “long pauses”.
As requested by this topic’s author, and acknowledged by participants, please take development talk for improving the incremental GC to Improving incremental gc.
The reverts are done for 3.14 and 3.15, thanks to @sergey-miryanov and @nas for doing those, and initial test results are looking good, thanks to @zanie and @EWDurbin for helping with those.
This would be a hard breakage of what patch releases mean. I have done Linux distributions and one of the thing you learn with a lot of pain is to never regress in a maintenance release. If a change is good for 90% of the users and bad for 10%, you still can’t do it in a patch release. You need to approach 1000:1 or so …
Because that is what creates the trust that allows users to follow your patch releases every time, without individually judging, which only few people have interest and time for.
If you subscribe to my logic, we are in a bad place. Because we have a hard regression for many in a release and can’t just fix it, because that would mean a regression for a significant fraction of users. (Again 10% is significant.)
Here are what I would do (as a Linux distributor):
If we can, in 3.14.5, ship both GC’s with a startup option, the 3.14.0 behavior stays the default! For 3.15, we could of course change the default (or revert to just one GC, if maintenance burden is a concern).
If we can not ship both, we must stay with the 3.14 inc GC for the 3.14 series. In that case, I would consider an early 3.15 release. Very early, actually. Maybe 3.15 becomes what 3.14.5 would be with very few things on top (we may have a few no-brainers). This would be visibly admitting that we have made a blunder. So be it.
If none of that is workable: We’d need to do a massive information campaign, telling everyone that we have a 3.14.5 that is a 3.15 … Linux distros will hate us, because they have the decision between a rock and a hard place: Break their users (by changing GC) or sticking to 3.14.4 with self-backported patches.
Then it’s a good thing that Python predates SemVer and so we don’t follow it.
We have also been doing releases for decades and in coordination with Linux distros.
It’s a bugfix in a bugfix release, so I disagree it can’t be fixed.
I think you might be misinterpreting what the incremental GC did; it just shortened GC pauses at the apparent expense of worse memory pressure for some people. The generational GC is battle-tested, safe, and what people are used to. This isn’t like we removed a module from the stdlib or something. Most people won’t notice, and those that will asked for this change because of the problems they have run into with the incremental GC. There is no semantic change, just a perf shift which we make no guarantees about anyway.
We had an RC to make sure the change worked for people, it’s already gone out (as Stefan pointed out), and both RC and the final point release were published as widely as possible.
Well, let’s hope not to many people have come to rely on the new inc GC or adjusted to it then … The release notes fail to inform about possible downsides.
Try talking to people who write games in Python. Long gc pauses can kill their “frame rate”, and lead to jerky motion. That may be as close to apps that want “real time” guarantees as Python uses get.
I made a pitch before that the news “should have” spelled out that people who fell into relying on inc gc could find that their workarounds backfire in 3.14.5. These include things like setting threshold0 to a “very low” value to force much more frequent inc collections, and/or calling gc.collect() to force full collections at times the app doesn’t much care. Those are effective workarounds people stumbled into when fighting OOM regressions due to inc gc.
Alas, just plain disabling cyclic gc at the start could make things even worse. Cycles abound in, e.g., “frameworks”, graph applications, and GUI systems.
All changes have downsides too, but we followed the tradition here of only noting the upsides of the change in our top-level news.
Which is, alas, just as true of introducing inc gc to begin with. We guarantee nothing about gc behavior except that objects won’t be deleted until they’re unreachable. It’s all “quality of implementation” stuff - pragmatics. Whether a person calls inc gc or old 3-gen gc “a bug” has no dispositive basis. Each works better for some apps, but worse for others. So do you want jerky animation, or run out of RAM? The answer for me is obvious, but because I don’t do animations in Python . My own apps don’t much care either way.
.