This is a larger topic pre-spun off my question about ~bool deprecation.
A topic of discussion at today’s SC meeting, there’s some general concern that changes to Python are sometimes not getting the visibility and deliberation they need to make informed, consensus-driven decisions that serve our users as well as possible.
There are lots of places that discussions and decisions occur. Here on DPO of course, as well as in PEPs, but also GitHub issues and PRs, social media threads, and even in-person meeting such as at conferences and sprints. It’s practically impossible for everyone to keep up on everything, even if they wanted to. And yet, there are lots of channels to monitor for the things you do care about, even as a proxy for our users who in some ways may be the last informed audience in the chain.
There have also been cases where changes happen purely in GitHub PRs, with little to no review. Requiring PR review from at least one other core dev has come up before.
It seems like there’s a structural problem that needs to be addressed. To be clear, I don’t think there’s anything malicious going on - we’re good people trying to do good things. It just seems like we want to be sure as a community that we’re making the best possible decisions for everyone, including and maybe especially our users[1]. It’s always better to do this before we’re committed down a difficult to reverse path, since as we all know, our backward compatibility guarantees often make it challenging and slow to change course.
I don’t have any concrete proposals. I just wanted to open the floor to see if I’m off-base, and if not, to brainstorm possible improvements in our development process.
Some other things to think about off the top of my mind:
Post-commit review isn’t something we should depend on as anything other than a last resort backstop.
What is the role of the Release Manager in all of this? Do RMs have the right level of power to revert decisions? Should they even be the “change cop” who has to monitor everything, even if they could?
Is there a “hierarchy of visibility” we should consider? It’s already fuzzy about when a change needs to rise to the level of a DPO conversation, and then further up to a PEP. That fuzziness is somewhat intentional, and I don’t think we’ll ever have a checklist to say “oh, now this needs a PEP”. We all kind of, mostly, know it when we see it.
Should we be collating an ongoing list of changes, possibly grouped by perceived impact, over and above the “What’s New?” for any particular release? Does anybody monitor the “What’s New?”. Are we good about making sure that impactful changes don’t slip through?
given that Python is one of the most popular programming languages in the world ↩︎
I think the scope of a change comes into play here. Using the ~bool deprecation as an example, that’s a change to a built-in and not some class in some stdlib module (which could also be broken down by the usage/importance of a stdlib module). To me, that suggests more of a need to surface the change.
But then that brings up another question in my head about why does that distinction matter? I think it’s impact, but I’m sure Petr would say all impact matters when it comes to anything that would affect backwards-compatibility. And that then leads to a discussion about whether we all have taken on more than we should with the size of the stdlib on top of the language if we want to treat it all equally.
So yes, I didn’t give any answers, but provided more questions.
I came with other conclusion from this case – deprecation warnings should be more visible. There were several cases when we remove some feature that was deprecated for 8 years, but then were forced to restore it just before a release because it caught many by surprise and broke tens or hundreds packages in popular distributives. We should restore old practice of using DeprecationWarning and PendingDeprecationWarning.
It would help also to prolonge deprecation period in most cases. Currently, people only start to use the freshest Python version 3.12, while 3.13 has not yet been released, and a feature that was deprecated in 3.12 is already removed in 3.14. We just do not have enough feedback about deprecations.
That might be a good discussion point too. One possibility is, rather than a 2 year/release deprecation period, deprecate until the oldest Python that doesn’t have the change goes EOL.
It’s probably the safest thing for us to do. And we have the SC to override this in situations where waiting the 5 years is too painful. Plus with warnings._deprecated() there’s less of a chance to forget. We can also change its signature to take the start of the deprecation and then calculate 6 versions forward by default for when to raise RuntimeError to remember to remove it.
That helps w/ trying to make changes more obvious by having the deprecations last longer and thus a bigger chance people will see it if they skip version upgrades, but is that enough of a visibility increase? Maybe this plus more beta/rc usage by the community?
And that’s visibility externally, but it doesn’t address visibility among ourselves before something hits main. I’m still wondering if we need to talk about how to scale back what we maintain overall so it’s easier to surface more, i.e. people shouldn’t be concerned that they are bothering folks by talking more than not but then everyone also worrying that it would lead to us drowning in discussions. It will admittedly lead to a slowdown in development if all API-affecting, non-bug changes get discussed for a week first, but maybe that’s not horrible either at this stage of Python’s life? Interestingly it might lead to a more coherent API design across the stdlib as more people would participate in its overall design.
Don’t think there’s ever a particularly good answer for deprecations turning into removals. Extending the period, so there’s never a supported Python that has the feature being removed as NOT deprecated, is something - but then it was mentioned some features have been deprecated for far longer than that and still cause surprises. Warnings are a good way to let people know - but many folks hate warnings, as they often break builds that aren’t broken (esp. if “our policy is warnings must be flagged as errors”), or are just plain irritating (e.g. ResourceWarning in short-lived test scripts where it absolutely doesn’t matter are one of my bugaboos). I think there have been several previous threads about whether various warnings should be emitted by default or not.
To some extent, is this not just a variation on the backward compatibility policy?
We have had cases where people have questioned new features that have been added without sufficient discussion (graphlib, for example, and more recently the new REPL which will now have an informational PEP produced for it). But generally, they have been a lot less impactful than incompatible changes. And to be honest, I hope this discussion doesn’t result in making such changes harder to implement - there would be a strong risk of the language stagnating if every new feature, no matter how small, had to go through endless discussion and/or the PEP process.
In my experience, the most controversial changes (in the sense of people feeling they didn’t have a chance to express their views) are those which affect backward compatibility - even if they conform to policy. That suggests to me that there’s inconsistency in how the core devs view the compatibility policy (how strictly it should be applied, or even when it applies). And maybe that’s what we should be looking at?
It’s interesting that your lead into this question was ~bool deprecation.
By my count, 10 core developers were involved in the original issue and PR prior to merge (and several more in e.g. prior python-ideas threads). It’s mentioned in the 3.12 What’s New. That’s a lot!
I think the main way you get more visibility than that is by writing a PEP. Maybe the thinking is we lower the bar for PEPs (but that would also inevitably lead to fewer people paying attention to PEPs / the SC seems like it’s quite busy already)
Disclaimer: I was the one to actually hit merge and I wouldn’t have if there didn’t seem to be clear consensus. I do definitely agree that we should be more willing to have longer deprecation periods than the minimum we often default to.
I think more visibility is always good, since it makes decisions more transparent and potential issues easier to spot.
Many things end up being discussed only on Github issues or even PRs, which, while still openly available, feels like we’re sometimes ducking away from longer discussions.
So +1 on defaulting to starting a topic on Discourse if there’s something to discuss w/r to backwards compatibility or new features which could potentially disrupt existing applications or scripts. It’s better to err on the side of transparency than not and more feedback is helpful in deciding whether or not a change warrants breakage or not.
My understanding is that RMs are tasked with coordinating and creating stable and bug free releases – not with steering the Python language development.
Of course, if an RM finds that a particular change causes major downstream problems during alphas or betas, they can choose to back out and delay the change to the next release, or send it back to the SC for additional review, but RMs should not have BDFL sort of powers. The SC should always have the final say.
I think the above already is our common understanding; it’s just not documented anywhere.
FWIW: I’m watching that document closely, since I need to know whether there’s going to be potential breakage or not – often in areas regular users are not involved in, since I’m doing some low level things with CPython.
I also think that things are constantly improving on the document. It’s probably the most useful resource for Python change monitoring we have.
In my mind, posting on DPO[1] the a middle ground between GitHub and PEP.
To be clear, I don’t think you did anything wrong and I’m definitely not casting any blame! I’m just asking whether there are opportunities to improve our process.
What about regular posting here of the What’s New? I’d love to be better at monitoring it too, but there are just so many channels of information to keep up on!
but maybe something more visible than the Ideas category ↩︎
IMO we should reconsider showing DeprecationWarning by default. For example, we could have:
Python 3.x: PendingDeprecationWarning hidden by default.
Python 3.x+1: DeprecationWarning visible by default.
Python 3.x+2: Feature removed/changed.
So make the incoming visible during at least one Python version. Instead of jumping directly from “DeprecationWarning hidden by default” for 2 releases and then make the incompatible change. PEP 565 “Show DeprecationWarning in __main__” is a half baked solution.
It’s common that users/developers complain that an incompatible change was not announced, whereas Python was emitting a DeprecationWarning for 5 to 10 years. Recent example: the warn() method of the logging module is deprecated since Python 3.3 (2012): for 12 years. I removed the warn() method, and then users started to complain (so I reverted the method which remains deprecated).
If nobody saw the DeprecationWarning on logging warn() for 12 years, maybe there is a fundamental issue with this warning: about its visibility
The above is tangential to the topic of Change Visibility. Could we please not rabbit hole on old DeprecationWarning discussion that seem to be rehashed every several years? We’ve had those before, going back to old behavior of interrupting application users with unactionable unsilencable warnings that only developers of applications can deal with is not the future we want (so says the past).
By the time a new warning is added to our codebase, it is already too late for pre-merge Change Visibility discussions to have happened.
One “wild idea” (not a concrete suggestion!) we discussed as the SC was the idea of a light weight behavior change cursory review/vetting process that we’d encourage everyone to submit changes through. Who could/should review this is an open question, the Release Manager should be in the loop but we cannot burn them out and expect them to do it all; the SC should keep an eye on it but we must also avoid being overloaded. Such a thing would not be intended as a PEP process or centi-thread debate generator – most things don’t deserve that. Outcomes from such a system of sending “Can we make this little behavior change?” questions into such a queue should be a strictly time-boxed “within a week-ish” concrete answer of:
Yes.
No + why not.
Reviewers have mixed feelings, please seek further input on Discuss and loop back afterwards if unclear. (I’d expect “No” to also cause some to do this)
This is amazing but counts as a Big Deal, people are gonna want a PEP to better communicate the significance of what it is, why, how, and a record of thoughts.
I presume most little things will rapidly congeal on one of the first two answers.
The intent as I imagine it would be as a litmus test for behavior change proposals, beyond what PR reviews and Issue discussions normally get. In a central visible place, with a maybe-common broader set of eyeballs.
The intent of such a “light weight behavior change queue” is to both raise awareness of proposed behavior changes and to avoid forcing people into the sometimes daunting seeming PEP process that not everything deserves. It would be intended as a pre-what’s-new before things can be merged check that we, collectively, really want Python to go that route. Done outside of less-visible Github PR or Issue discussions or informal chats specifically so that some consistent guides can be looking at things as a whole with an eye for things that might otherwise have gone unnoticed until it was too late in a release process to effectively undo.
Most non-PEP language behavior changes are not hurt by being ready to go but waiting a week or two before the PR is merged.
Changes that go in only to be reverted or redone later because others weren’t aware of them at the time incur a much higher cost on the project. Going through a light weight process like this doesn’t guarantee everything is good and that such “on second though, lets not” decisions won’t happen. But a success metric for such a system is that we all collectively feel more informed and reverting decisions should thus happen less frequently than the status quo of things having gone in that come as a surprise to many of us later.
Why might this fail? The same reason we’re already collectively unable to stay on top of things: Time vs Volunteers.
How to lay out specific requirements and implement it in such a way it doesn’t just invite bikeshedding down a rabbit hole on everything is an open question that could easily sink the idea on its own.
We could do something as simple as create a DPO channel just for change visibility for things that don’t start out as PEP-sized. Each topic could have a vote[1], something like PEP 10 scores, along with
I’m somewhat concerned that we will end up making participation & contribution more complex and less accessible.
imho, the existing process is already quite cumbersome, and might be scaring away potential contributors (or losing them in the pipeline). there’s some fuzzy hierarchy between GitHub PRs, GitHub issues, Discourse ideas posts, and PEPs, that contributors are expected to navigate. I suspect (to some extent based on my observation from triaging GitHub issues) that more frequently - a potential contributor assumes a typical GitHub-centric process (maybe they don’t even read our issue template), and it falls on a triager or core dev to make some judgement call and perhaps redirect them to Discourse. we then lose contributors that are not interested in figuring out Discourse, or end up with a discourse/issue pair that someone needs to juggle with unclear process (little or no replies on Discourse == support or no support? how long to wait for engagement? how much engagement is needed before proceeding?).
all that to say that adding another component to the “change request flowchart” (i.e. a “lightweight change review” or whatnot) might improve change visibility, but will surely make it harder to contribute.
it seems to me that this thread is mixing at least two different things - change visibility (the title of this thread), and change process.
assuming the discussion is intended to focus on the visibility aspect, I think we don’t have an agreed-upon problem statement, or at least I don’t have a good grasp of what the problem is exactly.
who feels like they don’t have sufficient visibility into changes? what changes they would have liked to have more visibility into? why did those changes fly under the radar of those individuals that indicated they didn’t have sufficient visibility? (e.g. did it skip the Discourse post step? or did the individual miss that post?)
the possible solutions depend on the answers to the above questions, and would come with different costs and tradeoffs if implemented, so I feel that we need a more comprehensive understanding of the problem and its scope and scale.
personally, I don’t have a problem with change visibility myself (maybe because I at least skim all GitHub issues & PRs and Discourse topics), so I’m hoping others can help clarify the problem statement and make it more concrete (with more examples?).
Economics tells us that if the cost of contributing goes up, the number of contributions will go down. In the current environment many of my contributions over past 23 years would not have been possible.
A new requirement for a Discourse discussion will preclude my participation. I haven’t felt safe on this forum for years and in light recent events will strive to avoid it altogether. Compared to tracker discussions, the threads on Discourse are time consuming, lack focus, and tend to attract low quality participation.
Consensus is a wonderful aspiration but rare in practice. For example, we never would have gotten the walrus operator or match-case statement. On the forum, I objected to the new random module CLI but it went in anyway (and now we’re stuck with something that to my eyes is embarrassingly bad).
Reviews are often hard to come by. We would never have gotten the C version of the lru_cache because I didn’t have time to review it. Most of the active participants on this forum never help evaluate feature requests, PRs, and bug reports. This is problematic because the entire purpose of switching to Github was to make it easy to participate.
Github was specifically designed to focus communication for interested participants to actively, cooperatively, and efficiently work towards useful issue resolution. Discourse serves the opposite role, letting people chat, make off-the-cuff remarks, explore pie in the sky ideas, free associate, brainstorm, and build relationships.
And that then leads to a discussion about whether we all have taken on more than we should with the size of the stdlib on top of the language if we want to treat it all equally.
.