Decision-Making Process Improvements (or My Frustrations With How PEP 734 Has Played Out)

Core Development
1

FTR, I’m allowing myself to voice my frustration a little. Feel free to disregard the following if it isn’t constructive. Hopefully, though, it helps inspire us to make some changes.

My motivating complaint here is that the request from the Steering Council to put the PEP 734 module under concurrent was a complete, total surprise, and at the last minute no less. There was zero discussion. Using a different name than interpreters had never come up in years of discussion and the consensus was that the name worked.

[Clarification]

tl;dr communication is hard!

I say it was a complete surprise but that is not strictly true. Barry had approached me privately a week or so earlier about the concurrent.interpreters name. I misunderstood his intentions and it did not register in my mind that the name change was something the Steering Council was discussing, much less, seriously. In that regard it was a complete surprise, but, to be fair to Barry, he did bring it up earlier and thoughtfully asked for my opinion. At the time, I gave him a mild rejection and mostly didn’t think about it again. (My mind was occupied elsewhere.) For my part, I claim responsibility for not understanding the situation that week before and thus not prioritizing a stronger response. That hindsight makes the whole situation extra frustrating.

[End Clarification]

The complete surprise I felt at this requirement genuinely canceled out the huge excitement I felt at having the PEP accepted, this at the very realization of a years-long goal and investment. It was a gut punch when I was expecting a hug and I’m still trying to make sense of it. (My internal reaction seems silly to me in terms of “it’s just a name”, and yet here I am. Clearly there’s more to it than “it’s just a name”.) It’s taken me until today to feel comfortable even writing down words for my thoughts here.

At the same time, I know very well the challenge inherent to the operation of the Steering Council. I hope that’s evident from my language summit talk this year (slides). They are doing a hard job with limited resources, which stands as a testament to their feelings for the community. I hope more people understand this clearly.

I also recognize that there wasn’t much time, relative to 3.14, for the Steering Council to start a discussion about the name of the module. Notably, they put in extra effort to make acceptance for 3.14 possible, for which I’m very grateful. It’s a hard situation and I can see where they’re coming from. They had a concern and made an executive decision in the face of an imminent deadline. In my mind, they decided to skip discussion for the sake of landing this in 3.14 and relate to that line of thinking.

That said, those challenges and that tricky situation don’t make the decision right (or wrong). In fact, the conclusion that concurrent.interpreters is the best name still doesn’t make sense to me. I’m still arguing that it’s not a good place for the module. See my other thread for that discussion.

I’ll also point out the reason we even have a Steering Council is because, back in 2018, so many people objected (vigorously) to Guido supposedly making executive decisions without consensus. PEP 13 calls this out specifically. Yet here I’ve twice run into a situation with PEP 734 where the Steering Council mandated sudden, specific, unexpected changes without any discussion: first with putting the module on PyPI (to which I agree in general, but not for PEP 734, but I did anyway) and now with concurrent.interpreters. This is not to insinuate some abuse of power or a deliberate attempt to circumvent our traditional (heavily discussion-based) process. I honestly think the way things are set up right now lends itself to inadvertent situations like this with the Steering Council.

FWIW, I tried to not resist the concurrent.interpreters requirement too much, for worry that the PEP would get bumped to 3.15 (due to the extra time needed for discussion), but maybe sorting out the challenges the Steering Council faces, which contributes to the friction, is more important than not waiting yet another year for stdlib support for multiple interpreters. (To me, the feature is important enough to make that calculus non-trivial.)

Finally, I don’t want to end this with any misunderstanding. I really appreciate the hard work that the members of the Steering Council do. I do not feel like there is any malice or ego involved here either. They genuinely want what’s best for Python and its people, just like all of us do. My frustration lies in there being too little communication and in the consequent surprises, just like I talked about at the language summit. (Ironically, this most recent point of frustration for me took place a few weeks after my language summit talk.) I continue to attribute a fair portion of that (though not all) to insufficient support for the Steering Council’s operation.

I’m a firm believer in our capacity to find solutions to existing friction in our processes, especially given how much we care about each other and this community.

Perhaps this would be a good point to pick up where our language summit discussion left off. Let’s find a way to avoid big surprises in the future.

Anyway, thanks for hearing me out.

35 Likes
2

My impression was that the main goal of the Steering Council was to free Guido from his duty to make decisions.

1 Like
3

I quit in 2018 because people stabbed me in the back on Twitter over decisions they didn’t agree with.

In this instance I am with the SC.

12 Likes
4

We have a Steering Council because Guido got so weary of the abuse he got (peaking with PEP 572) that he just walked away from making any more “executive decisions”. Nobody asked him to, and it wasn’t discussed in advance. This wasn’t a populist revolution, it was a dictator abdicating his position because he was fed up with it.

He just announced he was stepping down, and did so. That left a giant void, and he offered no suggestions for how to fill it. It was up to the community to work it out on their own, and the current structure won a vote among competing proposals.

The “Python community” is sometimes a deeply unpleasant place for many people, for many reasons. Most communities are, after all. Comes with being human. That didn’t really change either when Guido stepped down. We have far more layers of codified “rules” now, but human nature hasn’t changed.

I understand your frustration at having a name imposed on your project that you disapprove of, and seemingly without your participation in the decision-making process - but also understand that the Steering Council likely believes that “the name” was the least of their concerns, not worth delaying action over.

There’s no cure for that, although I think it self-evident that it “would be nice” if the SC had engaged with you directly about this decision, before it was made.

I happen to agree that concurrent.interpreters is a strange name for it (for reasons you already spelled out elsewhere), but like just about everyone else expect I’ll get used to it quickly.

I don’t think changes to procedures can help with this. People need to have sensitivity to how others will take their actions, and the will and the time to “do something” about possibly unpleasant consequences.

Toward that end, I think too it’s important that people speak up when they feel short-changed, as you did here. People are not telepathic, and the alternative to being explicit is passive-aggressive dysfunction of various kinds.

29 Likes
5

I want to clarify something about my point here:

tl;dr communication is hard!

I say it was a complete surprise but that is not strictly true. Barry had approached me privately a week or so earlier about the concurrent.interpreters name. I misunderstood his intentions and it did not register in my mind that the name change was something the Steering Council was discussing, much less, seriously. In that regard it was a complete surprise, but, to be fair to Barry, he did bring it up earlier and thoughtfully asked for my opinion. At the time, I gave him a mild rejection and mostly didn’t think about it again. (My mind was occupied elsewhere.) For my part, I claim responsibility for not understanding the situation that week before and thus not prioritizing a stronger response. That hindsight makes the whole situation extra frustrating.

21 Likes
7

According to PEP 13, the Steering Council

shall work to:

[…]

  • Seek consensus among contributors and the core team before acting in a formal capacity,
  • Act as a “court of final appeal” for decisions where all other methods have failed.

[…]

The council should look for ways to use these powers as little as possible.

Let me emphasize I do not question the good will of the SC members, and I understand that they had the right to consider that the issue of the module name was a detail that they could decide on right away, so as not to prolong the process unnecessarily (especially considering the time constraints for the 3.14 release), but I wonder whether the frustration Eric describes could have been avoided if the council had taken the PEP 13 guidance more literally, even on something as non-critical as the module name.

6 Likes
8

It is puzzling! Best I know, this project has been baking for nearly a decade, with no “bikeshedding” over the name. The new name was imposed without public discussion, let alone an attempt to “seek consensus”. No rationale was given. In fact, I have yet to see anyone speak in favor of the new name on any basis other than that The SC Has Spoken™.

Not saying there isn’t a good technical case to be made, just that I haven’t seen one, and haven’t thought of one.

Compounding the confusion, over in the related topic we saw:

But it the name “is not a core issue”, on what basis did the SC say anything about the name?

Some clarity about this process is needed.

11 Likes
9

TL;DR - The five of us agreed that it was a good idea to associate multiple interpreters in the same process with concurrency. We have the concurrent namespace already that we should actually make use of. We’re not convinced concurrent.interpreters will ever see widespread use, so not claiming yet another top level name for it felt better. The name won’t be the reason.

From my personal perspective, accepting this PEP at all is the surprise I’d expect people to have noted. The feature and module (available for a while now) just has not seen meaningful use. Asking around, the way I understood it is that heavy ML users such as Meta tried and rejected per-interpreter-gil subinterpreters years ago as the lack of Python level data sharing did not match application needs (it added complexity) where ML stacks do massive data sharing at the C level of things they also use at the Python level.

Maybe being in the standard library will increase the chances of adoption. I have my doubts. But for the acceptance side, I would actually like a concurrency world where explicit rather than implicit sharing is further developed in Python and something like this gains traction. Perl 5.8+ adopted this as their only threading mechanism. So if having it exposed in the stdlib encourages that development, maybe it pays off. But the time horizon on that happening is super long and not something we can sit around and wait for. Development resources are naturally focused on the practical thing that has proven to actually work in practice for 3+ decades: the messy world of shared everything free-threading that our underlying OS APIs provide.

So faced with something I doubt is going to see wide use and a general philosophy that we should be using namespaces more often, not occupying a top level name made sense to me as a path forward. I supported that rather than rejecting the PEP’s public module addition. Rejection would’ve been the easier path. Eric would be even less happy.

23 Likes
10

Thanks for exposing the reasoning, @gpshead! That’s all I wanted, and you make some good technical points.

FWIW, I was a bit surprised that the PEP was accepted.

AKA the Sunk Cost Fallacy - but still better than putting it under math :wink:.

7 Likes
11

I might not be the cleverest, but I fail to see how resting a module’s fate on its name is good reasoning. Is the SC saying they begrudgingly accepted this PEP only because they figured they could shove the module under the concurrent namespace, or am I misunderstanding? Why’d you even come out and say such a thing publicly? This is a complete (and completely surreal) kick in the guts for the PEP author. Seriously, what?

12

you’re misunderstanding.

6 Likes
13

While it’s a courtesy conspicuous mostly by absence in most online forums (including this one), life is better if people strive to follow the “principle of charity”:

The principle of charity is a philosophical principle that denotes that, when interpreting someone’s statement, you should assume that the best possible interpretation of that statement is the one that the speaker meant to convey.

In this case, I read @gpshead as saying:

  • The PEP was accepted on merit (else it would have been rejected).
  • But there was little apparent demand for it, and none from “giant users”.
  • The SC’a best guess is that it won’t be widely adopted.
  • So it didn’t merit a top-level name (a “precious resource”).

You can certainly dispute any or all of those, but they’re judgment calls. and all are defensible. I happen to agree with all of them.

The weakest technical point to me is putting it under concurrent. Yes, multiple interpreters can be run in a single process simultaneously, but so, e.g., can multiple files be open in a single process simultaneously. Concurrency isn’t the point of files, or of interpreters.

So while not perfect, I can easily enough accept the SC’s decision.

What remains unaddressed, and unacknowledged, is the meta-issue, highlighted by this topic’s title: “Decision-Making Process Improvements”. PEP 13 (the SC’s charter) counsels (among other things):

seek consensus among contributors and the core team before acting in a formal capacity,

I can’t stretch the principle of charity enough to interpret what was done here as meeting that. Under time pressure, I assume they just skipped that part. But it would, I think, go a long way to restoring harmony if the SC explicitly acknowledged @eric.snow’s frustration as having merit, and they regret that the “seek consensus before acting” part wasn’t followed in this case.

17 Likes
14

I am hugely in favour of applying the principle of good faith in all walks of life. However, I am failing to interpret their statement in any other way:

So faced with something I doubt is going to see wide use and a general philosophy that we should be using namespaces more often, not occupying a top level name made sense to me as a path forward. I supported that rather than rejecting the PEP’s public module addition. Rejection would’ve been the easier path. Eric would be even less happy.

If we take “supported that” to mean “supported not occupying a top level name”, the sentence becomes: “I supported not occupying a top level name rather than rejecting the PEP’s public module addition.” If they were to reject the PEP’s public module addition, what would become of the PEP? It’s either gonna be a top-level module or a submodule. If not the the former, nor the latter, then it seems to follow that the PEP as a whole would be rejected. The following sentence seems to support my reading: “Rejection would’ve been the easier path.”

There are people who are much more devoted to this community and language than I am - and I immediately regretted commenting last night - but reading this forum can be derealisation-inducing at times.

15

I assume that @barry reaching out to Eric was the attempt to seek consensus (at least from the PEP author, and I’m willing to assume that opening the question up to community bikeshedding wasn’t realistic given the tight timescale if this were to go into 3.14). It didn’t work as well as it should have, because of misunderstandings, but mistakes happen, and the intention was there.

I take that comment as meaning that an option the SC was considering was to accept the PEP but make the interpreters module private and undocumented, delaying exposing the public API until 3.15. Rather than do that, putting the module under a namespace was the option @gpshead supported.

But at this point, we’re starting to make a lot of assumptions, and I don’t think it’s helpful. I’d much rather go with simply assuming the SC made the best choice they could in a difficult situation, and that’s what they are elected to do. The post from @gpshead is very helpful in clarifying some of the details behind the decision, and I’d personally rather just be grateful for that extra information - I don’t want people picking over the details of the post to end up discouraging future information sharing like this.

20 Likes
16

PEP 734 has been a tough one.

First, let me state unequivocally that I personally greatly appreciate Eric’s years-long dedication, tenacity, drive, and technical excellence on the subinterpreters feature. Regardless of whether the subinterpreters feature is popular among Python developers, I think the work that has gone into CPython to support them has benefited Python in many ways. The insights Eric has gained by working on subinterpreters (e.g. PEP 683’s immortal objects) has also contributed valuable expertise to other “big initiatives” going on in Python.

Subinterpreters also provides an alternative option for Python users who want to utilize more of the processing power of their computers. E.g. with PEP 684’s per-interpreter GIL, the “global” interpreter lock is a little less global, and thus opens up more use cases for parallelism (some might even say concurrency :wink:).

All that being said, both the 2024 and 2025 SC’s have struggled with PEP 734 in at least two related ways. With apologies to Eric if I am misrepresenting his viewpoint:

  1. Is the API proposed in the PEP the “right” one? API design is difficult and as we all know, once a module is added to the stdlib, it becomes much harder to change and evolve. We’ve tried to address that with “provisional” APIs, but that experiment has been failure, IMHO[1]. We can’t semver stdlib modules, and so a strong recommendation these days is that a module should be released on PyPI, evolve there for a while, and only once its API has settled down and been proven on real-world use cases, should it be considered for the stdlib. PEP 2 needs a serious update.
  2. Is the subinterpreters feature popular enough to “graduate” into the stdlib and claim a top-level stdlib name? Eric has argued that the Python module is so closely tied to the underlying interpreter feature, and useful enough in its own right aside from the utility of the subinterpreters feature, that it does deserve to be in the stdlib and that putting it at the top level gives it the visibility required to make the feature more discoverable, and thus more popular[2].

With these questions in mind, the previous SC recommended a PyPI release of the subinterpreters module to try to answer these two questions: is the API right and is the feature useful enough? Eric did release to PyPI but ISTM rather unenthusiastically, given the hard to find name and the admonition at the top of the PyPI index page. Sadly, I don’t think this PyPI-first experiment really helped answer or clarify either question.

+1 to both of Greg’s statements. I think both 2024’s and 2025’s SC would have been perfectly comfortable rejecting PEP 734 and recommending just having a PyPI module for the, IMHO limited number of actual users of the Python level API. If the “normal”[3] policy for module inclusion into the stdlib had been followed, maybe after a while it would have made sense to add, but as of right now, I remain unconvinced. I’m also unconvinced that adding it to the stdlib will make the subinterpreters feature itself more popular, given all the other ways to do concurrent[4] programming in Python.

And yet, Eric certainly has brought the same tenacity to the PEP 734 discussion as he has to the subinterpreters feature. We’ve had countless discussions in public and private, in SC office hours, at conferences and summits, etc. about 734. I think it’s that tenacity that staved off our strong inclination to reject 734. Small victories, hard won.

As an aside, the documentation is quite good. You should go read it! Maybe it will inspire you to find other interesting use cases for subinterpreters.

Ultimately, it’s just a name, and I still don’t think that the name of the Python module will make any difference in the adoption of the feature. I hope I’m proven wrong, and if the users of it come back to some future SC and make a good argument for why it needs to be renamed, it’s pretty easy to do so, although it might take longer to deprecate the old name. It’s not like that hasn’t been done before though.

And I don’t think concurrent is inherently wrong. IMHO, there’s no perfect place for it within the stdlib, including at the top-level, and at least you can argue that a common use case for multiple interpreters is to boost concurrency. Despite the ability to use them in single threaded programs, I’ve yet to hear of an actual use case for that.

On the process question: as I’ve said before, I commend Eric for speaking his mind about the decision making process, and the SC’s role in Python’s technical evolution. As Tim asks above, I definitely acknowledge Eric’s frustration with the process, but I disagree that the SC hasn’t tried to reach consensus on the issue.

As with so much of Python over its 30+ years, we’re making it up as we go along, and I can proudly say that in the majority of cases, decisions are made with thoughtful deliberation[5], with Python’s best interests in mind, and with compassion and gratitude to all the folks who contribute to it. Human endeavors are messy, we all make mistakes, and we all have our implicit[6] biases. We must be self-reflective and never be afraid to question whether we need a course adjustment, but we also need to “commit and move forward” otherwise we’ll just thrash, and that is also unhealthy to our community.

Elsewhere related to process, I pointed out how to get a topic on the SC’s agenda.

  1. and that’s fine! ↩︎

  2. that’s an argument not dissimilar to the one made for concurrent ↩︎

  3. albeit poorly documented ↩︎

  4. yes, I’m using that word deliberately ↩︎

  5. yes, you can argue maybe too much at times :smiley: ↩︎

  6. and maybe no so implicit ↩︎

34 Likes
17

Thanks for the explanation! I know defending oneself can be a pain in the butt, and irritating to boot, but knowing the reasons makes it easier to live with.

Again, thank you.

9 Likes
18

I see this argument come up on a regular basis but, TBH, concurrency seems to have been a key motivating factor in most of the work done on subinterpreters in the last ~8 years.

@eric.snow 's original 2017 message on python-ideas starts with this sentence:

As part of the multi-core work I’m proposing the addition of the “interpreters” module to the stdlib.

The long-in-the-works-but-finally-superseded PEP 554 has this sentence in its first paragraph

Multiple interpreters operate in relative isolation from one another, which facilitates novel alternative approaches to concurrency.

And of course, a lot of work went into making the GIL per-interpreter in PEP 684, whose Motivation section starts with this very sentence:

The fundamental problem we’re solving here is a lack of true multi-core parallelism (for Python code) in the CPython runtime.

7 Likes
19

I agree. I also recall the recent chain of topics by @eric.snow [1] [2] [3] that attempted to reverse course on free-threading as the solution to Python’s concurrency story and heavily pushed subinterpreters as the alternative path. I am struggling with how to reconcile that push with the current displeasure with the inclusion under concurrent module.

My purely personal outsider impression is that this has less to do with technical considerations than with personal ones such as a desire for respect and recognition of effort, that is, “My hard work deserves more visibility”. It’s a completely understandable and human emotion that I believe can be dealt with in a more productive and healthier [1] manner than turning a bikeshedding molehill into a decision-making concern mountain.

  1. for oneself and also the community ↩︎

2 Likes
20

I’m not persuaded we disagree :smile:. In context, the only thing I was talking about was the only thing @eric.snow talked about in his original post: the very tail end of the process, when the new name was established by fiat.

It’s in that specific, isolated case that I saw no attempt to reach consensus (outside of the SC agreeing with each other in private). Eric later said that you had approached him in private about a week earlier about the name change, but that he missed your intent.

There appears to be an an attempt now to reach an ex post facto consensus, which I appreciate.

Across the years this project was brewing, I said nothing, because I had nothing to say. The process appeared to be working fine, a model on all sides for how things “should” proceed.

This isn’t an itch I have, so I stayed out of it. While I didn’t envision a use for it, Eric’s code changes were careful and generally improved the sometimes-messy code he was changing. You seemed to reach the same conclusion. So it didn’t matter to me that I had no particular use in mind: the project as a whole was benefiting from his changes. I’m, e.g., far more skeptical about the “free threading” code changes.

In Python that will likely be true, at least for some years at the start.

I’ve noted elsewhere that people should look to Tcl instead for “use cases”. Subinterpreters have long been an advertised user-facing language feature in Tcl, but to this day Tcl still doesn’t ship with thread support enabled by default.

As Eric frequently said all along, the primary feature of subinterpreters is isolation. You want to run code that doesn’t mess with “your” state - and can’t.

In Tcl world, this seems most often to come up in the context of “security sandboxes” (for which their seminal flavor of subinterpreters has optional special support; we didn’t go there (yet)).

There are certainly use cases for isolation anyway. For example, running a RAM-hungry package to do initial data-crunching. Even if it doesn’t stomp on my random state or decimal context settings or sys.path (etc, etc), it may leave behind mountains of trash reachable from its modules’ globals, or even if it doesn’t do that, may leave the heap in a badly fragmented state, never releasing memory reservations back to the OS. And so on. You don’t get a dead simple example here because the problems being addressed are messy.

Subinterpreters may be a sufficient solution. When the initial data crunching is done. discard the interpreter it ran in, and with luck its output will be the only visible evidence that it ever ran.

That is an itch I have, but it’s already been scratched: multiprocessing and concurrent.futures with processes are the ultimate in isolation and CPU concurrency. In the kind of case above, concurrency is neither needed nor wanted. Isolation is the whole banana. And I indeed use multiprocessing in such cases now.

9 Likes
21

Yes, there was definitely miscommunication about this last minute naming suggestion, for which I’m sorry.

Quite right. I mentioned rejection of 734 as our preferred option, but the SC could have also deferred it again to 3.15 to give the naming suggestion more time to be discussed.

2 Likes