Decision-Making Process Improvements (or My Frustrations With How PEP 734 Has Played Out)

TL;DR - The five of us agreed that it was a good idea to associate multiple interpreters in the same process with concurrency. We have the concurrent namespace already that we should actually make use of. We’re not convinced concurrent.interpreters will ever see widespread use, so not claiming yet another top level name for it felt better. The name won’t be the reason.


From my personal perspective, accepting this PEP at all is the surprise I’d expect people to have noted. The feature and module (available for a while now) just has not seen meaningful use. Asking around, the way I understood it is that heavy ML users such as Meta tried and rejected per-interpreter-gil subinterpreters years ago as the lack of Python level data sharing did not match application needs (it added complexity) where ML stacks do massive data sharing at the C level of things they also use at the Python level.

Maybe being in the standard library will increase the chances of adoption. I have my doubts. But for the acceptance side, I would actually like a concurrency world where explicit rather than implicit sharing is further developed in Python and something like this gains traction. Perl 5.8+ adopted this as their only threading mechanism. So if having it exposed in the stdlib encourages that development, maybe it pays off. But the time horizon on that happening is super long and not something we can sit around and wait for. Development resources are naturally focused on the practical thing that has proven to actually work in practice for 3+ decades: the messy world of shared everything free-threading that our underlying OS APIs provide.

So faced with something I doubt is going to see wide use and a general philosophy that we should be using namespaces more often, not occupying a top level name made sense to me as a path forward. I supported that rather than rejecting the PEP’s public module addition. Rejection would’ve been the easier path. Eric would be even less happy.

23 Likes