"interpreters" vs. "concurrent.interpreters"

Core Development
1

Hi all (and especially the Steering Council and our 3.14 RM),

The new module from PEP 734 is happily part of 3.14 now, the culmination of the primary phase of a project I started over a decade ago. I’m excited at the possibilities that are now open to our users. Thanks to all who have helped!

Now on to the bikeshed! I added the module as concurrent.interpreters per the instructions of the Steering Council. If we hadn’t been up against the 3.14 deadlines, I would have focused on making my case against the name change. However, I’m still concerned that concurrent.interpreters isn’t the right place for the module and I still hope there’s a chance to rename it to just interpreters.

Here’s why:

  • interpreters says exactly what it’s about
  • it doesn’t feel (to me) like the right place for the module (isn’t where I’d expect to find it)
  • the module is about exposing a core runtime feature, like sys, gc, atexit, or imp/importlib, so top-level makes the most sense
  • the module is only vaguely concurrency related, in the same way exec() or subprocess is related to concurrency
  • there’s a strong likelihood that users will be confused about what the module is for, because of “concurrent”
  • concurrent.interpreters vs. concurrent.futures.interpreter is confusing
  • it’s a longer name than interpreters
  • I’ve put a lot of thought, time, and effort into the module and the underlying runtime functionality, and with that context concurrent.interpreters makes no sense to me
  • the proposed module name has been interpreters since before 2018, with consensus and no suggestions to do anything else until literally the last minute, in the Steering Council’s acceptance message

Overall, the justification for the change to concurrent.interpreters (counter to the author’s intention and to the long-standing consensus) is unclear.

I’ve heard the following arguments in favor of concurrent.interpreters:

  • we should make better use of the concurrent namespace (agreed, but not with PEP 734)
  • discoverability (I’m still unclear on how concurrent.* helps with that here)
  • if you squint just right, it’s sort of concurrency-related

(I don’t mean to misrepresent the opposing position to my own; I literally don’t know more than this about the arguments in favor of concurrent.interpreters.)

Here are some public comments core devs have made both ways about the module name:

and some relevant comments from others:

Ultimately, I’d like to change the name back to interpreters (for 3.14+).

I realize it’s awfully late in the 3.14 cycle to be making this change. I’ll understand if we’re already past the point of no return. That said, I’ll also note that changing the module name before beta4/rc1 would be unlikely to introduce any instability nor be disruptive.

Thoughts?

p.s. Sorry I didn’t write this sooner. It’s been hard for me to process the situation (and I have a lot going on) and I’m only now comfortable with writing this down.

36 Likes
2

FWIW, I completely agree we should make better use of the concurrent namespace. It’s the perfect place for tools related to concurrency in general. That would include helpers for using multiple interpreters for concurrency specifically. (However, the PEP 734 module has never been that.)

For example, I’m working on a PEP for 3.15 related to cross-interpreter data and at least some of that capability will be general to safely sharing data between abstract threads of execution, and a module for that would be a great fit under concurrent.

2 Likes
3

FYI, I started a separate, related thread about some of my frustrations with how the decision-making has played out with PEP 734.

1 Like
4

IMO, we already have some significant difficulties with teaching subinterpreters. I’m sure that there are plenty Python developers who don’t understand the concept of subinterpreters clearly. For example, I’ve obtained a proper view on this topic in just last few weeks. And literally yesterday I’d answered some questions about subinterpreters and free-threading concepts at our local Python community.

It’s hard to explain even for advanced Python user, why we need subinterpreters if there will be some free-threading builds soon. Of course, we need good docs about it, but concurrent.interpreters import name will definitely hardens these explanations, not simplifies them.

13 Likes
5

PEP-1 describes a process for PEPs that include ideation, endorsement by a core developer, and disucssion by all interested parties. I wonder if perhaps the proposal was written with more of a core developer audience in mind, resulting in an explanation that’s more “tree-focused” than outlining the vision of the forest.

The PEP 734 says it’s a continuation of PEP 554. 554 says,

CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. [c-api] Multiple interpreters operate in relative isolation from one another, which facilitates novel alternative approaches to concurrency.

The 734 Abstract says:

This PEP proposes to add a new module, interpreters , to support inspecting, creating, and running code in multiple interpreters in the current process. This includes Interpreter objects that represent the underlying interpreters. The module will also provide a basic Queue class for communication between interpreters. Finally, we will add a new concurrent.futures.InterpreterPoolExecutor based on the interpreters module.

The Introduction continues:

Fundamentally, an “interpreter” is the collection of (essentially) all runtime state which Python threads must share. So, let’s first look at threads. Then we’ll circle back to interpreters.

The Motivation states:

The interpreters module will provide a high-level interface to the multiple interpreter functionality. The goal is to make the existing multiple-interpreters feature of CPython more easily accessible to Python code. This is particularly relevant now that CPython has a per-interpreter GIL (PEP 684) and people are more interested in using multiple interpreters.

(Given Python is going free-threaded (GIL-less), does that make this proposal less relevant?)

The Rationale says, in part:

That said, the proposed design incorporates lessons learned from existing use of subinterpreters by the community, from existing stdlib modules, and from other programming languages. It also factors in experience from using subinterpreters in the CPython test suite and using them in concurrency benchmarks.

From the multiple references to thread, concurrency, and GIL this seems to me to be about concurrency, so the Steering Council decision makes sense to me.

This user is definitely confused about what the module is about, if not concurrency. I see the paragraph explaining Interpreter Isolation but I have no idea why that would be important to me. I’d be interested in concrete. examples of when this would be useful.

1 Like
6

Are you putting this forward as your reaction/interpretation to the PEP, or are you simply mentioning this reaction to illustrate your point that because of how it’s written a reader without certain background might conclude that the module couldn’t have uses/benefits other than concurrent compute?
Should we really assume that’s the default reading of the PEP? It discounts several key parts, including the first couple excerpts you quoted describing the functionality exposed. Yes, it includes some mention of threads & how interpreters can also be used for concurrency, because that is one use case that can build on interpreters, and obviously one that rightly attracts a lot of attention because the software world cares about concurrency, but mentioning them doesn’t imply it’s the only use case. (Separately: by the same argument we should move threading and multiprocessing and possibly subprocess to the concurrent namespace too.)

As noted in the PEP thread, having separate import paths can allow using separate virtual environments that might even have conflicting versions. Also, it allows executing code/functions in a “sandbox” not in the security sense but rather a “fresh” or “empty slate” session, where globals and modules aren’t already full of whatever you’ve done in the current interpreter. This could be used for “playground” type applications that execute arbitrary code from somewhere that we’re not worried is intentionally crafted to mess things up.

6 Likes
7

You could perhaps get more satisfaction from asking in a Tcl forum. Here’s a start

Tcl has had subinterpreters since 1996 as a user-facing language feature, although they were supported at the C level earlier than that.

Guido picked Tcl up via using Tk for building IDLE (among other things), and was intrigued by the subinterpreter idea. Primitive support for them was added early in Python’s life via Python’s C API, but the considerable effort was never made to make them usable at the Python level. Incomplete, inconsistent, and buggy.

There’s no inherent connection with threads, A given thread can create any number of subinterpreters, but they’re restricted to running in the thread that created them.

If you can’t think of a good use case, then - ya! - you probably don’t need them :wink:

7 Likes
8

I agree that interpreters would be a better name, but not on the grounds that the module doesn’t have much to do with concurrency :slightly_smiling_face:

I still strongly believe in “flat is better than nested” and don’t see packages in the stdlib becoming popular. The only reason at the moment to use packages is to avoid naming conflicts - but that’s not a really a good reason at the language design level.

I still have the PEP on my TODO list to put the stdlib under a new std package. The name is already reserved on PyPI (not by me, but apparently by someone who already saw this coming).

Once we have that in place, the naming conflict argument no longer holds and we can happily continue with adding new modules to the stdlib in its flat name space. At that point, I’d also suggest to move futures back to std.futures and get rid off the concurrency package (well, modulo backwards compatibility shims).

PS: The discussion around whether interpreters should go into concurrency or not is a good example of how trying to associate modules with categories often collides with intuition and actual functional meaning. Very often you can make the case that modules fit into multiple such categories (or packages). A flat namespace helps avoid such ambiguities.

19 Likes
9

Also worth bearing in mind that PEPs have a tendency to write more text about side issues than the core issues. Typically the ideas and designs are very simple, and so most of the text goes into discussing the concerns that others have raised about the design, or the impact that the design could/will have on other aspects.

So when the bulk of the PEP text appears to be about one thing, it usually means it’s the thing that “the community” talked about, rather than the actual intention of the PEP (my favourite example is packaging PEP 708, which has basically a single sentence design[1] followed by many many words on how to disable it in specific contexts).

  1. “If two indexes provide the same package, refuse to install either” ↩︎

11 Likes
10

As release manager, I’m okay with renaming to interpreters if the steering council agree – it’s a simple enough change for any early adopters/beta testers.

As a core team member, I also prefer interpreters over concurrent.interpreters – it feels easier to use and explain. (On the other hand, having zstd and other modules such as gzip and zlib under compression does feel more natural.)

42 Likes
11

I don’t think arguing over the module location is meaningful anymore. It is bikeshedding and our choice will not be looked back upon by users as a weird or unusual or a problem in the long run.

We made a decision. My personal take: Stick to it and move on. This is not a core issue.

3 Likes
12

I wonder where you’re taking that confidence from.

Isn’t the discussion and feedback about the somewhat surprising choice at the last minute in the approval process proof enough that the SC should at least reconsider the decision ?

32 Likes
13

I’ve always found concurrent.futures a bit odd or out of place, even after years of using it. Rather than adding yet another module to concurrent and maintaining this odd state where some concurrency-related modules are outside the concurrent package while some are inside it, I’d prefer to see futures moved to the top-level namespace, especially since multiprocessing and threading are already there. However, I’m aware that deprecating the concurrent namespace would be a massive breaking change, so it should probably only be done if we ever get an std namespace.

3 Likes
14

There’s an old saying, “if they start bike shedding over the name you’ve already won.”

Let’s give up the bike shedding here, it’s a waste of time.

4 Likes
15

If this naming has such a strong consensus to be interpreters over concurrent.interpreters, as the most liked posts in this thread would indicate, why not directly open a PR to update to the PEP? Citing this discussion as it being generally preferred.

Then this is a specific proposal that can either be accepted or rejected, rather than an open bikeshedding discussion, and it can be addressed directly instead of part of the broader choice of accepting or rejecting the PEP as a whole.

Updating a PEP at this point has precedent even within the 3.14 release cycle (i.e. t-strings).

My understanding, reading through the governance documentation, is that the only other recourse would be to call for a vote of no confidence, which even for those strongly in favor of a name change seems overblown.

2 Likes
16

A point of order: only issues opened on the steering-council issue tracker automatically show up on our weekly agenda.

That seems difficult to find when searching for “std” on pypi.org. But in any case, I think @brettcannon might have registered it? The time to deeply think about stdlib reorg would be that time, but while it’s been suggested for a long time, we don’t have a concrete proposal yet. I hope you do @malemburg!

3 Likes
17

I could understand the saying if “you” rather than “they” meant the module author here.

1 Like
18

Exactly!!!

1 Like
19

Not me. The Warehouse folks have their own blocklist of names and I don’t think I specifically suggested this be on there (but maybe I did? Been doing this too long to know for sure :sweat_smile:).

1 Like
21

It’s most likely in the hidden list of reserved names - not really registered by anyone.

I’ll eventually get to writing something up… more likely later this year. Certainly not before or during summer. I am still in the collecting ideas and addressing issues phase.

8 Likes