"interpreters" vs. "concurrent.interpreters"

Core Development
1

Hi all (and especially the Steering Council and our 3.14 RM),

The new module from PEP 734 is happily part of 3.14 now, the culmination of the primary phase of a project I started over a decade ago. I’m excited at the possibilities that are now open to our users. Thanks to all who have helped!

Now on to the bikeshed! I added the module as concurrent.interpreters per the instructions of the Steering Council. If we hadn’t been up against the 3.14 deadlines, I would have focused on making my case against the name change. However, I’m still concerned that concurrent.interpreters isn’t the right place for the module and I still hope there’s a chance to rename it to just interpreters.

Here’s why:

  • interpreters says exactly what it’s about
  • it doesn’t feel (to me) like the right place for the module (isn’t where I’d expect to find it)
  • the module is about exposing a core runtime feature, like sys, gc, atexit, or imp/importlib, so top-level makes the most sense
  • the module is only vaguely concurrency related, in the same way exec() or subprocess is related to concurrency
  • there’s a strong likelihood that users will be confused about what the module is for, because of “concurrent”
  • concurrent.interpreters vs. concurrent.futures.interpreter is confusing
  • it’s a longer name than interpreters
  • I’ve put a lot of thought, time, and effort into the module and the underlying runtime functionality, and with that context concurrent.interpreters makes no sense to me
  • the proposed module name has been interpreters since before 2018, with consensus and no suggestions to do anything else until literally the last minute, in the Steering Council’s acceptance message

Overall, the justification for the change to concurrent.interpreters (counter to the author’s intention and to the long-standing consensus) is unclear.

I’ve heard the following arguments in favor of concurrent.interpreters:

  • we should make better use of the concurrent namespace (agreed, but not with PEP 734)
  • discoverability (I’m still unclear on how concurrent.* helps with that here)
  • if you squint just right, it’s sort of concurrency-related

(I don’t mean to misrepresent the opposing position to my own; I literally don’t know more than this about the arguments in favor of concurrent.interpreters.)

Here are some public comments core devs have made both ways about the module name:

and some relevant comments from others:

Ultimately, I’d like to change the name back to interpreters (for 3.14+).

I realize it’s awfully late in the 3.14 cycle to be making this change. I’ll understand if we’re already past the point of no return. That said, I’ll also note that changing the module name before beta4/rc1 would be unlikely to introduce any instability nor be disruptive.

Thoughts?

p.s. Sorry I didn’t write this sooner. It’s been hard for me to process the situation (and I have a lot going on) and I’m only now comfortable with writing this down.

26 Likes
2

FWIW, I completely agree we should make better use of the concurrent namespace. It’s the perfect place for tools related to concurrency in general. That would include helpers for using multiple interpreters for concurrency specifically. (However, the PEP 734 module has never been that.)

For example, I’m working on a PEP for 3.15 related to cross-interpreter data and at least some of that capability will be general to safely sharing data between abstract threads of execution, and a module for that would be a great fit under concurrent.

2 Likes
3

FYI, I started a separate, related thread about some of my frustrations with how the decision-making has played out with PEP 734.

1 Like
4

IMO, we already have some significant difficulties with teaching subinterpreters. I’m sure that there are plenty Python developers who don’t understand the concept of subinterpreters clearly. For example, I’ve obtained a proper view on this topic in just last few weeks. And literally yesterday I’d answered some questions about subinterpreters and free-threading concepts at our local Python community.

It’s hard to explain even for advanced Python user, why we need subinterpreters if there will be some free-threading builds soon. Of course, we need good docs about it, but concurrent.interpreters import name will definitely hardens these explanations, not simplifies them.

9 Likes
5

PEP-1 describes a process for PEPs that include ideation, endorsement by a core developer, and disucssion by all interested parties. I wonder if perhaps the proposal was written with more of a core developer audience in mind, resulting in an explanation that’s more “tree-focused” than outlining the vision of the forest.

The PEP 734 says it’s a continuation of PEP 554. 554 says,

CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. [c-api] Multiple interpreters operate in relative isolation from one another, which facilitates novel alternative approaches to concurrency.

The 734 Abstract says:

This PEP proposes to add a new module, interpreters , to support inspecting, creating, and running code in multiple interpreters in the current process. This includes Interpreter objects that represent the underlying interpreters. The module will also provide a basic Queue class for communication between interpreters. Finally, we will add a new concurrent.futures.InterpreterPoolExecutor based on the interpreters module.

The Introduction continues:

Fundamentally, an “interpreter” is the collection of (essentially) all runtime state which Python threads must share. So, let’s first look at threads. Then we’ll circle back to interpreters.

The Motivation states:

The interpreters module will provide a high-level interface to the multiple interpreter functionality. The goal is to make the existing multiple-interpreters feature of CPython more easily accessible to Python code. This is particularly relevant now that CPython has a per-interpreter GIL (PEP 684) and people are more interested in using multiple interpreters.

(Given Python is going free-threaded (GIL-less), does that make this proposal less relevant?)

The Rationale says, in part:

That said, the proposed design incorporates lessons learned from existing use of subinterpreters by the community, from existing stdlib modules, and from other programming languages. It also factors in experience from using subinterpreters in the CPython test suite and using them in concurrency benchmarks.

From the multiple references to thread, concurrency, and GIL this seems to me to be about concurrency, so the Steering Council decision makes sense to me.

This user is definitely confused about what the module is about, if not concurrency. I see the paragraph explaining Interpreter Isolation but I have no idea why that would be important to me. I’d be interested in concrete. examples of when this would be useful.

1 Like
6

Are you putting this forward as your reaction/interpretation to the PEP, or are you simply mentioning this reaction to illustrate your point that because of how it’s written a reader without certain background might conclude that the module couldn’t have uses/benefits other than concurrent compute?
Should we really assume that’s the default reading of the PEP? It discounts several key parts, including the first couple excerpts you quoted describing the functionality exposed. Yes, it includes some mention of threads & how interpreters can also be used for concurrency, because that is one use case that can build on interpreters, and obviously one that rightly attracts a lot of attention because the software world cares about concurrency, but mentioning them doesn’t imply it’s the only use case. (Separately: by the same argument we should move threading and multiprocessing and possibly subprocess to the concurrent namespace too.)

As noted in the PEP thread, having separate import paths can allow using separate virtual environments that might even have conflicting versions. Also, it allows executing code/functions in a “sandbox” not in the security sense but rather a “fresh” or “empty slate” session, where globals and modules aren’t already full of whatever you’ve done in the current interpreter. This could be used for “playground” type applications that execute arbitrary code from somewhere that we’re not worried is intentionally crafted to mess things up.

5 Likes
7

You could perhaps get more satisfaction from asking in a Tcl forum. Here’s a start

Tcl has had subinterpreters since 1996 as a user-facing language feature, although they were supported at the C level earlier than that.

Guido picked Tcl up via using Tk for building IDLE (among other things), and was intrigued by the subinterpreter idea. Primitive support for them was added early in Python’s life via Python’s C API, but the considerable effort was never made to make them usable at the Python level. Incomplete, inconsistent, and buggy.

There’s no inherent connection with threads, A given thread can create any number of subinterpreters, but they’re restricted to running in the thread that created them.

If you can’t think of a good use case, then - ya! - you probably don’t need them :wink:

5 Likes
8

I agree that interpreters would be a better name, but not on the grounds that the module doesn’t have much to do with concurrency :slightly_smiling_face:

I still strongly believe in “flat is better than nested” and don’t see packages in the stdlib becoming popular. The only reason at the moment to use packages is to avoid naming conflicts - but that’s not a really a good reason at the language design level.

I still have the PEP on my TODO list to put the stdlib under a new std package. The name is already reserved on PyPI (not by me, but apparently by someone who already saw this coming).

Once we have that in place, the naming conflict argument no longer holds and we can happily continue with adding new modules to the stdlib in its flat name space. At that point, I’d also suggest to move futures back to std.futures and get rid off the concurrency package (well, modulo backwards compatibility shims).

PS: The discussion around whether interpreters should go into concurrency or not is a good example of how trying to associate modules with categories often collides with intuition and actual functional meaning. Very often you can make the case that modules fit into multiple such categories (or packages). A flat namespace helps avoid such ambiguities.

11 Likes
9

Also worth bearing in mind that PEPs have a tendency to write more text about side issues than the core issues. Typically the ideas and designs are very simple, and so most of the text goes into discussing the concerns that others have raised about the design, or the impact that the design could/will have on other aspects.

So when the bulk of the PEP text appears to be about one thing, it usually means it’s the thing that “the community” talked about, rather than the actual intention of the PEP (my favourite example is packaging PEP 708, which has basically a single sentence design[1] followed by many many words on how to disable it in specific contexts).

  1. “If two indexes provide the same package, refuse to install either” ↩︎

5 Likes