Expected future for PEP 734?

Core Development
1

I’ve recently taken an interest in making sure the packages I maintain are compatible with sub-interpreters, and per-interpreter GIL for that matter.

One of the drivers for making this effort was of course seeing PEP 734 being mentioned more actively on the internet during the last year. I’ve never used subinterpreters myself though, given how they are only currently available via the C API, but if/when the PEP is implemented, I can see how more people might jump into using subinterpreters, and hence start requesting support from popular packages.

So I’ve gone ahead and done my work in my packages to make them subinterpreter-friendly. Or at least in theory, since I had no tests. And because I don’t want to write a C extension just to create and use a subinterpreter, I’ve been trying to use the WIP interpreters module in my tests, experimental as it may be. I jumped through a few hoops:

  • I first tried the interpreters_pep_734 package directly, but quickly realised that it hasn’t been maintained: the README file mentions that most development is being done in upstream CPython, the last commit was many months ago, and the package doesn’t work correctly against 3.13.0rc*.
  • I then thought I would try to immitate what interpreters_pep_734 was doing by using _interpreters myself directly. After some battling I made it work, although with some funny edge cases I didn’t fully understand at the time.
  • Then I realised that the “upstream CPython” bit apparently means the test.support.interpreters module in CPython, which I’m now using, and works correctly.
  • At some point I was using channels instead of queues, until I realised the former seems like a leftover from the former PEP 554, they are never mentioned in 734.

While I’ve read the PEP, and at least what seems the most important associated public discussion in this forum, I’m still a bit unclear about the expected future of the PEP. On the one hand the fact that’s been brewed within CPython seems encouraging, but OTOH the fact the PEP is deferred and the lack of further public discussion since the first half of the year doesn’t bid well.

In summary, I have the following questions:

  • Is PEP 734 expected to be eventually implemented? If so, what’s the --very rough-- plan?
  • If not in the stdlib, would it be offered as an external package? In the discussions I saw how there were people both in favour and against having it in the stdlib, hopefully the former will prevail.
  • Why are channels present in the test.support.interpreters.channels and _interpchannels if PEP 734 doesn’t consider them?
2 Likes
2

Maybe use Topic category “Core Development” instead?

3

I took the liberty of moving the thread to Core Development.

1 Like
4

Many thanks @guido for the idea, @MegaIng just beat me to it, thank you very much!

5

In the absence of a more authoritative answer from @eric.snow (I’m not sure how often he checks the discussion forum for new topics of interest):

  • having the Go-inspired channels API from PEP 554 in the test suite doesn’t represent a commitment to ever stabilise them as a supported API. The Queue abstraction is more familiar to users of the threading and multiprocessing modules, so it’s a better candidate for a future public API. (I assume the channels API in the test suite is there because some tests are still using it, but if not, it will likely make sense to drop it entirely).

  • as for when the PEP status switch will be flipped from Deferred back to Draft, that will be up to Eric as the PEP author based on his view of whether all the open questions have been resolved to his satisfaction (the main one I’m aware of that hasn’t been fully resolved is deciding exactly how the Queue objects should behave when the interpreter owning a queue entry goes away), and that he has the time available to champion the PEP through the review process.

In the meantime, the ongoing work on subinterpreters has been aiding the projects that already use them via the C API, so there’s no risk of the underlying capability unexpectedly going away.

You may also find @yselivanov’s MemHive project (and the associated PyCon talk) of interest: MemHive: sharing immutable data between Python subinterpreters [LWN.net]

3 Likes
6

Up until today, I was too. :slight_smile: I had a productive discussion with the Steering Council a few hours ago. The path forward is substantially more clear now.

I’m going to un-defer it soon. The lack of discussion is mostly because there wasn’t much left to discuss. :slight_smile: In one of the last posts, the Steering Council set some expectations relative to trying a PyPI package first and we’ve been sorting out the logistics.

Yeah, sorry about that. I got stuck on some other work and was planning on getting back to it soon. I’m glad you found test.support.interpreters, but I don’t want people to have to resort to that.

The plan is to have a PyPI package for 3.13+ (and maybe 3.12) that’s hosted on a cpython PyPI org. We’ll assess where things stand early next year with the goal of deciding about inclusion in 3.14. I’m hopeful, and am planning accordingly, but it isn’t guaranteed.

In the meantime, I’m planning on getting interpreters_pep_734 working soon. Again, apologies for the lapse.

The goal is to have it in the stdlib for 3.14. Either way, I’m going to make sure interpreters_pep_734 on PyPI works in the near term and we’ll likely have a separate package there soon under the future cpython PyPI org.

They aren’t in the PEP because I wanted to start as minimal as possible and queues are a simpler, existing Python data structure. I hope to add channels to the stdlib package eventually (assuming the PEP is accepted). Regardless, channels will be available through the PyPI package.

6 Likes
7

CC @steering-council

8

Just pinging in now this topic is active to remind people interested in playing roudn subinterpreters that my package “extrainterpreters” is live on pypi - and have parity with threading.Thread in the sense one can call a function to be run in another interpreter (rather than just passing it a string).

more on topic: while activelly developing it, I felt the pain of having no native mechanism to pass objects back and forth across interpreters - and I am ultimatelly still experimenting some approaches.

2 Likes
9

How does that work? Do you just pickle the function (and hence restricting it to a global, as nested functions or lambdas cannot be pickled)?

1 Like
10

I reimport the module where the function lives, in the other interpreter. With some inspect.get_source to work when one experimenting in the REPL.

For most parameters, and other objects, I use pickle.

11

The two go-to solutions for extended pickling support are cloudpickle and dill. For example, cloudpickle is able to recreate nested functions and even generators (AFAIR) by serializing the code object’s bytecode, among other things.

They would be a natural choice for sending data between subinterpreters.

2 Likes
12

Re-importing the module is good enough for “ordinary” functions - special pickling can be nice for things like lambdas .

But changing the focus, having gone through the PEP as it is now, it is missing a Lock ,
I already make extensive use of memoryview, if only as a buffer where to pass pickle data around - but one of the first things I needed was a cross-interpreter lock.

(I had not fiddled with the code recently, so I am still to write a plain extrainterpreters.Lock Python class, but there is one there which I use to lock structs and messages)

13

Presumably the current design tries to gently encourage you to use queues instead of locks. Can you explain why you can’t use those for your use cases?

If you really do need a cross-interpreter lock, presumably using a POSIX lock could work. Presumably you could put that in a memoryview buffer (IIRC its size is around 40 bytes) and then pass the memoryview object to other subinterpreters using a queue.

Possibly in 3.13+ you might be able to use the single-byte parking lot lock from the free-threading API, though I don’t know if the hash table that backs those locks is per-interpreter or truly global. (In fact, I have no idea if free-threading and subinterpreters can even be combined – @eric.snow?)

1 Like
14

Presumably the current design tries to gently encourage you to use queues instead of locks. Can you explain why you can’t use those for your use cases?

Well - I need a lock to… implement a queue. (since the Queue in the PEP is not in place yet)
Actually, there is a queue n this project, using locks and selectors - but I blundered it: it is not quite working as of yet (but the lock itself is, and there is another “postbox” data structure that is working perfectly - using the lock)

If you really do need a cross-interpreter lock, presumably using a POSIX lock could work. Presumably you could put that in a memoryview buffer (IIRC its size is around 40 bytes) and then pass the memoryview object to other subinterpreters using a queue.

I could check that Posix lock - I am currently using an “atomic_char” from ISO C11 so I can build a lock using single byte in a memoryview. - Maybe this approach can be practical enough for use in the sdtlib itself. - extrainterpreters/src/extrainterpreters/memoryboard.c at ce004a7b6ec0614509dad58b5acfe914ac6eb676 · jsbueno/extrainterpreters · GitHub

It is worth noting that I am using the strategy of passing a memory-view buffer address across interpreters, and rebuilding the another memoryview with the same buffer in the other interpreters: exactly the same approach described in PEP 734, and I got to it independently -
I guess this helps validating the idea

Possibly in 3.13+ you might be able to use the single-byte parking lot lock from the free-threading API, though I don’t know if the hash table that backs those locks is per-interpreter or truly global. (In fact, I have no idea if free-threading and subinterpreters can even be combined – @eric.snow?)

Well, my current tests will run with GIL disabled - although I have nothing in place trying
to use anything specific thing from free threading.
IIRC all tests currently use a single thread per interpreter)

(env313t) [jsbueno@fedora extrainterpreters]$ py.test -v
=========================================== test session starts ===========================================
platform linux -- Python 3.13.0rc1+
.
.
.
tests/test_struct.py::test_struct_doublefield PASSED                                                [100%]

===================================== 54 passed,
15

Thank you all very much for your responses, specially to @eric.snow for all the many clarifications regarding the status of the PEP. And in general, for championing this PEP throughout the years, I’m sure it’s no easy work.

Now that I’ve dived more deeply into subinterpreter territory I’m starting to better understand the discussions around object “sharing” (here in this discussion and in other places), and I can see how that’s kind of a new phenomenon that has to be approached in an intelligent way.

That’s great news! It seems like I timed my question perfectly then :slight_smile:

Thanks for clarifying that particular point too. I’ll probably stick to Queues for my small tests, but it’s good to know that channels will also be available.

That’s great! I’ll keep an eye on it and will try it out once it’s available.

Thank you very much for that reference. I am an LWN reader, but I missed this – probably because of my lack of subinterpreter usage/interest until now. I’ll give it a read, just skimming through it gave the feeling of being very interesting and useful.

I might be completely off my mark here, but: I’m using a Queue already from CPython’s test.support.intepreters.queues, which is basically what’s in PEP 734. Are you referring to that Queue or something else? It would seem to me like “the Queue in the PEP” is already there.

Like @jsbueno, I have also run subinterpreters in 3.13t, including the creation of a threading.Thread for the subinterpreter execution (via Interpreter.call_in_thread). This might be highly circumstantial though, but at least goes to show that there’s nothing baked in to stop users trying it.

1 Like
16

Hah - nice. I had not seem that one.

As it is my project had been stalled since more or less the release of Python 3.12, due to thigns like life and PEP 703 (which was a cold blow in the cool side of working in global-lock-less Python code with multiple subinterpreters)

I’ve been organizing the project now.
So I think I will likely ditch my complicated queue in favor of this one (and likely add a wrapper to allow for arbitrary objects, which will go through serialized)

The cross interpreter lock anyway is needed if one is to use the transfereable memoryviews to share data.

One other idea I have in mind (though that is definitely something for this PEP) is a way to “borrow” objects to other interpreters - transferring them into a container that first asserts it owns the sole reference to the object, and then make that object itself available in other interpreter.
I had not gone deep into Rust but I suppose it is the same thing it does with its borrowing mechanism. I’ve tested having objects from one interpreter being simply used in another, and everything goes fine as long as none of the two interpreters tries to de-allocate the object by itself. (Of course it has to take into account composite objects - all attributes need also to be locked into the safe container for as long as the object is alive in a subinterpreter)

17

Sorry for this question, but can that be imported at runtime by a 3rd party package? Or should I resort to copy-pasting that queue code?

18

We don’t consider the test and test.support packages to be part of the standard library, they do not have API stability guarantees and will usually not be available in most Python installations. Vendoring a copy of any parts from there that you want within your own code makes sense if you find a need for something from there.

But is there something subinterpreter related that interpreters-pep-734 · PyPI does not provide? IIUC, that is the code from test.support.interpreters as packaged for use by @eric.snow. Perhaps in need of another update and maybe some automation to keep it in sync?.

2 Likes
19

I can successfully import test.support.interpreters, but from what Gregory says it’s purely anecdotal that it works, and I’m certainly not betting on it. From the get go I’ve know my approach is experimental and not required to work, but I’m only driving some tests with it, so I skip them if I can’t find the module.

FWIW I’ve tried in Ubuntu 24.04 (via the deadsnakes PPA), in GitHub Actions (via the actions/setup-python action) and in cibuildwheel, where it works in most containers (all expect windows apparently?).

Yes, the PyPI package is not fully usable ATM, I think it’s gotten slightly out of sync with upstream CPython. Eric has already responded that he’ll take care of bringing it up to speed soon, specially after the latest discussions he’s had with the SC.

2 Likes
20

FYI, I’ve updated the interpreters_pep_734 package. It should work correctly on 3.13 (and 3.14) now. I’m planning on keeping it sync’ed with the CPython main branch. In the near future I should have it working with 3.12 as well.

On top of that, I plan on adding the InterpreterPoolExecutor implementation from gh-124548. I may also look at adding other experimental data types (but not add them to the PEP).

5 Likes