Expected future for PEP 734?

[eric.snow]

@jsbueno

Just pinging in now this topic is active to remind people interested in playing roudn subinterpreters that my package “extrainterpreters” is live on pypi

FTR, I think this is completely awesome! I definitely encourage folks to try it out. I expect that people are going to figure out cool ways to use multiple interpreters that I never dreamed of.

@jsbueno

more on topic: while activelly developing it, I felt the pain of having no native mechanism to pass objects back and forth across interpreters - and I am ultimatelly still experimenting some approaches.

This is definitely where the most uncertainty is. I tried to come up with a bare minimum mechanism (PEP 734 queues and PEP 554 channels), but there are many possibilities to explore one we can open the floodgates.

@jsbueno

But changing the focus, having gone through the PEP as it is now, it is missing a Lock ,
I already make extensive use of memoryview, if only as a buffer where to pass pickle data around - but one of the first things I needed was a cross-interpreter lock.

(I had not fiddled with the code recently, so I am still to write a plain extrainterpreters.Lock Python class, but there is one there which I use to lock structs and messages)

I purposefully did not add a cross-interpreter lock type. My expectation has been that the concurrency model that multiple interpreters naturally support will make the most sense if synchronization is more closely tied to the communication-focused data types (queues and channels). In theory they should be sufficient and flow better.

That said, we currently don’t have nearly enough experience with this concurrency model in Python to have a good sense of what primitives will be best. I’m definitely open to the many possibilities. However, I was trying to be cautious about starting out with primitives that are familiar from other concurrency models, thus why I was avoiding a dedicated per-interpreter lock type.

@jsbueno

One other idea I have in mind (though that is definitely something for this PEP) is a way to “borrow” objects to other interpreters - transferring them into a container that first asserts it owns the sole reference to the object, and then make that object itself available in other interpreter.
I had not gone deep into Rust but I suppose it is the same thing it does with its borrowing mechanism. I’ve tested having objects from one interpreter being simply used in another, and everything goes fine as long as none of the two interpreters tries to de-allocate the object by itself. (Of course it has to take into account composite objects - all attributes need also to be locked into the safe container for as long as the object is alive in a subinterpreter)

Yeah, this would be an absolutely interesting idea to explore. See PEP 554 for many other similar ideas.

[a-reich]

This is awesome, thanks ! I’m going to play around with this soon.

Separately, as this just came up in the other thread, do you have any thoughts about whether extension packages will start supporting subinterpreters anytime soon? Or what would be needed to help with that?

[ncoghlan (Alyssa Coghlan)]

@a-reich

Separately, as this just came up in the other thread, do you have any thoughts about whether extension packages will start supporting subinterpreters anytime soon? Or what would be needed to help with that?

The key bit of complexity is migrating static globals to per-interpreter state without introducing significant performance regressions. Extension modules that expose C APIs (either directly or via capsules) also face additional challenges.

Beyond the module-specific issues, I haven’t kept track of whether or not Cython, SWIG, pybind11, etc support emitting modules that use multi-phase init with full subinterpreter support.

[pitrou (Antoine Pitrou)]

@ncoghlan

Beyond the module-specific issues, I haven’t kept track of whether or not Cython, SWIG, pybind11, etc support emitting modules that use multi-phase init with full subinterpreter support.

Well… looking at the C++ source code generated by Cython for a sample project, it seems that multi-phase init and module state are mutually exclusive (either CYTHON_PEP489_MULTI_PHASE_INIT or CYTHON_USE_MODULE_STATE can be enabled, not both).

So you get things like:

#if CYTHON_PEP489_MULTI_PHASE_INIT
static PyObject* __pyx_pymod_create(PyObject *spec, PyModuleDef *def); /*proto*/
static int __pyx_pymod_exec_lib(PyObject* module); /*proto*/
static PyModuleDef_Slot __pyx_moduledef_slots[] = {
  {Py_mod_create, (void*)__pyx_pymod_create},
  {Py_mod_exec, (void*)__pyx_pymod_exec_lib},
  {0, NULL}
};
#endif

#ifdef __cplusplus
namespace {
  struct PyModuleDef __pyx_moduledef =
  #else
  static struct PyModuleDef __pyx_moduledef =
  #endif
  {
      PyModuleDef_HEAD_INIT,
      "lib",
      0, /* m_doc */
    #if CYTHON_PEP489_MULTI_PHASE_INIT
      0, /* m_size */
    #elif CYTHON_USE_MODULE_STATE
      sizeof(__pyx_mstate), /* m_size */
    #else
      -1, /* m_size */
    #endif
      __pyx_methods /* m_methods */,
    #if CYTHON_PEP489_MULTI_PHASE_INIT
      __pyx_moduledef_slots, /* m_slots */
    #else
      NULL, /* m_reload */
    #endif
    #if CYTHON_USE_MODULE_STATE
      __pyx_m_traverse, /* m_traverse */
      __pyx_m_clear, /* m_clear */
      NULL /* m_free */
    #else
      NULL, /* m_traverse */
      NULL, /* m_clear */
      NULL /* m_free */
    #endif
  };
  #ifdef __cplusplus
} /* anonymous namespace */
#endif
#endif

I frankly don’t understand what the implications are, as the module initialization changes are always confusing to me. Perhaps a Cython developer (@scoder, @da-woods) is available to comment.

[a-reich]

Cython deliberately errors if a loaded module is imported in a different interpreter.
AFAICT PyO3 does the same.