As _interpreters and PEP 734 get closer to fruition, it would be good to start getting some public C APIs down for subinterpreters to see what people are doing with them in the wild. We already have a large set of private APIs under the _PyXI namespace, so let’s try and make some of that public.
Right now, I only want to address interpreter switching. A few issues and PRs (namely #123134, #123488, and #123728) have shown that there’s a bit of trouble and ambiguity around switching between interpreters on the fly. The documentation lacks examples, doing it with PyThreadState_Swap is pretty messy, and PyGILState_Ensure is flat out unsupported if subinterpreters are active. IMO, this is less than ideal for the public C API.
I propose to add public variations of _PyXI_Enter and _PyXI_Exit, with an opaque PyXI_Session structure alongside them. So, something like:
static PyObject *
my_func(PyObject *self, PyObject *args)
{
PyInterpreterState *interp = /* interpreter to switch to */;
PyXI_Session session;
if (PyXI_Enter(&session, interp) < 0)
{
return NULL;
}
// Do something in the interpreter
/* ... */
PyXI_Exit(&session);
Py_RETURN_NONE;
}
(I’ve intentionally left out the nsupdates parameter that’s present in the private API–that seems like it’s probably subject to change, and may be confusing for any users that aren’t calling Python code.)
There’s still a lot that’s undecided with subinterpreters internally right now, so there’s a limit to what we can expose, but this is pretty universal (as in, it probably won’t need to change) and a good way to gauge community feedback regardless.
This sounds like a good idea, but the devil is in the details.
What should remain a private detail and what should be exposed/guaranteed publicly?
How should error signalling/handling work?
How are the lifetimes of the interpreters and thread states managed? What does the user need no ensure? Where should we add some safeguards?
Most of those questions Eric has already solved himself with the development of _interpreters, the idea here is to just be a thin wrapper on top of his existing work. Hopefully, for just interpreter switching at least, this should need very little maintenance on top of what already exists for the _interpreters module.
In this case, basically nothing, other than “PyXI_Enter is better than PyThreadState_Swap or PyGILState_Ensure”
Signals are a little difficult, I think the situation right now is that subinterpreters ignore signal handling, and only the main interpreter cares about them. But that means, if the main interpreter isn’t doing anything (such as when it’s busy executing something in another interpreter), then it will just wait until it has its GIL back before throwing any error.
The thread states, and therefore the interpreter state, are managed by PyXI, and I think that’s a good thing. From what I’ve seen with users, they don’t like messing with thread states. They’ll still need to touch thread states a little bit because of Py_NewInterpreterFromConfig, but at least switching to that interpreter again will be much less painful.
Are there any stragglers that are opposed? If not, I’ll bring this to GitHub (either the C API WG first or just straight to a CPython issue, depending on what needs to be discussed).
I think it’s time to write out the docs, with examples and notes like “interp must be valid”, “you must hold the GIL”, or “if PyXI_Enter fails, the current interpreter is unchanged”, so we can agree which parts of _PyXI work by design, and which are undecided or accidental.
I would like to keep these APIs private for now. The reason being that subinterpreter-project is still very much in flux atm, we don’t have the python module ready in stdlib for users. I would rather focus of making that happen sooner, also if we add this to public APIs we have to get this exactly right otherwise the backwards compatibility won’t let us evolve this as the rest of the C API always does.
Not totally disagreeing, but we already have interpreter switching in the public C API–you just have to jump through silly hoops with PyThreadState_Swap to do it (in fact, that’s documented). My proposal is to make this effortless, which will help early birds find bugs in subinterpreters before it’s too late to do huge refactors.
If there’s opposition to putting it under the PyXI namespace for the fear that we won’t have anything else in in it ready by the feature freeze, then we could put it under the “generic” namespace (think something like Py_SwitchInterpreter).
