Hi everyone! First of all, I am sorry if this is a wrong category to post my question to, but I figured, since my question is pretty technical and talks about the internals of the interpreter, core-dev would be a better fit. Please move it to another category if I was wrong in my assessment .
I am trying to do something that seems to go beyond the anticipated usage of the embedded interpreter, namely I am trying to run a Python computation within “fibers” (not unlike the
python-fibers library, except that I am doing this within an embedded interpreter, not as a Python package). This post serves two purposes: first, I would like to check that my understanding of how to do this safely is correct; and then I would like to see if it would be feasible to better support this (admittedly, rare) use-case in the future.
To avoid ambiguity, by “fibers” I mean that an arbitrary computation can be suspended (yield), have its state (CPU registers, etc.) stored in a data structure, and this computation can then be restored and resumed, potentially on a different OS thread. I think, the details of the implementation do not matter here, but, if it is important, we can assume that I am using
call/cc from Boost.Context.
This all revolves around
PyThreadState. Initially, I tried doing what the documentation suggests: keeping a
PyThreadState per OS thread; in this case, if a computation is moved to a different thread, it will resume with a different
PyThreadState. This actually seems to work fine, although I am very worried about some of the fields of the structure which seem to be part of the computation context (e.g.
context, just to give a few examples).
Next I tried the opposite approach: keeping a
PyThreadState per fiber and carrying it along when moving a fiber to a different thread. This seems to be a safer alternative, although there are some fields in
PyThreadState that seem to be thread-specific (e.g.
thread_id :). However, this does not work at all. I can manage (create, move, delete) the state myself, but I cannot update the state stored in thread-specific storage, which is actively used by
PyGILState_Ensure. If there is a mismatch between the
PyThreadState currently executing and the
PyThreadState stored in the current thread’s TSS, things break, and there doesn’t seem to be any way for me to update the TSS.
Based on the above, my current understanding is that
PyThreadState really contains two types of information: thread-specific and computation-specific. So, the approach I am considering is to manually slice
PyThreadState into these two pieces, and then use
PyGILState_Ensure to give each threads its own
PyThreadState, which will be used for its thread-specific state, while at the same time keeping computation-specific part of
PyThreadState next to each fiber and transplanting it into the current
PyThreadState when moving to a different thread. Incidentally, this is also the approach chosen by
python-fibers, however their selection of fields to move seems somewhat arbitrary to me. Hence, I would be glad to receive any advice on which fields I should treat as part of the computation state and transplant when moving fibers. Obviously, this depends on the specific version of cpython, so what I am looking for is general guidance.
Lastly, I am wondering if this kind of separation is something that could be maintained upstream? I feel like this would be useful to have and it is aligned with the general direction of async, coroutines, and PEP 567, while only requiring a fairly conservative change to the internals of the interpreter (and a bunch of new public APIs for managing it). What do you think?