Thread-safety and freethreading Limited API

Core Development
,
1

For the purposes of this question, I’m assuming that the plan is still to enable Limited API support on the freethreading build with back-dated support (i.e. it’ll be possible to build a module that’s compatible with Py_LIMITED_API=0x03080000 and 3.14t, but you may not to recompile).

Do we have a feel for how people are expected to handle thread-safety?

A few examples:

  • In recent builds we’d ensure that accesses to a list has thread-safe reference counting by PyList_GetItemRef. Obviously that doesn’t exit for earlier versions of Limited API. Should we be preferring PyObject_GetItem or PySequence_GetItem when writing Limited API extensions now in anticipation of free-threaded Limited API support?
  • How should we be handling things like critical sections to guard the internals of extension types?
  • Is there a way of old Limited API extension modules to set the Py_mod_gil slot? Or is the assumption that they won’t and that this is how we ensure thread safety?

I realise a lot of this is currently unanswerable, and will end up defined by community expectations. So I’m just trying to work out if there are any thoughts about where this is going?

3 Likes
2

I am not the expert here, but my assumption is that you could write a compat section/library like this:

#if MY_FREETHREADING_BUILD
#define MY_MOD_GIL_SLOT_ENTRY {Py_mod_gil, Py_MOD_GIL_NOT_USED},
#else
#define MY_MOD_GIL_SLOT_ENTRY /* nothing */
#define PyList_GetItemRef PySequence_GetItem  /* needed for thread safety */
#define Py_BEGIN_CRITICAL_SECTION {
#define Py_END_CRITICAL_SECTION }
#define Py_LIMITED_API 0x03080000
#endif

and then compile separately for 3.14+ (both free-threading and not) and abi3 (3.8+ but only non-free-threading builds).

3

Won’t we need something like an “abi4” or “abi3t” with the free-threaded PyObject layout to enable cross-version ABI support? I imagine all this would need to be worked out before wheels could be built with free-threaded limited API support.

1 Like
4

I think there was a “cunning plan” which involved compiling against Py 3.14 headers but with Py_LIMITED_API set to less. It’s summarised here `--disable-gil` builds are not compatible with the limited API · Issue #111506 · python/cpython · GitHub.

I’m asking the question on the assumption that this is true, and it will be possible to build modules that target Python 3.7+ and free-threading builds.

If that isn’t currently the plan then the question is kind of irrelevant :slight_smile:

3 Likes
5

If we end up going with something like https://github.com/python/cpython/issues/111506, with dynamic lookup / weak symbols (and compiling against 3.14 but targeting older versions of Python), then we could provide a PyList_GetItemRef and similar that uses PyList_GetItem on older versions of Python.

For Py_mod_gil, I think you could use Py_Version to dynamically trim off the new slots in older versions of Python before you call PyModuleDef_Init().

5 Likes
6

Thanks Sam. Sounds like there’s nothing specific to do now, but it’ll hopefully be solvable later.

1 Like