Make some PyMutex APIs public

I am proposing making some PyMutex APIs public, as part of the general non-limited API (i.e., in Include/cpython). I’d appreciate feedback on this proposal:

3 Likes

Will the use of these APIs be only valid if python is built in free threading mode?

The intention is that they are available in both the default and free-threaded build of CPython.

1 Like

tl;dr I’m in favor, with some questions:

  • what specific types/functions would we make public right now (i.e. for 3.13)?
  • are we happy with the internal API design enough to expose it as-is? (me: yes)
  • should we make any decisions about the existing PyThread_type_thread API? (me: there’s no hurry)

(Disclosure: I reviewed the PR[1] that added PyMutex to the internal API and have reviewed quite a few of the subsequent related PRs.)

You have a big +1 from me. It seems like a nice improvement over PyThread_type_lock[2] in several meaningful ways. Personally, I’ve found PyMutex to be nice to use and appreciate that it’s faster and uses less memory. (FTR, PyMutex is based on the “new” (2016) locking design in WebKit. [3]) Furthermore, I really like that, unlike PyThread_type_lock, you can statically initialize global PyMutex variables. That’s important because, with PyThread_type_lock, you have to deal with the possibility of a race when initializing the variable ` at run-time.

One question: what about scope? It would help to have a list of which functions would be made public in 3.13. Would we expose only PyMutex_*() (or even only some of those), i.e. everything in Include/internal/pycore_lock.h, or would we also promote other related stuff, like critical sections? What about PyRawMutex, PyOnceFlag, PyRWMutex, PyEvent, or PySemaphore? What about PyParkingLot_*()? I can think of how those related APIs could be useful to users, though we’d probably want to assess each separately. There’s certainly no need to expose all the related internal APIs all at once (or even ever). In the end, it would help to have a list of the type(s) and functions you want to make public.

Another thing: what about API design? Notably, the PyMutex API has a slightly different shape to it than the PyThread_type_thread API, including with slightly different names for the roughly equivalent functions. Are the names/signatures we’re using internally the ones we want to set in stone when we expose them publicly? Personally, I think the existing internal names and signatures are fine. Plus, having them be distinct from the older API might actually be helpful for people reading code (i.e. the value of “visual diff”).

Finally, how might the PyMutex and PyThread_type_lock APIs relate to each other? Should we think of it as a one-to-one match in functionality? If we wanted to, could we implement the PyThread_type_lock API as a wrapper around the PyMutex API or by using PyMutex instead of the raw OS primitives? FWIW, it’s probably way too early to talk much about deprecating PyThread_type_lock in favor of PyMutex. The two types coexist fine, plus I’m pretty sure there are use cases where PyThread_type_lock is still the better option. I don’t think any decision to make PyMutex public should depend on any decision about PyThread_type_lock.

(FTR, we’ve changed a number of internal locks[4] in the runtime state from PyThread_type_lock to PyMutex (partly for the sake of nogil), and I’m not aware of any disruption. Clearly there’s opportunity to switch other locks as well (but there’s no urgency). Perhaps most interesting would be updating threading.Lock to use PyMutex. However, none of our internal usage really has bearing on making PyMutex public; mostly its a non-trivial data point toward the conclusion that its API isn’t bad. :smile:)


  1. gh-109344 (issue: gh-108724) ↩︎

  2. (our platform-independent wrapper around the OS-specific mutex APIs) ↩︎

  3. The WebKit team explained their design in a great blog post: Locking in WebKit | WebKit ↩︎

  4. For some key examples see: Pull requests · python/cpython · GitHub ↩︎

3 Likes

One question: what about scope?

I think this is more clear in the issue. For now, it’s just PyMutex_Lock(), PyMutex_Unlock(), and the type PyMutex. The “slow-path” out-of-line functions [1] need to be in the header, but I don’t think they should be considered part of the public API.

We can consider additional PyMutex functions or additional types later. I’m not sure we’ll ever want to make everything in pycore_lock.h public.

are we happy with the internal API design enough to expose it as-is?

me too!

should we make any decisions about the existing PyThread_type_thread API?

I also agree that there’s no hurry.


  1. _PyMutex_LockSlow and _PyMutex_UnlockSlow ↩︎

2 Likes