I have a couple of questions about the design of the new PyContext_AddWatcher API, new in CPython v3.14a1:
The registered callbacks are stored in PyInterpreterState, but the “current context” is per-thread. When there is a context switch, there’s no way to tell which thread was affected. Why isn’t this a per-thread registration? Or at least pass a PyThreadState pointer to the callback?
It is idiomatic in C for a callback registration function to also take an opaque void * that will be passed back to the callback. This makes it possible to reuse the same code for multiple simultaneous registrations. (This would be particularly useful if the registrations were per-thread instead of per-interpreter.) Is there a reason it was omitted?
Good point. An earlier version of a change I proposed would have made this not always true, but it’s an important property to preserve so I revised the PR.
Ah, right. Stashing callback state on the context (or even thread-local storage) should be suitably expressive for all use cases, as long as the callback is always called from the affected thread.
Thinking about this more, getting the watcher config/state from the context can be problematic in a couple of ways:
The config/state is not guaranteed to be available in the context. For example, the code can do ctx = contextvars.Context() instead of ctx = contextvars.copy_context(). Or the code can do copy_context(), but early during initialization and squirrel the context away for later use. The latter seems plausible for 3rd party libraries (where the user might not have much control).
The same watcher callback cannot be registered multiple times simultaneously, each with its own config/state. This came up while I was refactoring TestContextObjectWatchers in Lib/test/test_capi/test_watchers.py to add some more tests for a change I’m working on.
I’m not sure how likely either is to come up in normal use, but it might be worth changing the API before the 3.14 release cements the design.