User data pointers for code watcher callbacks

Core Development C API
1

I previously posted this in Python Help but I get the impression most of the people there are not using the C API and since this is a potential interface change it seems appropriate to ask about here.

As far as I can tell you are forced by the code watcher API to use global vars. The callbacks take raw C pointers, and they don’t carry a void* user data parameter. Is there a trick to this? Or a reason the standard user void* pattern for C callbacks isn’t used? AFAICT this prevents being able to use the module state functions, because PyModule_GetState needs a pointer to the module to retrieve the state. Using a global instead would prevent being able to reload the module, since you will end up with two module objects but only one global.

/*
 * A callback that is invoked for different events in a code object's lifecycle.
 *
 * The callback is invoked with a borrowed reference to co, after it is
 * created and before it is destroyed.
 *
 * If the callback sets an exception, it must return -1. Otherwise
 * it should return 0.
 */
typedef int (*PyCode_WatchCallback)(
  PyCodeEvent event,
  PyCodeObject* co);

/*
 * Register a per-interpreter callback that will be invoked for code object
 * lifecycle events.
 *
 * Returns a handle that may be passed to PyCode_ClearWatcher on success,
 * or -1 and sets an error if no more handles are available.
 */
PyAPI_FUNC(int) PyCode_AddWatcher(PyCode_WatchCallback callback);
1 Like
2

Yeah, this was brought up for PyContext_AddWatcher too. I’m not a fan of passing void * pointers because they complicate things (e.g. who manages the lifetime? what if the pointer needs to hold a Python object and do reference counting on it?), so I think it’s easier to just let users pass their own arbitrary PyObject * rather than a void *.

Though, something @rhansen suggested was instead of the AddWatcher APIs taking a C function, why not a callable Python object? (You can wrap C functions into an object pretty easily with PyCFunction_New if you’re worried about ease of use.)

1 Like
3

If I’m understanding right though I can’t pass a PyObject right now or a python callable, can I? Since I assume they’re not a C function pointer type. Or you’re saying that could have been an alternative interface?

I would guess that a python callable would be higher overhead, and the interface seems intended for things like profilers where that could be significant.

1 Like
4

Yeah, that’s what I mean.

Eh, not much. I guess it would be slightly slower because there’s a few extra pointer dereferences, but it wouldn’t be much different if you were to just call Python code from the callback as the API is right now. C functions wrapped in a PyCMethod will be just as fast.

1 Like
5

FYI: gh-127124: Change context watcher callback to a callable object by rhansen · Pull Request #127247 · python/cpython · GitHub

If that PR is accepted maybe it can serve as a reference for adding an alternative to PyCode_AddWatcher.

1 Like