C Python API: os.register_at_fork analogue with C function pointer arguments

Python can register callbacks using os.register_at_fork() to occur surrounding forks. The c/python api documents functions like PyOS_BeforeFork(), but does not provide a way to register functions implemented in C to occur prior to a fork. Rather than have to take a C function, make a python wrapper for it, use the c/python api to get the function pointer to os.register_at_fork(), and call it with the C function’s python wrapper as an argument, I propose adding a new PyOS_RegisterAtFork c/python api function that would take 3 C function pointers (or NULL) that would operator analogously to os.register_at_fork().

1 Like

C callbacks generally aren’t simple. They typically need some context, usually a void* that you specify at registration and it gets passed to the function. And the pointer needs lifecycle management (freeing). Those aren’t needed in all use cases, but callback registration API generally should allow them.
You can think of Python callables as wrappers that include the function pointer and all the other bits it needs.

Of course, BeforeFork is process-wide thing, so it might be that complexity will never be needed. Raw function pointers could be fine. We might never add un-registration. Un-freed memory at process exit might be fine.
But all those are some of the things to think about before adding “simple” API.

Maybe we should instead invest in better API to make a Python callable from a function pointer – e.g. properly documenting functions like PyCFunction_New.

1 Like

My idea was PyOS_RegisterAtFork arguments would be of type void (*func)(void), the analogue of the function pointer arguments of os.register_at_fork. This would avoid the additional complexity required for arguments of type void (*func)(void *), and given this would happen once per process, not freeing a function pointer (or deferring to the code that registered the callback) doesn’t seem bad.

Could you provide an example of defining a python function from a C function? I haven’t been able to figure it out. If it were simple to define a python function using a C function, this process does not seem too terrible for the initial use case above:

  • define python function, before_fork_cb, using C function
  • PyImportModule os, acquire handle to os.register_at_fork, reg_at_fork
  • Make PyDictObject, d = {“before”:before_fork_cb}
  • register py_cfunc using PyObject_Call(reg_at_fork, NULL, d)

Fill in a PyMethodDef and call PyCFunction_New(methoddef, NULL). Or instead of NULL, use an object you need to keep alive for the function to work.

PyCFunction_New is, unfortunately, not documented. IMO it should, and I opened an issue for that: PyCFunction_New is not documented · Issue #102468 · python/cpython · GitHub

Yes. It’s boilerplate-y, but not needed very commonly, so I don’t think we need to expose a helper for that.

Thank you for your help, I really appreciate it!

@encukou should it be as simple as this? I’ve tried a handful of ways, but all end with a segfault with python 3.9.13.1. I wonder if this is because the parent process is the one allocating the memory, but the child process is trying to _Py_DECREF it.

// Registered function
void fork_callback_after_in_child(){
    return;
}
// Registering function
PyObject *os_register_at_fork_kwargs = PyDict_New();
PyMethodDef def = {"fork_callback_after_in_child", (PyCFunction) fork_callback_after_in_child, METH_NOARGS};
PyObject *pyfunc = PyCFunction_New(&def, NULL);
Py_INCREF(pyfunc); // dont think this is necessary
if(PyDict_SetItemString((PyObject *) os_register_at_fork_kwargs, "after_in_child", pyfunc) != 0){
  ERROR("register_at_fork: failed to register fork_callback_after_in_child");
}
PyObject_Call(os_register_at_fork, PyTuple_New(0), os_register_at_fork_kwargs);

Core dump comes from:

(gdb) where
#0  _Py_DECREF (op=0x7f1d9e19863b <fork_callback_after_in_child>) at ./Include/object.h:422
#1  run_at_forkers (lst=<optimized out>, reverse=<optimized out>) at ./Modules/posixmodule.c:553
#2  0x00007f1d9dccb63e in os_fork_impl (module=<optimized out>) at ./Modules/posixmodule.c:6697
#3  os_fork (module=<optimized out>, _unused_ignored=<optimized out>) at ./Modules/clinic/posixmodule.c.h:2750
#4  0x00007f1d9dd40748 in cfunction_vectorcall_NOARGS (func=0x7f1da01c15e0, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:489

Sorry for the delay. Covid put me offline for a few weeks, and I’m getting through a backlog so I didn’t look at this this in detail – though I’ll need to, to make sure the docs will be correct.
I suspect the issue is that the PyMethodDef is not copied, so it must outlive the function object (e.g. be static).

1 Like

posix already provides C APIs for before and after fork via C function pointers. Just call that from C with your C function pointers.

https://pubs.opengroup.org/onlinepubs/009604499/functions/pthread_atfork.html

It does not.make sense to me to expose the concept of a C function pointer at the Python level. You could do this using ctypes of for some strange-to-me reason you wanted to do that from Python itself.

1 Like