Add ability of defining methods in C extension modules that can be awaited

Async-SIG
1

I have had a few cases where I simply wanted to create a simple C extension that does the following:

  • defines a single method that can be awaited where it would run in the currently running event loop without some dummy type objects in order to do the equivalent of async def in C extensions (which I consider to be bloat much). What I am thinking of is a define for PyMemberDef that is similar to CO_COROUTINE that acts as a shortcut and makes it much easier to make C extensions with methods that are awaitable. This is because at it’s current state I feel that it is unacceptable that nothing was done about it since the addition of async def to python’s syntax that this was not done as well. And yes I have a use case for this in some of my code in C where I have a module with normal type objects along with some global methods that must be awaitable.

I know what you might be thinking “Why not use cython and not write it manually in C?”. Well to me cython has bugs and issues for 1 in some places, 2 cython to me is bloat, and 3 I do not want to force people to install it when they do not want it just to obtain C code that can easily be made manually and then compiled without the need for cython. Also doing this could perhaps simplify the code that cython can generate for future versions of python which then enhances the api for everyone entirely and makes everything much easier as well.

2

How do you envision this working?

A synchronous function is simple: it gets called once, and then returns something. That’s a model that translates easily to a C function, so we support defining synchronous functions in C.

But an asynchronous method is much more complicated: it gets called, passes control back to an event loop, possibly several times, then eventually returns something. Internally, it’s implemented similar to generators. It’s not obvious to me how you would implement that in C in a simple way: you’d probably have to define some sort of struct to hold the state of the coroutine between await points, then define various slots to make it awaitable and what not.

So if you want to make it easier to define coroutines in C, you’ll have to come up with a concrete proposal for how to do it. Just adding a flag to the function definition isn’t going to be enough.

2 Likes
3

I was originally thinking along the lines of a special type of function pointer where the normal args are pushed to the right be 1 space with the first argument in them being PyObject *loop which would be the running event loop passed by the interpreter but yeah you do bring up an important point about state of the coroutine between await points and about possibly needing to await other coroutines from within that C code as well (that could potentially be implemented in the same C extension, in another extension, or even in pure python itself) which all needs to be considered as well.

4

How would you await anything in C extension module function? Even ignoring contextvars management, the equivalent of await something is not something you want to do in C.

1 Like
5

If you’re able/willing to use callbacks, it may not be so hard? The “async” C code would then register some Python functions as callbacks, and “yields” would be converted into calls of those functions. The Python functions could mostly be boilerplate (getting access to GIL + Python thread state). If the C code would run on its own internal event loop - this could work…

6

I am certain this exact topic has come up in the past (probably in the last year, or two). I don’t recall where exactly but probably someone with some time on their hands could find it and link it here. Then we’d spend less time arguing back and forth over what we could actually do and how.

7

I recall it, too, in about that timeframe. But I went looking for it a yesterday and couldn’t find it. That might be a comment on how well I can use Discourse search. Hopefully someone finds it.

8

This one? Adding a C API for coroutines/awaitables

9

As a band aid solution for whatever your use case is, I did write this implementation a little while ago that does exactly this. You can take a look at my original discussion for more information, but it works like the following:

static int coro_callback(PyObject *awaitable, PyObject *result) {
    printf("coro result is: ");
    PyObject_Print(result, stdout, Py_PRINT_RAW);
    return 0;
}

static PyObject *my_function(PyObject *self, PyObject *args) {
    PyObject* coro;
    if (!PyArg_ParseTuple(args, "O", coro))
        return NULL;

    PyObject* awaitable = PyAwaitable_New();
    if (PyAwaitable_AddAwait(awaitable, coro, coro_callback, NULL) < 0) {
        Py_DECREF(awaitable);
        return NULL;
    }

    return awaitable;
}

Note that this isn’t really stress tested or anywhere near perfect, but it should cover basic async features.

10

I recall a longer thread. Maybe a GitHub issue?