C API for asynchronous functions

Core Development C API
1

I discussed this topic a year ago and feel like it might be a good idea to revisit it now that I’ve properly built a reference implementation. I suggested adding some sort of API for calling asynchronous Python functions from C. It seemed to be a somewhat impossible task, but I did manage to come up with an API that works relatively nicely.

You can see my implementation here

It works like this:

  • Instantiate a “transport” class (this is called Awaitable in my implementation) via PyAwaitable_New
  • Use PyAwaitable_AddAwait to submit coroutines to be handled by Awaitable, along with callback functions to be used for the results
  • Save values needed in callbacks via PyAwaitable_SaveValues
  • In either the main function or one of the callbacks, call PyAwaitable_SetResult to set a “return value” for the awaitable (i.e. what is returned after doing an await)

To show it visually, here’s an example:

static int // -1 is an error, 0 is OK
awaitable_cb(PyObject *awaitable, PyObject *result)
{
    PyObject *some_object;
    if (PyAwaitable_UnpackValues(awaitable, &some_object) < 0)
        return -1;
    // result is a borrowed reference to the return value of the coroutine 
    return PyAwaitable_SetResult(awaitable, result);
    // PyAwaitable_SetResult returns 0 or -1, so we can just return the value of that here
}

static PyObject *
test_awaitable(PyObject *self, PyObject *args)
{
    PyObject *some_object;
    PyObject *coro;
    if (!PyArg_ParseTuple(args, "OO", &some_object, &coro))
        return NULL;
    // precondition: coro is a coroutine instance
    PyObject *awaitable = PyAwaitable_New();
    if (!awaitable)
        return NULL;

    // this saves a reference to some_object inside of the awaitable
    if (PyAwaitable_SaveValues(awaitable, 1, some_object) < 0) {
        Py_DECREF(awaitable);
        return NULL;
    }
    
    // the fourth argument here is a callback for if an error occurs when awaiting the coroutine
    if (PyAwaitable_AddAwait(awaitable, coro, awaitable_cb, NULL) < 0) {
        Py_DECREF(awaitable);
        return NULL;
    }
    return awaitable;
}

Now, when you call test_awaitable from Python, you can use it like any other async function:

async def foo():
    await asyncio.sleep(1)
    return 1

print(await test_awaitable(foo())  # prints 1

I do think this does need further development, but it is an async C API that works. This does introduce a problem of callback hell, so that is something to think about. Perhaps a PEP could be written for clearer specifications or more opinions?

4 Likes
2

Wow, this is nice!

But, as far as I can see, it only uses the public C-API and does not need any changes in CPython itself.
Your next step would be to release it as a C library – even if you do want to integrate this into CPython, it’ll be easiest if it exists in an immediately usable form (and even after it’s accepted, as a “backport” for existing Python versions).
Similarly: instead of a PEP, write documentation. You can turn docs into a PEP if you want to go that way.

If you go that way, please choose a different prefix for the API names; keep Py* for the PEP.

To set expectations: I don’t think this needs to be in CPython itself. I know that sounds disappointing, but after all, Python is made to be extendable; we want to have interesting projects outside the core.

3

Much to the contrary: I very much do think that we need a core maintained async C API in CPython.

We currently don’t have a way to write Python C extensions which work in async mode without using a shim written in Python which takes care of the interactions with the asyncio (and other) event loops.

Peter’s implementation provides a way to call async functions from C. That’s a great start. Next would be to have a way to define async functions written in C, so that you can, for example, implement cursor.execute() calls in the DB API as an async function/method in C.

3 Likes
4

Perhaps both could be done? A self-maintained version by me for backports, and then a fork for CPython’s core?

In regards to writing async functions in C, this implementation actually does do that as of now (or at least it mimics its behavior), as per my example. Returning an awaitable makes the C function itself usable via await.

If you’re interested in seeing a “proof of concept” to make sure that this works, I’ve done quite a bit of work implementing some of ASGI in C with this API.

3 Likes
5

Agreeing with what’s been said so far, I haven’t been able to fully understand the proposed implementation or its API. Maybe it’s due to the terminology used, maybe there should just be more English sentences explaining the purpose and detailed usage besides the implementation and a fragment of an example.

6

My apologies, I had to dumb things down a bit to keep the initial post relatively short. I’ll turn PyAwaitable into a library with proper documentation as suggested, and we can go from there.

7

Has anything new been added to this? It would be great if this was a thing and could be mixed with the same structure and stuff that normal sync apis are defined in C extensions (as static PyMethodDef arrays that can be a member of a custom typeobject or directly inside of the module definition itself.

And Honestly if it could be flagged as such in the CPython source code using METH_COROUTINE with that being a new flag for them to use this implementation when it gets completed. Also it would allow me to migrate most of my async bits on my discord bot core code to the C extension module for it instead of having a split where I can store only non-async code inside the C extension and async code within the Python scripts. It would simplify it to only the __init__.py and __main__.py files.

8

CC: @encukou

There’s a scrapped PEP on this topic. I’m going to be turning my reference implementation into a library, as my version only uses the CPython public API.

9

See @ZeroIntensity’s proposal: Revisiting a C API for asynchronous functions. It spells this METH_ASYNC rather than METH_COROUTINE, but otherwise it is what you ask for.

But, I think that a flag for PyMethodDef is a quality-of-life improvement that should be added on top of a proven, working, popular mechanism. It’s a goal to aim for, but not something to do right now. (Compare how asyncio used generators for a while before async/await was added to the language.)

1 Like