# Revisiting a C API for asynchronous functions

**URL:** <https://discuss.python.org/t/revisiting-a-c-api-for-asynchronous-functions/50792>\
**Category:** C API\
**Created:** [April 12, 2024, 1:04pm UTC](https://discuss.python.org/t/revisiting-a-c-api-for-asynchronous-functions/50792 "2024-04-12T13:04:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZeroIntensity](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zerointensity/32/10639_2.png) [@ZeroIntensity](https://discuss.python.org/u/ZeroIntensity)\
**Post date:** [April 12, 2024, 1:04pm UTC](https://discuss.python.org/t/revisiting-a-c-api-for-asynchronous-functions/50792/1 "2024-04-12T13:04:17Z")

</div>

Hello everyone!

Similar topics of mine have gone through discussion previously [in January](https://discuss.python.org/t/c-api-for-asynchronous-functions/42842) and [last year](https://discuss.python.org/t/adding-a-c-api-for-coroutines-awaitables/22786). I would recommend reading these two to fully understand this discussion.

Over the two weeks or so, I’ve gone back and forth with @encukou on developing a PEP on this topic. My [reference implementation](https://gist.github.com/ZeroIntensity/5cd960b0b9ce1e27a99fb11aee434d46) only uses the public API, and would be a better fit as a library rather than an addition to CPython. For this reason, my previous draft PEP was scrapped in favor of turning it into a library. The details of my previous implementation are long and complicated, but in short, `PyAwaitableObject *` is an object that stores coroutines, then uses ` __await__ ` and ` __next__ ` to yield them to the event loop. For exact information on how this works, see [my scrapped PEP](https://gist.github.com/ZeroIntensity/8d32e94b243529c7e1c27349e972d926#overview).

I’ve come up with a new design that proposes a `METH_ASYNC` flag (inspired by [this discussion](https://discuss.python.org/t/add-ability-of-defining-methods-in-c-extension-modules-that-can-be-awaited/32962)) but it certainly requires some discussion before a PEP could be written. Ideally, if an API like this was to be added to CPython, my reference implementation would exist as a backport library to allow others to pick it up immediately (similar to what `asyncio` did).

The redesigned API looks like this:

```c
static PyObject *
hello(PyObject *self, PyObject *awaitable, PyObject *args)
{
  PyObject *coro;
  if (!PyArg_ParseTuple(args, "O", &coro))
    return NULL;
  
  if (PyAwaitable_AWAIT(awaitable, coro) < 0)
    return NULL;

  Py_RETURN_NONE; // equivalent to PyAwaitable_SetResult(awaitable, Py_None)
}

static PyMethodDef HelloMethods[] = {
  {"hello", hello, METH_VARARGS | METH_ASYNC, NULL},
  {NULL, NULL, 0, NULL}
}

```

The two key upgrades in the new design are:

- The user does _not_ have to make any call to `PyAwaitable_New`, nor worry about `Py_DECREF`ing the awaitable on an error check. This simple change eliminates a lot of boilerplate of `Py_DECREF` calls.
- The user may return any `PyObject *` instead of just the awaitable, which may be more clear for setting return values.

I have written a prototype for this [here](https://github.com/ZeroIntensity/cpython-meth-async). This can go a step further and use `PyObject *` return values in callbacks as well (this is not included in the prototype implementation):

```c
static PyObject *
hello_callback(PyObject *awaitable, PyObject *result)
{
    // Assume that result is an int
    long value = PyLong_AsLong(result);
    if ((value == -1) && PyErr_Occurred())
        return NULL;

    if (value == 24)
        return PyLong_FromLong(42);

    PyAwaitable_RETURN_IGNORE; // This indicates that the return value should not be updated, and will remain as None
}

static PyObject *
hello(PyObject *self, PyObject *awaitable, PyObject *args)
{
  PyObject *coro;
  if (!PyArg_ParseTuple(args, "O", &coro))
    return NULL;
  
  if (PyAwaitable_AddAwait(awaitable, coro, hello_callback, NULL) < 0)
    return NULL;

  Py_RETURN_NONE; // equivalent to PyAwaitable_SetResult(awaitable, Py_None)
}

static PyMethodDef HelloMethods[] = {
  {"hello", hello, METH_VARARGS | METH_ASYNC, NULL},
  {NULL, NULL, 0, NULL}
}

```

A few notes:

- My prototype implementation is simply a thin wrapper on top of my previous reference implementation, but this would not be the case in an actual CPython version.
- The previous implementation, as stated before, uses a new `PyAwaitableObject *`, but perhaps a new object is not needed? Maybe the `PyObject *awaitable` in the above example could refer to a `PyCoroObject *`, but that would obviously require some modifications to the coroutine implementation.
- I chose `PyAwaitable_` as the prefix previously as that was the object’s name. In this new design, perhaps this prefix could change (such as maybe something like `PyAsync_`). Alternatively, if the coroutine object just gets reused, the `PyCoro_` prefix should be chosen instead.

Nonetheless, I’m not totally sure whether `METH_ASYNC` would be needed in the first place if my library exists, but that’s what this discussion is for 😃

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [April 15, 2024, 6:42am UTC](https://discuss.python.org/t/revisiting-a-c-api-for-asynchronous-functions/50792/2 "2024-04-15T06:42:14Z")

</div>

I don’t quite see the benefit over creating and returning the awaitable object from a normal function, something like this (using names from the scrapped PEP):

```c
static PyObject *
hello(PyObject *self, PyObject *coro)
{
  PyObject *awaitable = PyAwaitable_New();
  if (!awaitable) {
    return NULL;
  }
  if (PyAwaitable_AWAIT(awaitable, coro) < 0) { 
    Py_DECREF(awaitable);
    return NULL;
  }
  if (PyAwaitable_SetResult(awaitable, Py_NONE) < 0) { 
    Py_DECREF(awaitable);
    return NULL;
  }
  return awaitable;
}

static PyMethodDef HelloMethods[] = {
  {"hello", hello, METH_O, NULL},
  {NULL, NULL, 0, NULL}
}

```

Eliminating a few lines of boilerplate in C code typically isn’t worth adding a new feature. Even if it was in this case, it’s better to do it after the underlying design (with a “verbose” API) gets some real-world use.