Ok, I see your point. I do see how this can get complicated pretty quickly, but I still believe that having some API for this is better than having nothing at all. I came up with an API that looks like this:
static int module_something_callback(PyObject* awaitable, PyObject* result) {
int a;
int b;
PyObject* bar;
PyAwaitable_Unpack(awaitable, &a, &b, NULL, &bar);
PyObject* result = PyObject_Call(bar, PyLong_FromLong(PyLong_AsLong(result) + (a + b)), NULL);
PyAwaitable_SetResult(awaitable, result);
PyAwaitable_RETURN_SET_RESULT;
}
static PyObject* module_something(PyObject* self, PyObject* args) {
PyObject* awaitable = PyAwaitable_New();
int a;
int b;
PyObject* foo;
PyObject* bar;
if (!PyAwaitable_ParseTuple(awaitable, args, "iiOO", &a, &b, &foo, &bar))
return NULL;
PyObject* foo_coro = PyObject_Call(foo, PyTuple_New(0), NULL);
// lets pretend that PyAwaitable_AddAwait checks that its a coroutine, just for this example
PyAwaitable_AddAwait(awaitable, foo_coro, module_something_callback);
return PyAwaitable_AsCoroutine(awaitable);
}
The above would be equivalent to the following Python code:
async def something(a: int, b: int, foo: Callable[[], Awaitable[int]], bar: Callable[[int], int]):
foo_coro = foo()
result = await foo
return bar(result + (a + b))
- Starting in
module_something, we build our “magic awaitable” through PyAwaitable_New. Then, we initalize our variables for PyAwaitable_ParseTuple. The idea with that is to do the same job as PyArg_ParseTuple, but also store the parsed arguments on the awaitable object to be used in callbacks later (PyAwaitable_Unpack).
- Next, I changed the original idea of
PyAwaitable_GetCoro to a simple PyObject_Call, since all we are doing is calling the object.
- Now, with
PyAwaitable_AddAwait, it’s essentially just adding the coroutine object to an array on the object to be used later, as well as store the callback. More on this later.
- Finally, we return our magic awaitable as some sort of coroutine. At this point, all awaits have been registered to our awaitable and will be managed by it.
- In
module_something_callback, I stated above that the arguments parsed by PyAwaitable_ParseTuple would be stored on the object. Ideally, this would be how you access them, passing NULL for unneeded parameters.
- We take the result of
foo (result parameter), and then add it with parameters a and b.
- Then, we take the result of
bar, and set it to the “result” of our coroutine. PyAwaitable_RETURN_SET_RESULT would be a macro that just tells our awaitable that a return value was set inside the callback.
Regarding the “state machine”, that’s what Python coroutines are, right? From some research I did before , I thought coroutines were just generators with a state value that keeps track of it.
I came up with an object model that looks like this:
typedef int (*awaitcallback)(PyObject*, PyObject*);
typedef struct {
PyObject* coro; // python coroutine object
awaitcallback callback; // actual callback function
} AwaitableCallback;
typedef struct _Awaitable {
PyObject_HEAD
AwaitableCallback** callbacks; // array containing callbacks and their coroutines
Py_ssize_t callback_size; // size of the array above
int state; // current state of the coro
PyObject* result; // final return value
void** tuple; // arguments saved by PyAwaitable_ParseTuple
Py_ssize_t tuple_size; // size of the array above
} Awaitable;
I do realize that this API isn’t exactly perfect, but more just a start of what could be something useful. I’m still looking for some feedback on what needs to be improved and/or fixed here.
In terms of why this is needed, take a look at ASGI as an example:
async def app(scope, receive, send):
await send(...)
In my case, I would like to write my app function in C, but I need to await send, so I can only mix C in with the actual function.
Also, when using C, a lot of the time you end up performing operations that block I/O, so we would want something like asyncio.to_thread, which is completely ok, but may be inconvenient for some cases, and in my opinion having to access a module just to use normal Python syntax seems a little bit tedious.
Technically, asyncio.to_thread will actually slow down your code as well. A big reason people use the C API is to well, make their code faster. I get that we shouldn’t really care about performance in Python, but it’s always nice when we can make it a little bit faster.
Above all, I just think it’s a little silly to have to access Python code to use async/await from C.
Almost certainly users would rather write their own synchronous functions and wrap them up in a Python (or better yet, Cython) coroutine that calls them as needed.