PEP 818: Adding the Core of the Pyodide Foreign Function Interface to Python (2)

PEPs
1

This is the second discuss thread for PEP 818. The previous thread is here. I got the following feedback:

  1. Various name changes were requested, which I implemented.
  2. The PEP was too long, so I reduced the length by about half by removing implementation details from the spec section.

I also added support for proxying and conversion of buffers, and for asyncio including an event loop, conversion between Python and JavaScript awaitables, and deferred destruction for arguments to an asynchronous JavaScript function called from Python.

In terms of motivation, I got confirmation that the maintainers of Emscripten-forge (an alternate Emscripten Python runtime) strongly support the proposal. They don’t have an easy way to use Pyodide’s foreign function interface currently so they reimplement it. But they would much prefer to share code, and this proposal is the most direct way to do that.

5 Likes
PEP 818: Adding the Core of the Pyodide Foreign Function Interface to Python
2

Regarding the asyncio integration, I’m wondering if we could make it more generic to allow integration with other Python event loops (for instance, Trio). It seems the only asyncio-specific code is ensure_future, add_done_callback and Future.

Also IIUC promise cancellation doesn’t really exist in JavaScript (even with abort signals), so I’m wondering what implications this has on the Python side.

4 Likes
3

if we could make it more generic to allow integration with other Python event loops

You mean something similar to set_asyncgen_hooks? I think it makes sense, I will add it.

Also IIUC promise cancellation doesn’t really exist in JavaScript (even with abort signals), so I’m wondering what implications this has on the Python side.

Well abort signals allow cancellation of certain promises, in the sense that when used the promise will be rejected and the IO work that the promise is waiting for also stops. But we don’t have any integration with Python cancellation and I don’t really know how to do such integration. For the most part the implication is that when a Python task is cancelled, the current IO that it’s waiting on keeps going. When it finishes, no further work will happen on the cancelled Python task. But the current IO job is not affected.

It would be possible to introduce some API like CancellablePromise(promise, on_cancel). Then you could do

controller = AbortController.new()
resp = fetch(some_url, signal=controller.signal)
wrapped = CancellablePromise(resp, controller.abort)

Annoyingly this essentially has to be done from user code. But I think it can be implemented downstream.

4

For integrating any Python event loop, I just meant that the Python code of the WebLoop should not depend directly on asyncio, but should use an API that could be implemented using asyncio or e.g. Trio. But given that AnyIO plays this role at the Python level, maybe this is not so important.

The way I see cancellation in JavaScript using abort signals is that the code has to be cooperative all the way down, and in general it is not. So a lot of IO work can be done even though it was cancelled on the Python side. There is an interesting blog post about that here.

2 Likes
5

FWIW there is this Trio branch which allows running it in Pyodide, and this demo:

1 Like
6

Thanks David! I actually made that repo with an LLM to research this topic. I guided it through the Trio internals, which I’m very familiar with, but left it to Pyodide-related decisions.

The main conclusion is that clearly it’s possible to make a version of Trio that supports Pyodide. But, without the hooks being discussed, awaiting JS async functions from Python or vice-versa is awkward:

# Call JS function from Python 
result = await trio.pyodide.call(js.jsFn, x)
# Allow Python function to be called from JS
js.pyFn = trio.pyodide.callable_from_js(py_fn)

With hooks these can be simplified to:

result = await js.jsFn(x)
js.pyFn = py_fn

The LLM suggests the following signatures for the hooks; I don’t know enough about Pyodide to know how sensible they are. With those in place, Trio wouldn’t need to interact with the WebLoop at all (but its presence would be harmless).

jstypes.ffi.set_awaitable_hooks(
    js_to_py=...,  # (jsawaitable, done_callback) -> Python awaitable
    py_to_js=...,  # (pyawaitable) -> JavaScript promise
)
jstypes.ffi.get_awaitable_hooks()  # -> the current pair, like sys.get_asyncgen_hooks()

On the topic of abort signals: these align very well with Trio’s cancel scopes. But there doesn’t need to be any specific support in Pyodide for them; normal library code can provide the necessary helpers for translating to/from cancel scopes. The resulting code would look like this:

# Call JS function from Python 
with trio.pyodide.abort_signal() as signal:
    result = await js.someAsyncFunction(arg, signal)
# Python function intended to be called from JS
async def fn(x, signal=None):
    with trio.pyodide.cancel_on_abort(signal):
        ... # becomes cancelled when signal is aborted
8

I think we can experiment with adding such hooks in Pyodide, and once we’re confident with them there we can add them to CPython. For now I’d prefer to keep the PEP features a subset of what Pyodide does. That reduces the maintenance hazard if we make mistakes in the initial design.