P2w: A Python-to-WebAssembly AOT Compiler (v0.2.1)

WebAssembly
1

I’ve been working on p2w, a side project that compiles a subset of Python to WebAssembly, using the WASM GC proposal for memory management. The compiler translates Python source directly to WAT, with no intermediate libraries (IL), which makes the codegen surprisingly easy to follow, though probably not good enough if we ever want proper optimisation passes.

The project was inspired by Eli Bendersky’s blog post Compiling Scheme to WebAssembly and his bobscheme compiler, as well as other previous projects, including several Python-to-JavaScript compilers.

What’s the point?

Mostly curiosity: how far can Python semantics be mapped onto WebAssembly and its GC extensions? Can we run Python in the browser without shipping a full interpreter?

I don’t pretend it has any industrial quality, and I’m not sure if this could be useful to anyone (even myself). But it does compile and run most of the close to 100 toy example programs I use to measure its progress, which is satisfying enough.

What works today (at least in part)

  • Data types: integers (arbitrary precision via i31/i64 boxing), floats, booleans, strings, bytes, lists, tuples, dicts, sets
  • Control flow: if/elif/else, for/while with break/continue/else, match statements
  • Functions: default args, *args/**kwargs, closures, lambdas, decorators
  • Classes: inheritance, properties, static/class methods, special methods
  • Generators, context managers, f-strings, walrus operator
  • Exception handling: try/except/finally, raise, chaining
  • JavaScript interop for browser integration
  • 100+ golden test programs, property-based testing with Hypothesis

On performance

Benchmarks show x3 to x10 acceleration over CPython — but only when using unboxed native integers (i32) via type annotations like x: i32, so it’s not really a fair comparison, but it’s also largely the point of the experiment. This comes at the expense of strict Python compatibility, since i32 will happily overflow where Python integers would not.

What this is not

This is a side project. Many Python features are still missing or incomplete. There’s no package ecosystem, no stdlib beyond builtins, etc. No serious application has been written in it yet. Expect (very) rough edges.

Source is on GitHub. Package is on PyPI.

13 Likes
2

Very cool! I gotta say, I am amused that the target language is called “Wat” :slight_smile:

(from your README:)

Nice. I was under the impression that WebAssembly code ran on a separate thread to anything that can use the DOM. Is that no longer the case? Am I out of date? If it’s now possible for Python code to compile directly to something that runs in a browser and has full access to the DOM, that’s extremely cool and I want to delve into it.

Do you have any sort of list of features that aren’t supported? And, is it your goal ultimately to support all features, or do you intend for this to forever be restricted (like PyPy’s rpy)?

1 Like
3

“Seamless” in the sense you can do:

import js

# Get canvas and context
canvas = js.document.getElementById("chart")
ctx = canvas.getContext("2d")

Which is (in theory) pretty cool. Internally, it still has to interact with JavaScript.

Do you have any sort of list of features that aren’t supported?

By design: anything that an AOT compiler can’t (reasonably) support, including “eval”.

Otherwise, a generic test suite would be useful to validate what works and what doesn’t (for every Python implementation). At some point, I used the micropython test suite ( micropython/tests/micropython at master · micropython/micropython · GitHub ) but not anymore.

1 Like
4

Yeah, that’s what I saw that piqued my interest.

How does this interaction with JS work? Is every call done asynchronously by sending a message to the JS event loop and getting something back? At what point does this happen? For example,with js.document.getElementById("chart"), a naive approach would be “hey JS, what is document? hey JS, take that thing, what is its getElementById? hey JS, call that thing with the parameter "chart" and give me back the result”, which would make for a pretty inefficient messaging system. OTOH, batching things up would mean you have to figure out when to actually send the batch across, increasing complexity.

Completely fair.

Cool, maybe I’ll try the core CPython test suite and see how much fails. There’s going to be a ton of stuff in the stdlib that will fail (presumably things like opening files and sockets won’t be permitted), but it would be neat to get a rundown of each failure categorizing them as “Not possible”, “Planned/desired”, “Not out of the question but I only have 168 hours a week”, etc.

1 Like
5

The parts of the CPython test suite that are used for emscripten builds might be a better check, as I assume that this compiler will have most, if not all, of the same limitations that emscripten does.

2 Likes
6

It almost certainly will do. But simple web assembly modules can be much much smaller, than a full posix emulation setup from emscripten (which can also be finecky).

7

Okay, so, one big one that’s gonna be needed is imports :rofl: I don’t think I can really try this without the ability to import other modules. But I did find a couple of fairly simple scripts of mine that don’t have a single import in them, and got this error when trying to compile:

$ python3 -m p2w squaresum.py |wat2wasm - -o squaresum.wasm
-:71:30: error: unexpected token "(", expected i32, i64, f32, f64, v128, externref or funcref.
  (global $tmp_pop_dict (mut (ref null eq)) (ref.null eq))
                             ^
-:71:35: error: unexpected token null.
  (global $tmp_pop_dict (mut (ref null eq)) (ref.null eq))
                                  ^^^^
-:71:40: error: unexpected token eq.
  (global $tmp_pop_dict (mut (ref null eq)) (ref.null eq))
                                       ^^
-:71:55: error: unexpected token eq.
  (global $tmp_pop_dict (mut (ref null eq)) (ref.null eq))
                                                      ^^

This is using wabt 1.0.32 as provided by Debian Bookworm, and also 1.0.36 as in Devuan Excalibur. What version of wabt have you been using? I see on GitHub that they’re up to 1.0.40, would have to try that next.

8

How does this interaction with JS work?

It’s extremely ad-hoc at this point, but all happens in a single thread, to answer your question. There must be (given the current specs, AFAIK, though I also understand there are proposals under way to get rid of it) a JS module between the WASM context and the browser.

Do you have any sort of list of features that aren’t supported?

In addition to “eval()”, I should add that the general philosophy is that things work better (or at all) when the Python source to be compiled is properly annotated.

1 Like
9

wasm-gc is not supported by wabt

10

Hmm, I don’t know enough about wat to be able to understand how that relates. Are you suggesting something that I can do differently?

11

Current version on my machine is:

❮ wat2wasm --version
1.0.39
12

The current project requirements list a WASM runtime with GC support. I’m not sure how this would work if wat2wasm lacks GC support.

@sfermigier Could you provide compiled WASM files for the demos?

13

Oh. Hmm. The project also stipulates that wabt is needed “for wat2wasm”. But also, I’m not sure if that’s the problem I was seeing.

14

It seems this was added with the GC proposal, so wat2wasm cannot compile it:

  • Add a new form of typed reference type ref $t and a nullable variant (ref null $t), where $t is a type index; can be used as both a value type or an element type for tables

I came to this conclusion while trying to compile the demos, since I’m not very familiar with the WebAssembly specs.

15

Sorry, the README was not up-to-date. The proper tool to use is wasm-tools, not wabt.

I have made a new release (0.2.2) and updated the README.

It’s now possible to call p2w with the -r option which calls the tools and the runner (nodejs for now) in sequence:

❮ cat fib.py
def fib(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a

print(fib(30))

❮ uv run p2w -r fib.py
832040

Or to call the tools one by one:

❮ p2w fib.py > fib.wat
❮ wasm-tools parse fib.wat -o fib.wasm

Then you have to run the .wasm file, which is a bit tricky. It needs a piece of JavaScript which is currently in src/p2w/runner.py

16

Ah. That doesn’t seem to be packaged by Debian, at least not under that name.

17

wasm-tools can be installed with:

cargo install --locked wasm-tools

Alternatively, on Linux distros that tend to favor stable packages, I install homebrew to get access to recent packages without breaking the system, and contains wasm-tools:

18

What sort of size are the generated files?

19

Too big. There is plenty of room for optimisation.

Remember it’s a POC.

2 Likes