Hello,
this is a spin-off of a discussion which originated in the WebAssembly category:
Among the other things in that thread we also talked about the possibility of publishing WASM wheels on PyPI. To quote myself:
This seems to be technically possible: Pyodide is a Python/WASM distribution which compiles the core interpreter and a (big) set of extension modules. The extension modules are distributed in the form of wheels using their own CDN, and installed by their own tool called micropip.
There is another project doing something similar: emscripten-forge is a “conda-style” python distribution based on mamba, which compiles its own python and various extension modules.
Recently we tried to load an extension module compiled by/for emscripten-forge, repackage it as wheel and install/import it into pyodide: it “just worked”. IMHO this proves that the current technology behind WASM/emscripten/dynamic linking is mature enough that you can reliably build and distribute 3rd-party extension modules.
So, to put it more concretely:
does the packaging sub-community think it would be a good idea to support WASM wheels on PyPI?
I’ve implemented support for shipping rust binaries as wasm/wasi in maturin when compiling with maturin build --target wasm32-wasi. Currently this is done by depending on wasmtime-py and a small launcher script, but i’d be really interest in getting support for first-class wasm wheels, both binaries and extension modules.
@konstin this may need a new thread, but what are you doing with the rust binaries as wasm/wasi? Do they interact with a WASI or native CPython interpreter? Or are they run like standalone programs? If they’re simple data that the Python code pipes through wasmtime, they might not affect the wheel tags at all and could go into a none-any wheel.
They are just simple standalone scripts, essentially an abstraction over wasmtime hello-world.wasm. They do currently use none-any tags, but they depend on wasmtime which reduces platform support quite a bit. It’s more of technical demo if we can conveniently ship wasm wheels. I’d be super interested though into extending this into something that can actually interact with python code, e.g. could pyo3 or cffi target pyodide and maturin build hypothetical py38-emscripten-wasm wheels?
I suppose it would be possible to expose the https://hpyproject.org/ API to WASM, loading WASM extension modules into CPython’s from various architectures. Not sure if it would be worth the trouble.
could pyo3 or cffi target pyodide and maturin build hypothetical py38-emscripten-wasm wheels
emscripten/wasm32 wheels creation via Pyodide is already part of the mathurin CI
WASM binary compatibility work? Is it any better than Linux ABI compatibility?
In the case of Emscripten, it’s both the build toolchain and the OS. There is no backward compatibility guarantee, so the plan so far in Pyodide is to fix the Emscripten version for a given Python version. We also agreed that this Emscripten version would be used both by Pyodide and emscripten-forge, so that the produced packages are compatible.
If you do fix the Emscripten version, it means you also fully control the runtime environment, which makes a lot of things easier compatibility-wise. One thing that needs to be specified is what libraries are linked into the interpreter and provided as part of Emscripten main module though.
There are docs about it here. As far as I know for now the only Python package with Rust extensions we build is cryptography (but that uses setuptools-rust). There was some work on building tokenizers with mathurin in pyodide#2816. @hoodmane who worked on this, might know more.
In Pyodide we have two rust wheels cryptography and bcrypt. Rust packages with no C code generally seem to be significantly easier to build than normal C code. There is more work to be done here though: maturin has 23 test crates and they are only testing 2 of them against Pyodide. It would be better if we could enable more of those.