PEP 783: Emscripten Packaging

PEPs
61

Well I link to the Pyodide abi documentation for the definition of the platform. I could put that information in the pep if people think it’s appropriate but would we then update the pep with each new abi? I thought making an external “living document” and linking it from the pep was the appropriate approach.

So if I have a private Emscripten build of Python and want to build my own wheels for it, is there any guarantee that there will be a wheel tag I can use for that Emscripten version?

I tag them with platform tag like emscripten_3_1_58_wasm32. But these shouldn’t be uploaded to pypi because they are underspecified in terms of abi. I had this in the pep before but removed it because @pf_moore felt it was inappropriate.

people do build stuff on their own for their own reasons and I think we should try to allow for that.

I think it’s helpful for packaging to have basic tools for this but for pypi upload they’d presumably need to define their own abi and get their own platform tag for it?

62

That’s normal if the PEP is marked as “active” and thus is a living document.

I guess what you’re saying is Pyodide is the platform and will always own the meaning of the tag, and thus hosting the details externally on the platform’s site is fine. And so if we are all good with making Pyodide a platform on the level of macOS and manylinux then having Pyodide keep its own docs on this is reasonable. It also means that any other Emscripten-based platform will either need their own tag of let Pyodide control what they do if they want to reuse the tag (and I’m not saying that’s a bad thing, just a thing).

63

In the long run, perhaps we will want to upstream control of the Emscripten runtime into CPython. At that point recording the details of the ABI in a PEP would make the most sense. Then Pyodide could turn into a set of packaging tools for the platform. I think in the meantime, keeping the platform definition for humans closer to the code that defines the platform is a good way to go.

any other Emscripten-based platform will either need their own tag of let Pyodide control what they do if they want to reuse the tag

Agreed. I think this is unavoidable in any case.

1 Like
64

I spoke with Hood about this at PyCon. I’d love to see this move forward. The externally hosted definition of the platform is linked from the PEP, and there’s a PR waiting in the packaging repo. cibuildwheel’s been able to build these for a long time now.

One thing I think helps is considering the future of the platform. Having a tag for an implementation is common; PyPy and GraalPy are some examples. However, the interesting thing here is that if CPython support progresses to Tier 2 (it’s not even back up to tier 3 yet, this is forward thinking), then CPython will want to have a defined Emscripten platform tag (likely identical to the pyodide one). I think that’s completely fine; either the source of the pyodide tag definition moves to CPython, or a new tag can be introduced - if they become synonyms, that’s similar to what happened with manylinux_2*.

I think this comparison between manylinux and emscripten is a good one, actually, though they are different. A “linux” wheel works on just the system it was generated on, much like an emscripten wheel. Tagging it with manylinux means it is ensured to work on a larger set of Linuxes, while tagging an emscripten wheel with pyodide means it was generated with a specific emscripten version and set of ABI flags to work on Pyodide or any build of Python with matching flags. From a distribution standpoint, this is nearly identical; you have a way to describe specific wheels you build locally and only work on a specific system, and a way to describe wheels that are suitable for distribution and work with an ecosystem.

65

What are the next steps here?

66

We are waiting on @dstufft to decide on whether or not to approve the PEP. Unfortunately, there is no timeline for when he will read it and as far as I understand there is no alternative for us other than waiting.

67

Did I miss the post where the PEP was formally submitted for pronouncement? As far as I can see, the discussion just died down without that.

68

Oh. Nobody who I’ve asked about it said I need to make such a post. How do I do that?

69

Just post the request here, and @-mention Donald.

You need to request approval on any PEP when it’s ready - the PEP sponsor should be able to tell you that (the sponsor is supposed to guide you through any parts of the process you need help with). In this case, it’s a packaging PEP, so the normal process (an issue on the SC tracker) isn’t used, we just make a request here, as I said. That’s not well documented (the packaging workflows are a bit less formal than the core ones) but if in doubt, just ask :slightly_smiling_face:

1 Like
70

Thanks for explaining @pf_moore! In that case, I formally submit this PEP cc @dstufft.

5 Likes
71

Hello,

I appreciate the effort behind this PEP and share the goal of establishing a standardized wheel format for Emscripten-based Python distributions. However, I would like to propose an adjustment to ensure the format is inclusive and adaptable to a broader range of use cases, including our own.

Context

We are actively developing emscripten-forge, a general-purpose software distribution for the web browser. Unlike Pyodide, emscripten-forge is not limited to the Python ecosystem. It also supports R, GNU Octave, C++ libraries, and command-line utilities. Our aim is to ensure that the CPython build included in our distribution is compatible with the proposed wheel format.

The naming of the wheel format

The current proposal ties the wheel format explicitly to Pyodide (e.g., pyodide_${YEAR}_${PATCH}_wasm32). While distributions like emscripten-forge can use the same libraries and build flags to maintain ABI compatibility, the name suggests a dependency on Pyodide-specific features beyond just the ABI.

For example, WebAssembly-based software often needs to access external resources, such as making HTTP requests. Pyodide addresses this with a Python-JavaScript bridge based on Emscripten-embind, exposing browser APIs like fetch to Python. Emscripten-forge uses a similar bridge, pyjs, which combines Embind with Pybind11. The APIs of pyjs and pyodide.ffi are functionally similar, but not identical. Fortunately, packages like urllib3 work seamlessly with both bridges (by adding a mock pyodide package), but this compatibility is not guaranteed for all packages if the formats diverge.

Proposal

To avoid fragmentation and ensure broad compatibility, we propose:

  1. Renaming the wheel format to emscripten_${YEAR}_${PATCH}_wasm32 to reflect its broader applicability.

  2. Defining the ABI independently of the Pyodide distribution, in a neutral repository related to Python packaging. This would allow Pyodide, emscripten-forge, and other stakeholders to contribute as peers.

  3. Establishing a common denominator for APIs (e.g., pyodide.ffi and pyjs) that package authors can rely on, reducing the need for downstream patches. This could be a subset of the Pyodide API, that would be importable under a name that does not contain the string “pyodide”.

This approach would foster collaboration, minimize fragmentation, and ensure that the wheel format serves the entire Emscripten-based ecosystem.

Benefits

  • Inclusivity: The format would not be tied to a single distribution.

  • Compatibility: Packages like urllib3 would work across distributions without modification.

  • Future-proofing: A shared specification would encourage consistency and reduce maintenance burdens for package authors.

I believe these changes would strengthen the PEP and benefit the entire community. We would be happy to contribute to this effort, work with @hoodmane, and help define the common ABI and API standards.

9 Likes
72

I don’t know too much about this space other than that I recently became a maintainer of msgspec and that we have to support this target to match the support matrix of our users. I was pleasantly surprised to find out that builds work out-of-the-box for me locally (in a Linux container) and that there has been support in cibuildwheel for a while now. The main issue is that users have no easy way to depend on libraries because PyPI lacks support due to there being no official standard.

If technically possible, I would love for the suggestions in the previous comment to be incorporated into the PEP.

2 Likes