PEP 783: Emscripten Packaging

PEPs
61

Well I link to the Pyodide abi documentation for the definition of the platform. I could put that information in the pep if people think it’s appropriate but would we then update the pep with each new abi? I thought making an external “living document” and linking it from the pep was the appropriate approach.

So if I have a private Emscripten build of Python and want to build my own wheels for it, is there any guarantee that there will be a wheel tag I can use for that Emscripten version?

I tag them with platform tag like emscripten_3_1_58_wasm32. But these shouldn’t be uploaded to pypi because they are underspecified in terms of abi. I had this in the pep before but removed it because @pf_moore felt it was inappropriate.

people do build stuff on their own for their own reasons and I think we should try to allow for that.

I think it’s helpful for packaging to have basic tools for this but for pypi upload they’d presumably need to define their own abi and get their own platform tag for it?

62

That’s normal if the PEP is marked as “active” and thus is a living document.

I guess what you’re saying is Pyodide is the platform and will always own the meaning of the tag, and thus hosting the details externally on the platform’s site is fine. And so if we are all good with making Pyodide a platform on the level of macOS and manylinux then having Pyodide keep its own docs on this is reasonable. It also means that any other Emscripten-based platform will either need their own tag of let Pyodide control what they do if they want to reuse the tag (and I’m not saying that’s a bad thing, just a thing).

63

In the long run, perhaps we will want to upstream control of the Emscripten runtime into CPython. At that point recording the details of the ABI in a PEP would make the most sense. Then Pyodide could turn into a set of packaging tools for the platform. I think in the meantime, keeping the platform definition for humans closer to the code that defines the platform is a good way to go.

any other Emscripten-based platform will either need their own tag of let Pyodide control what they do if they want to reuse the tag

Agreed. I think this is unavoidable in any case.

1 Like
64

I spoke with Hood about this at PyCon. I’d love to see this move forward. The externally hosted definition of the platform is linked from the PEP, and there’s a PR waiting in the packaging repo. cibuildwheel’s been able to build these for a long time now.

One thing I think helps is considering the future of the platform. Having a tag for an implementation is common; PyPy and GraalPy are some examples. However, the interesting thing here is that if CPython support progresses to Tier 2 (it’s not even back up to tier 3 yet, this is forward thinking), then CPython will want to have a defined Emscripten platform tag (likely identical to the pyodide one). I think that’s completely fine; either the source of the pyodide tag definition moves to CPython, or a new tag can be introduced - if they become synonyms, that’s similar to what happened with manylinux_2*.

I think this comparison between manylinux and emscripten is a good one, actually, though they are different. A “linux” wheel works on just the system it was generated on, much like an emscripten wheel. Tagging it with manylinux means it is ensured to work on a larger set of Linuxes, while tagging an emscripten wheel with pyodide means it was generated with a specific emscripten version and set of ABI flags to work on Pyodide or any build of Python with matching flags. From a distribution standpoint, this is nearly identical; you have a way to describe specific wheels you build locally and only work on a specific system, and a way to describe wheels that are suitable for distribution and work with an ecosystem.

65

What are the next steps here?

66

We are waiting on @dstufft to decide on whether or not to approve the PEP. Unfortunately, there is no timeline for when he will read it and as far as I understand there is no alternative for us other than waiting.

67

Did I miss the post where the PEP was formally submitted for pronouncement? As far as I can see, the discussion just died down without that.

68

Oh. Nobody who I’ve asked about it said I need to make such a post. How do I do that?

69

Just post the request here, and @-mention Donald.

You need to request approval on any PEP when it’s ready - the PEP sponsor should be able to tell you that (the sponsor is supposed to guide you through any parts of the process you need help with). In this case, it’s a packaging PEP, so the normal process (an issue on the SC tracker) isn’t used, we just make a request here, as I said. That’s not well documented (the packaging workflows are a bit less formal than the core ones) but if in doubt, just ask :slightly_smiling_face:

1 Like
70

Thanks for explaining @pf_moore! In that case, I formally submit this PEP cc @dstufft.

5 Likes
71

Hello,

I appreciate the effort behind this PEP and share the goal of establishing a standardized wheel format for Emscripten-based Python distributions. However, I would like to propose an adjustment to ensure the format is inclusive and adaptable to a broader range of use cases, including our own.

Context

We are actively developing emscripten-forge, a general-purpose software distribution for the web browser. Unlike Pyodide, emscripten-forge is not limited to the Python ecosystem. It also supports R, GNU Octave, C++ libraries, and command-line utilities. Our aim is to ensure that the CPython build included in our distribution is compatible with the proposed wheel format.

The naming of the wheel format

The current proposal ties the wheel format explicitly to Pyodide (e.g., pyodide_${YEAR}_${PATCH}_wasm32). While distributions like emscripten-forge can use the same libraries and build flags to maintain ABI compatibility, the name suggests a dependency on Pyodide-specific features beyond just the ABI.

For example, WebAssembly-based software often needs to access external resources, such as making HTTP requests. Pyodide addresses this with a Python-JavaScript bridge based on Emscripten-embind, exposing browser APIs like fetch to Python. Emscripten-forge uses a similar bridge, pyjs, which combines Embind with Pybind11. The APIs of pyjs and pyodide.ffi are functionally similar, but not identical. Fortunately, packages like urllib3 work seamlessly with both bridges (by adding a mock pyodide package), but this compatibility is not guaranteed for all packages if the formats diverge.

Proposal

To avoid fragmentation and ensure broad compatibility, we propose:

  1. Renaming the wheel format to emscripten_${YEAR}_${PATCH}_wasm32 to reflect its broader applicability.

  2. Defining the ABI independently of the Pyodide distribution, in a neutral repository related to Python packaging. This would allow Pyodide, emscripten-forge, and other stakeholders to contribute as peers.

  3. Establishing a common denominator for APIs (e.g., pyodide.ffi and pyjs) that package authors can rely on, reducing the need for downstream patches. This could be a subset of the Pyodide API, that would be importable under a name that does not contain the string “pyodide”.

This approach would foster collaboration, minimize fragmentation, and ensure that the wheel format serves the entire Emscripten-based ecosystem.

Benefits

  • Inclusivity: The format would not be tied to a single distribution.

  • Compatibility: Packages like urllib3 would work across distributions without modification.

  • Future-proofing: A shared specification would encourage consistency and reduce maintenance burdens for package authors.

I believe these changes would strengthen the PEP and benefit the entire community. We would be happy to contribute to this effort, work with @hoodmane, and help define the common ABI and API standards.

9 Likes
72

I don’t know too much about this space other than that I recently became a maintainer of msgspec and that we have to support this target to match the support matrix of our users. I was pleasantly surprised to find out that builds work out-of-the-box for me locally (in a Linux container) and that there has been support in cibuildwheel for a while now. The main issue is that users have no easy way to depend on libraries because PyPI lacks support due to there being no official standard.

If technically possible, I would love for the suggestions in the previous comment to be incorporated into the PEP.

2 Likes
73

I don’t t think naming it pyodide precludes using it for emscripten-forge. The name pyodide comes from the fact it’s being driven by the pyodide distribution, which has 100+ packages that are also taken into consideration choosing the correct flags and best version. That doesn’t mean Emscripten-forge can’t use it or even collaborate with Pyodide in helping define the best flags.

I guess if there’s another name that would be more generic, but not something CPython would define, that might be an alternative? manyemscripten? sharedemscripten? That might make it easier to move the definition in the future.

On your points:

  1. There is more to the name than just the Emscripten year and version, it’s also a set of flags. I think that should be implied by the name, similar to the “many” part of manylinux. You have to look up manylinux to see what that entails. pyodide does that, as it tells you where to look to find the definition of the flags. I think the native tag is already emscripten_something? I think a native unshareable tag (like linux vs. manylinux) is the correct way to do this.
  2. This could be done, but it would be a lot of work, and you’d want to build a collection of extensions, which pretty much just becomes pyodide again. I think it would be better to contribute needed flags to pyodide, then use that as the source of this wheel tag. If CPython raises the support tier eventually, it possibly could be tied with the build CPython would produce, but that would also make building extensions harder (since things like NumPy only start supporting CPython in the beta phase or even later).
  3. Hood has been working on this common denominator for APIs, see the language summit report or most recent development summit. That’s very much a work in progress.

I think it would be great for Emscripten forge to support the pyodide tag. In order to not block this on a proposal for a lot of work that I don’t think is likely to happen soon, I think a good way forward would be to compare empscripten-forge’s build flags and environment with Pyodide’s, and see if it would be possible to keep them in sync.

This leaves the door open for an “official” tag if CPython raises the support level in the future, but as I said earlier, that would be quite difficult to come up with an optimal flag setup - Emscripten-forge would have less say in the flags used, I believe, since the tags would have to be defined much earlier by CPython.

2 Likes