PEP 776: Emscripten Support

PEPs
1

I’ve written a draft of a PEP to formalize Emscripten support. I am interested in feedback.
It is loosely based on PEP 730 and PEP 738 but we’re in a slightly different situation since the steering council has already approved tier 3 support for Emscripten and our packaging ecosystem is a bit more mature.

The main goals are:

  • to have a reference describing all the most important Emscripten-specific runtime details
  • describe what Pyodide and what cpython on Emscripten do now
  • describe some minor intended improvements for the cpython Emscripten runtime
  • describe the packaging ecosystem
  • request that Pyodide wheels can be uploaded to pypi.

All the proposed runtime changes that I believe could be controversial are listed as out of scope. The proposed changes that I do have are all of a similar scale to features that I have contributed in the past with only a github issue. I think the most important questions here are the packaging questions. Hence why I’ve listed it as a packing PEP.

Here is the current PEP text.

7 Likes
2

It may be slightly tangential to your PEP, but based on your section on ABI stability, am I right in thinking the Stable ABI isn’t a thing on Emscripten?

(Just curiosity mainly)

3

Nice work @hoodmane !

For full disclosure - I’ve been working with Hood to land the Emscripten patches; we’re maybe 1 or two commits away from having a buildbot running. I’ve also seen an earlier draft of this PEP.

The one high-level concern with the PEP is the intermingling of Pyodide and CPython at the project level that is implied by some of the wording.

Don’t get me wrong - the Pyodide folk (including, but not limited to @hoodmane) have done amazing work with Pyodide. Almost all (if not all?) the Emscripten improvements that have landed to date in CPython have their genesis in Pyodide, and I imagine that will continue into the future. Maintaining compatibility between the two projects going forward is also a highly desirable goal.

However, in the long term, it seems undesirable for CPython to directly tie itself to specifications that are downstream of CPython. As an example, Pyodide provides a very good basis for an ABI specification for wheel compatibility. However, if CPython adopts that ABI specification in the form of a wheel tag, I feel it should be as independent, CPython-controlled standard, with it’s own identity (i.e., not using pyodide_X as a wheel identifier).

Essentially, I’m wary of CPython being bound to following a Python standard that CPython isn’t in control of. Pyodide isn’t obligated to follow any of CPython’s governance procedures, and while I doubt it would be done with malice, the possibility would exist for Pyodide to publish an update to their ABI specification that would be incompatible with CPython’s support policies. At that point, the ecosystem would need to make a decision over which version of the “Pyodide” standard to follow. It seems safer for all concerned to avoid even the appearance of that sort of dependency - even if, in practice, the actual standard that is adopted is “what Pyodide says”.

3 Likes
4

in the long term, it seems undesirable for CPython to directly tie itself to specifications that are downstream of CPython
I feel it should be as independent, CPython-controlled standard

Strictly speaking, CPython doesn’t control the packaging ecosystem. The code for pip, wheelhouse, manylinux, wheel, build, setuptools, etc all live in the PyPA organization not in the Python organization. It is more an issue of whether the packaging people wish to allow Pyodide to determine a format for wheels to be uploaded to PyPI. If I go on PyPI, I can find wheels with tags like pp310-pypy310_pp73. CPython has significantly less control over PyPI than over Pyodide. So there is clear precedent for what I am proposing here.

The current situation is that the Pyodide runtime has a very large number of features that are not present in CPython and are necessary for the platform to work well. We would like to upstream these into CPython runtime, and we are open to transferring control of the ABI to an organization that is bound tighter to the Python ecosystem than Pyodide. However, I think it would most likely take until Python 3.16 to complete that. In the meantime, we have a large number of people building and using Pyodide wheels for Python 3.12 and they cannot distribute them on PyPI.

1 Like
5

To be pedantic, it is a thing, it just isn’t a useful thing. The stable ABI is a stable ABI for the libpython. Emscripten lacks a stable ABI for libc. So if you have an abi3 wheel with the platform pyodide_2024, in practice the only version of Python that anyone is building with the pyodide_2024 platform is Python 3.12. But it is possible to build a Python 3.13 interpreter compatible with the pyodide_2024 and use the wheels tagged py312_abi3_pyodide_2024 with it.

1 Like
6

It sounds to me like there should be two PEPs here. One for core support, and a second one for packaging. That’s awkward, I know, but it’s an inevitable consequence of the “hands off” approach the core devs take with packaging.

7

It sounds to me like there should be two PEPs here

This makes sense to me. This comes out in the “goals” section, which is a bit unspecific. If divided into two PEPs, the core support PEP would be without any requests that haven’t been already approved by the steering council and should hopefully be completely uncontroversial. I’d be interested to hear more opinions on the “should this be two peps” question.

8

Actually, having just skimmed the PEP, I noticed that you have marked it as a “Packaging” PEP. As such, I’d likely be the PEP delegate for it, and I can certainly say that I would not be the right person to make a decision on whether emscripten should be moved to tier 3 support - that’s a core decision. On the other hand, the structure of the wheel tags for emscripten very much is a packaging topic, and the SC would almost certainly want to delegate that (to me, by default, as it’s a packaging interoperability matter).

However, I’m not clear if you expect standard packaging tools to work on emscripten - will people use pip to install packages (you say that ensurepip is omitted from emscripten builds, which suggests not), or standard build backends like hatch or setuptools to generate emscripten-compatible wheels? Given the large number of missing stdlib modules, will “generic” .py3 wheels be expected to work on emscripten?

Also, you say that the goal is for emscripten builds to be available on PyPI. How will that happen? Do you expect projects like numpy or cryptography to take on the task of building emscripten wheels? Or do you expect to have separate projects, emscripten-numpy, etc.? I wonder whether the piwheels model might actually be a better model for emscripten packages, for now at least. It’s not clear from the PEP why “emscripten wheels can be uploaded to PyPI” is important at this point, if the emscripten wheels aren’t (yet?) being built by the projects themselves.

9

Thanks for your questions @pf_moore.

I can certainly say that I would not be the right person to make a decision on whether emscripten should be moved to tier 3 support

Fortunately, that is not a matter for discussion in this pep, since it was already decided in October.

I’m not clear if you expect standard packaging tools to work on emscripten - will people use pip to install packages

Pip did not used to work in Emscripten because of lacking requests/urllib3 support. Now that urllib3 supports Emscripten, there are no major blockers for making pip work. But it’s not clear how desirable it is. Instead, pyodide provides an Emscripten cross-venv that install wheels with a slightly patched native pip. The patches are here. We also make a PEP 503 simple index and add it as an extra-index-url in pip.conf to allow platformed wheels to be installed from jsdelivr. Pip cannot build wheels for Pyodide because building wheels for Pyodide requires our custom pyodide build build frontend.

Currently this is all set up for Node. It’s intended to allow packages to be built and tested with Pyodide with as few changes as possible to their linux CI job. It also is used by cibuildwheel to build packages for Pyodide.

It would make a lot of sense to have tooling to bundle a pyodide venv for deployment in the browser. Currently nobody maintains such tooling, but one can do it with some manual effort.

Do you expect projects like numpy or cryptography to take on the task of building emscripten wheels?

Yes. Numpy is already doing this as well as about 60 other packages including pydantic, pandas, awkward-cpp, scikit-image, statsmodels, pyarrow, and hypothesis. Packages can use cibuildwheel, or they can use a github actions job that invokes pyodide build which can build wheels for many projects without extra effort.

Several build backends also have explicit support for Pyodide now, including scikit-build-core and maturin/setuptools-rust/pyo3.

2 Likes
10

There are currently two models for how this works. One is that pyodide works sort of like a rolling distribution of many packages listed here. Those wheels are built by the pyodide project itself.

The other model is that projects like NumPy build pyodide/emscripten/wasm wheels in their own CI. Those can be uploaded to e.g. the scientific python nightly wheels index e.g. if you look at numpy’s files then you can see:

numpy-2.3.0.dev0-cp312-cp312-pyodide_2024_0_wasm32.whl

That one is uploaded from numpy’s CI.

You can try this out at the SymPy Live Jupyterlite/pyodide shell. There if you just type import numpy it will (invisibly) install the version from the pyodide distribution. However if you refresh the page and instead do:

import micropip
await micropip.install('numpy', index_urls=['https://pypi.anaconda.org/scientific-python-nightly-wheels/simple'])

then it will install the version of NumPy from the nightly wheels index:

>>> numpy.__version__
'2.3.0.dev0+git20250318.0aafd85'

Either way you get to use NumPy etc in the browser.

What would be nicest for projects like NumPy that build wheels is if pyodide can be treated just like another platform/architecture combination to add to the wheel building CI matrix. Then you can use the same tooling (e.g. cibuildwheel) and upload to the same places (e.g. PyPI).

1 Like
11

Thanks for the clarifications. I’ll focus solelly on the packaging side of the PEP, which as far as I can see is essentially the Packaging section. I suggest that you frame this similarly to the manylinux and musllinux PEPs, as they are the closest equivalent we’ve had previously.

The “Packaging goals” seem reasonable, although I’m not sure describing the pyodide tooling is necessary here - that should go in the pyodide documentation, and simply be referenced in the PEP if and when relevant.

The discussion of the wheel format needs clarifying. “Emscripten wheels will use either the format emscripten_<version>_wasm32 or pyodide_<abi>_wasm32” - I assume you mean that these are platform tags, in which case you should say so, and probably clarify what ABI tag wheels should use (the ABI tags we have are very CPython-based, so may well not be applicable for emscripten). You also need to explain how to determine the supported tags for a given interpreter. You might also want to review the implementation of the manylinux and musllinux tags in packaging (here and here) to get a feel for how tags are implemented in practice.

You should probably also consider dependency specifiers (and in particular markers). If a library, written in pure Python, wants to use 3rd party code to replace a stdlib module that’s not supported in emscripten, they will almost certainly need to frame that dependency as “only needed on emscripten”. There’s a sys_platform tag, and other tags that explose values from the platform module, so this shouldn’t be an issue, but there’s no markers that expose sys._emscripten_info, so if that might be relevant in choosing dependencies, you’d need to consider adding extra markers.

You need to consider the backward compatibility implications on the packaging ecosystem. I think it’s fine - tools will simply ignore wheels with an unknown package tag - but it’s something that should be covered explicitly. New markers might be an issue, though, as older tools will fail if they encounter markers they don’t know about.

You may also want to consider adding a new classifier (which would need to be requested) to declare support (or not) for emscripten, which could be used by pure Python libraries.

There’s also the “How to teach this” section. I’m never clear on how much depth this section ought to go into - when reviewing PEPs I tend to rely on my own questions. With this PEP, I’d be asking:

  • How do I adapt an existing build process to build emscripten wheels? Or at least, try to do so - I may be OK with building emscripten wheels if it’s straightforward, but not want to get deeply into the emscripten ecosystem.
  • How do I run my tests against emscripten? Particularly for a pure Python project, where I’d by default assume my code works “everywhere”.
  • Assuming I don’t want to get sucked into fully supporting emscripten, what do I do if my tests fail?
  • And for that matter, what do I do if my tests succeed? (This is where “Add a classifier saying the code works on emscripten” comes in).

The “How to teach this” section isn’t the right place to answer these questions, but it should discuss how we’ll ensure that people with such questions can find answers.

1 Like
12

This is also an area where structuring the PEP(s) will be critical. The ABI tag is provided by the runtime (sys.abiflags with some processing), while the platform tag is entirely independent from the runtime. Which means the former needs a core PEP because it will go into the core codebase, while the latter needs a packaging PEP because it matters only to packaging.

(An additional implication is that the ABI tag affects the extension file suffix, which may be irrelevant here but is generally of importance to the core runtime since the standard importer needs to know what filenames to look for, while the platform tag affects the wheel file name, which the core runtime never needs or uses. Hence, they are separate concepts under separate management.)

13

Indeed those are the platform tags. So for Python 3.12, we accept (in addition to non-platformed wheels) cp312-cp312-emscripten_3_1_58_wasm32 and cp312-cp312-pyodide_2024_0_wasm32 as well was things like abi3-cp9-pyodide_2024_wasm32. The full patch to packaging needed for this is here, and just requires teaching packaging about the pyodide_ver_wasm32 tags. I will update the PEP text to clarify this.

You should probably also consider dependency specifiers (and in particular markers).

People are already checking whether sys_platform is 'emscripten' in the wild, for instance here

I think sys._emscripten_info is not needed as a marker. At least I have never seen a use case for checking against it.

You need to consider the backward compatibility implications on the packaging ecosystem. I think it’s fine - tools will simply ignore wheels with an unknown package tag - but it’s something that should be covered explicitly. New markers might be an issue, though, as older tools will fail if they encounter markers they don’t know about.

I think this should be okay then since we don’t need any new markers.

You may also want to consider adding a new classifier

There already is a Classifier: Environment :: WebAssembly :: Emscripten.

“How to teach this” section

This sounds like a good set of questions. The best resource we have to answer these questions right now is this page. Does that look good enough to you?

14

The ABI tag is provided by the runtime (sys.abiflags with some processing), while the platform tag is entirely independent from the runtime. Which means the former needs a core PEP because it will go into the core codebase, while the latter needs a packaging PEP because it matters only to packaging.

As far as I can tell we don’t need any change to sys.abiflags or the ABI tag, whereas we do want changes to the platform tag. Hence why I’ve labeled this as a packaging pep.

An additional implication is that the ABI tag affects the extension file suffix, which may be irrelevant here but is generally of importance to the core runtime since the standard importer needs to know what filenames to look for

The Emscripten shared libraries have the suffix .cpython-314-wasm32-emscripten.so. The Emscripten Python built from the main branch of the cpython repo now says:

$ ./python.sh -c 'import sysconfig; print(sysconfig.get_config_var("SOABI"))'
cpython-314-wasm32-emscripten

so I think no change is required to the runtime for this either.

1 Like
15

Yeah, that looks great. So “How to teach this” can probably simply say something like “The Pyodide website contains information on how to support pyodide in your packages”.

The rest of your comments look good, too.

16

Okay I opened a PR which I addressed all of your specific comments @pf_moore. It will take a bit longer to see what changes are needed to address this suggestion:

1 Like
17

Not really part of the PEP process but I think that the documentation can be improved if the intention is that Python package maintainers will try to support pyodide as a platform. I found it hard to figure out how to use or test pyodide being someone with no background in web, JS etc and also not a clear understanding of what pyodide even is. I was only able to get things working with direct help from others who were more knowledgeable about pyodide and web things.

The page that you have linked there has much more useful information for me than e.g. the getting started page but is not one that I would necessarily select from the main page. It could be improved by being more prominently linked but also by separating it into one part that is just about running pyodide/venv/pip and another that is about using emscripten. Pure Python package authors can follow the instructions there without emscripten. Those who do need emscripten likely need more information like how to build non-Python dependencies so at least the emscripten docs should be linked.

I just tried following the instructions at the linked page and the first hurdle was that I started out installing pyodide-build into a CPython 3.13 venv when it should be 3.12 right now. If you use 3.13 then the steps fail a few steps later with a cryptic error.

Otherwise having followed the instructions with 3.12 I can setup a venv and run the pyodide python:

pip install pyodide-build
pyodide venv .venv_pyodide
source .venv_pyodide/bin/activate
python

Instructions for doing this should be made more prominent in general. The getting started page is all about running pyodide from JS which is understandable if the target is someone who wants to make a website using pyodide but this way is much more understable/usable for non-web Python package authors even if it is not really the way that pyodide is actually expected to be used.

The instructions indicate that I should be able to use pip to install pure Python packages or pyodide wheels. In my venv pip does not work though:

$ pip
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '.../.venv_pyodide/bin/python3.12-host'
  isolated = 0
  environment = 1
  user site = 0
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/oscar/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '.../.venv_pyodide/bin/python3.12-host'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007a79e7035740 (most recent call first):
  <no Python frame>
1 Like
18

Thanks for the feedback @oscarbenjamin. I agree that a lot more work needs to be done to make this easier to use for Python people. Pyodide’s documentation is written more for JavaScript developers, and we seem to get more documentation patches from JavaScript developers. It also has the problem that it is written by experts who don’t really empathize with new users.

If you use 3.13 then the steps fail a few steps later with a cryptic error.

This is a known problem, but I think we may have resolved all of the blockers for fixing it. I’ll look into it and see if I can’t make pyodide venv fail with a better error message.

EDIT: I made this PR which hopefully fixes this.

this way is much more understable/usable for non-web Python package authors even if it is not really the way that pyodide is actually expected to be used.

Well it is a major use case. I’ll see if I can add some text to the readme / home page directing Python package maintainers to that page.

In my venv pip does not work though

Can you provide a reproducer? I wonder if there is a problem related to using uv Python.

1 Like
19

That would be good. It could also be improved in the docs just by saying “use an appropriate Python version. See [here] for versions”.

It might be a uv problem. I just tested this as a reproducer on Ubuntu 22.04 with uv installed:

uv python install 3.12
uv venv -p 3.12 .venv
source .venv/bin/activate
uv pip install pip
pip install pyodide-build
pyodide venv .venv_pyodide
source .venv_pyodide/bin/activate
pip # error

We should discuss this elsewhere though. Where would be a good place to open an issue?

I just wanted to note here in this PEP discussion that if we are going to say that the pyodide docs will explain how Python package authors can support pyodide as a platform then it would be good to improve the docs with a focus on providing information for those developers.

It is of course understandable that the pyodide docs focus on JS developers since they are the “users” but maybe there can be a big signpost on the main page or README and perhaps links from some other pages to something that is explicitly aimed at Python package developers (“pyodide for Python developers”). Bear in mind that for example when I first looked at this I didn’t really even know what pyodide is.

1 Like
20

That would be good. It could also be improved in the docs just by saying “use an appropriate Python version. See [here] for versions”.

Ideally we should do both. PR to improve the error message is here: Better Python version mismatch errors by hoodmane · Pull Request #142 · pyodide/pyodide-build · GitHub

Where would be a good place to open an issue?

The relevant spot is here.

the pyodide docs focus on JS developers since they are the “users” but maybe there can be a big signpost on the main page or README and perhaps links from some other pages to something that is explicitly aimed at Python package developers

Agreed, we really want to provide information for both audiences. Here’s a start in that direction.