This is a discussion thread on PEP 783. The rendered pep is here. Please let me know what you think.
1 Like
@pf_moore I would appreciate your comments when you have a chance. The main changes are:
- split out from PEP 776
- an exhaustive list of what decisions go into the ABI
- a reference to a new page in Pyodide’s documentation which specifies the ABI:
Pyodide Platform ABI
I don’t really have much to say. The only thing that confuses me is, if pyodide ties the platform tag to the Python version, why have separate tags? For instance, if (as the PEP says) pyodide_2025_0 is used for Python 3.13, what’s the point of allowing for pyodide_2025_1, given that there’s only one Python version released per year? Actually, why is 3.13 using a 2025 tag, given that 3.13 was released in 2024?
I don’t think the above comment actually matters in practice, as the rules for calculating tags are explicit (and easily translated for non-Python implementations like uv). I just found it confusing.
I’ll let tool authors (especially build backend authors) comment on the PEP beyond this.
if pyodide ties the platform tag to the Python version, why have separate tags?
If you are suggesting to just call them all pyodide, it would work for something like numpy-2.0.2-cp312-cp312-pyodide_wasm32.whl but not for abi3 wheels where the platform is important e.g., cryptography-44.0.2-cp37-abi3-pyodide_2025_0_wasm32.whl. It would be an option to reject abi3 wheels though.
On the flipside, once things settle down upstream, this leaves us the possibility of using the same ABI for more than one year.
what’s the point of allowing for
pyodide_2025_1, given that there’s only one Python version released per year?
The thought here is that it provides an emergency escape valve if there is a completely intolerable toolchain issue that makes building prohibitive for downstream users, or a major bug with the runtime implementation of the ABI. Of course we would strongly prioritize fixing the toolchains/runtimes over breaking the ABI. Nothing like this has happened to us yet but the ecosystem is sufficiently immature that we worry about it.
why is 3.13 using a 2025 tag, given that 3.13 was released in 2024?
Currently we are waiting 6 months after the release of a new Python version to bring it into Pyodide. So the Pyodide stable branch is still using Python 3.12, whereas the main branch is now on 3.13. We do this because we also build and test 265 packages in tree, and even now a couple of them don’t yet build on Python 3.13.
This of course has significant downsides, and we are working on separating our package builds from our runtime so we can switch to releasing them separately. We had just had a discussion about this which I’ve put here.
2 Likes
especially build backend authors
I guess I’ll try summoning some build backend authors directly.
scikit-build-core - @henryiii
pyo3 - @messense @davidhewitt
meson-python - @rgommers @FFY00
setuptools - @jaraco @dstufft
I would very much appreciate any comments any of you have on the pep.
setuptools also - @abravalheri
Currently pyodide ties a particular version of each Python library to a particular version of pyodide. If the expectation is that pyodide will use wheels from PyPI then this opens the question of how to select a version of the library and also what libraries should do about multiple pyodide versions. Is the expectation that libraries would upload multiple wheels for pyodide_2025_0, pyodide_2025_1, pyodide_2026_0 etc?
What should an installer do if the latest version of a PyPI package does not have an appropriate wheel for the given pyodide version? The current (often problematic) behaviour of pip is that it would try to build from the sdist rather than checking if older versions of the package have wheels (--prefer-binary). Building from sdist won’t work in pyodide I think because all extension modules are cross-built which is a bit different from other Python platforms.
The PEP refers generically to “package installers” but it is not clear whether there is any expectation that existing widely used installers such as pip would ever be used in pyodide. I assume that they are not expected to be used there since e.g. pip and uv only have CLI interfaces and pyodide cannot run CLI things (no subprocesses). Otherwise is this just referring to pyodide’s own version of pip, micropip etc?
1 Like
If this isn’t impacting pip and uv, or “general installers” in the broader sense, then how is it an interoperability standard? If the bulk of the packaging workflow is handled by custom tools, what’s the advantage of locking everything into a standard? Sorry if this is a dumb question - I have no idea what micropip even is, and maybe my ignorance is showing 
Is this purely about being able to publish wheels on PyPI (and maybe other indexes like Artifactory or devpi)?
1 Like
this opens the question of how to select a version of the library
Well if you use pip or uv or micropip as your installer, they have logic to do this. If you install from one of Pyodide’s lock files, Pyodide will always install the version present in the lock file.
Yes, though to be clear, it is our goal never to create a 2025_1 ABI. Any package that doesn’t do abi3 builds already has to upload one wheel per Python version, so it’s no worse for them. The only packages where this will be annoying is ones that use abi3.
What should an installer do if the latest version of a PyPI package does not have an appropriate wheel for the given pyodide version?
It should fail. In Pyodide virtual environments we add a pip.conf file with:
[install]
only-binary=:all:
If you remove this, pip will not be able to successfully build a binary wheel right now anyways. I think it’s unlikely that pip will want to support building for Pyodide in the future either.
The PEP refers generically to “package installers” but it is not clear whether there is any expectation that existing widely used installers such as pip would ever be used in pyodide.
We are already using pip with Pyodide.
It’s primarily about that yes. We can patch/configure pip to behave the way we want without much trouble but we have no way to deal with uploads. It appears that some people are just lying about the platform of their wheels to say that they are py3-none-any, but of course we would never encourage people to do this.
Pip is completely agnostic over the matter of building for any given platform. It simply calls the build backend. Whether the build works is down to the build backend (which may require particular tools, like a C compiler, or an emscripten toolchain, to be present if it’s to succeed). Adding only-binary=:all: will prevent (for example) installing of pure Python packages that are shipped in source form only. That may be acceptable to you, but it’s not something that a standard can assume.
Just to be explicit here, I’m not against emscripten support, but I do think that trying to integrate two very different models of operation is probably a lot bigger task than just defining a packaging tag. So if all we are doing is defining a tag, we need to be very clear what that’s intended to achieve, and carefully scope the proposal to exclude problems we’re not expecting to solve.
1 Like
So if all we are doing is defining a tag, we need to be very clear what that’s intended to achieve, and carefully scope the proposal to exclude problems we’re not expecting to solve.
If we successfully define the platform tag and allow PyPI uploads, I would be very happy with that outcome. It is my main goal. In the long term, it might be nice to go further than that. Let’s discuss the other problems a bit and if we think we can’t/shouldn’t try to solve them now I am happy to declare them out of scope.
Pip is completely agnostic over the matter of building for any given platform.
What’s weird about Pyodide is that we use existing build backends but have a custom build front end that takes care of cross builds. In particular, what needs to be done differently is hacking the sysconfigdata and platform information so that they report information about Emscripten and installation of cross-build dependencies. For example, if the package reports a dependency on numpy, we install a manylinux numpy and then replace the headers and static libraries with the headers and libraries from the numpy-emscripten wheel. We also put custom compiler wrappers on the path.
The build backend then can be completely unaware that it is building for Emscripten. Though some of the newer build backends use build systems like meson, cmake, or cargo that are capable of cross-builds and are choosing to explicitly support Emscripten.
These questions of “how should build systems approach cross-builds” and “how should installers approach cross-builds” will benefit from a joint effort with Android and iOS. @freakboy3742 and I are hoping to collaborate on this.
Just chiming in to confirm that yes - this is a topic that I’m interested in, and I’m hoping to have a coherent proposal (or at least a summary of historical proposals and possible approaches) before PyCon US.
Can you clarify what needs to be done to define a tag and which problems you mean to exclude? I think defining a tag entails saying:
- What is the Pyodide platform and how would someone build a wheel compatible with it?
- How should an installer decide if a wheel is compatible with a target Pyodide platform?
I think I answered these things in the current pep text. And you seem to mean we should exclude:
- What should
pip/some other installer do if someone asks it to install an Emscripten wheel from source?
I will update the PEP should be updated leave this unspecified and explicitly suggest that an acceptable and likely behavior is to fail in some way. We can furthermore recommend but not require that people always use only-binary=:all:.
Are there other things I’m missing here?
A minor technicality, but you don’t install a wheel from source. My point is that you install a sdist, and you do that by asking the build backend to generate a wheel. Whether the backend generates an emscripten-specific wheel or not is entirely down to the backend, and the frontend (pip, or whatever) has to be prepared for that. I just checked PEP 517, and there’s actually not even a requirement that I can see that the wheel generated by build_wheel is compatible with the target environment! But I think it’s fair to say that most frontends would expect to either get a compatible wheel or a failure.
I’m a strong -1 on having any suggestion of only-binary=:all: in the PEP. Apart from the fact that it’s not a standardised option, there are all sorts of edge cases that make such an approach incredibly fragile. I’ve already mentioned pure-python sdists, and there’s also cases like --target where the frontend could be building an environment for a platform other than the current one.
You also need to address @oscarbenjamin’s point - what if the latest version of a PyPI package doesn’t have a wheel for the required pyodide version? Is it OK to backtrack through all older versions looking for one? Are projects expected to ship wheels for all pyodide versions? The manylinux spec avoids this by having systems be compatible with all older manylinux versions, as well as the latest version. That’s not how pyodide tags work, if I understand things correctly.
Yes, subject of course to the constraints of the request. I think this is all completely normal?
Yes the ABI tags are not backwards compatible. So far, Pyodide only allows people to use a specific newish version of each binary package. Very few people complain about this.
It is up to the individual project what sort of support they want to offer, but I don’t expect them to support very old Pyodide versions. It would be reasonable for them to only support the most recent one or two Python versions. I think cibuildwheel intends to support all Pyodide ABIs that are not for EOL Python versions.
But is that normal pip or some patched version of pip? If it is a patched version is there an expectation that the patches would ever be upstreamed?
Let me put the question a slightly different way: if some customisation of pip or of how packages are installed is needed to handle the pyodide case then who is generally expected to implement and maintain that customisation? Is it the pyodide maintainers (e.g. you?) or pip maintainers (e.g. @pf_moore)?
The current default behaviour of pip is that it would download the sdist for the latest version and try to build it. Various options exist such as --prefer-binary, --only-binary etc to control this but those options are all off by default. Lengthy discussions have considered changing the default to e.g. --prefer-binary but it is known that any change would break some things.
1 Like
But is that normal
pipor some patched version ofpip?
Patched. venv/bin/pip is changed to look like this.
If it is a patched version is there an expectation that the patches would ever be upstreamed?
Not specified by this PEP. My hope is that @freakboy3742 and I can figure out a solution that handles other cross-build-only platforms as well.
who is generally expected to implement and maintain that customisation? Is it the pyodide maintainers (e.g. you?) or pip maintainers (e.g. @pf_moore)?
Pyodide maintainers.
The current default behaviour of pip is that it would download the sdist for the latest version and try to build it.
And this will definitely break.
Just want to note that this already is working in practice. You can use scikit-build-core to build wheels for Pyodide; several packages do this, like boost-histogram and awkward-cpp. They build the wheels using cibuildwheel and then set up a documentation site with live versions of the current packages. The only problem is those wheels can’t be uploaded to PyPI, and instead must be manually placed in a folder (which is what the CI does). cibuildwheel has supported Pyodide for quite a while now.
It would be a really bad for package maintainers if there were more than one platform for a single Python version, but I think leaving the possibility in case of an emergency is very reasonable. It’s unfortunate that ABI3 isn’t really useful, but it’s pretty much what almost every package has to deal with today; whenever a new Python version comes out, you need to upload new wheels. (Even ABI3 packages are needing to upload 3.13t wheels now!) It would be really nice if the Pyodide release (the flags + emscripten version) was synchronized to CPython, so that a package could release 3.14 and pyodide 3.14 wheels at the same time. The only reason that’s an issue today (the 6 month delay) is due to the hundreds packages that are also built in every Pyodide build; having a standard wheel tag on PyPI would help reduce that, and I think maybe the model could be changed to have a core Pyodode (just CPython for Emscripten) and then a collection of package builds to cover missing/core PyPI wheels (Raspberry pi’s piwheels is a similar model, I believe).
I’d love to see better cross-compiling support, besides Wasm/iOS/Android, cross-compiling Windows ARM, Linux archs, and even macOS might benefit.
1 Like