If this is going to be a year, matching the CPython year of release does seem a little nicer. This +1 sort of promises that there will be at least a several month delay. I think it’s fine to just say the 3.13-targeting release is 2024_* even if it wasn’t defined till 2025. Same for a patch release, if you needed to make a 2024_1 that’s fine even if the year is 2025. Using years for version numbers doesn’t require the version “year” matches the actual year of release.
This is why I’m finding this conversation difficult. I don’t want to block something that’s working, and just requires a simple step to improve it. But equally, I don’t want our standards to have some sort of implied dependency on a bunch of customisations and special cases that aren’t standardised, and could change without notice.
So that’s not supported by pip, and we could break it without meaning to with no notice. I assume you know that and it’s fine, but if “normal” pip doesn’t work with pyodide, and normal uv doesn’t either, then a PEP that suggests that “installers” (in the general sense) should add support for emscripten wheels is misleading, because adding the support described in the PEP won’t mean that the installer now works for pyodide…
So again, the normal resolution process isn’t suitable for pyodide, meaning that a standard which suggests that all installers need to do is support the tag is misleading.
Let me ask a different question, as a pip maintainer. Given that (normal) pip doesn’t work on pyodide, there’s no need for us to add support for emscripten wheels. So it’s entirely reasonable for us to decide not to support PEP 783[1]. Where would that leave your patched copy of pip? Given that you’re already using pip in an unsupported way, you’re out of luck reporting problems as bugs in pip.
As things stand, I’m not happy with PEP 783 putting requirements on “installers” that won’t apply to uv, and which will only apply to pip in the sense that they are needed to keep an unsupported use of pip working.
If we take a “practicality beats purity” stance, the PEP is probably fine, and all of this is a distraction. But in the past when we’ve done that, we’ve ended up in a mess because unanticipated edge cases turn out to be important, and we can’t fix them without significant compatibility issues. The situation with emscripten (and iOS) support is that I don’t think the overall platform, workflow, and environmental support is stable[2] enough that we can judge how things might change in the future. And that concerns me if we’re writing standards right now that could need to serve us for 10 years or more.
I’m going to ask the same question I asked earlier - is there any reason pyodide couldn’t use a “piwheels” type of solution, and have a supplementary package index that hosts emscripten wheels independently of PyPI? If hosting really is the “only problem”, then that should work even though there’s no standard tag, or installer support. And it can be done now, and act as a working solution while we understand better how emscripten support affects the wider packaging ecosystem (if at all).
In fact, by saying “the only problem is that we can’t upload to PyPI”, you’re suggesting that pyodide doesn’t use the rest of the packaging ecosystem. So why insist that using PyPI is essential, when using pip/uv clearly isn’t? After all, PyPI is only special in that it’s the default index used by the standard installers…
Agreed, but that’s a much bigger question, and very much a distraction as far as this topic is concerned.
2 Likes
So that’s not supported by pip, and we could break it without meaning to with no notice. I assume you know that and it’s fine, but if “normal” pip doesn’t work with pyodide, and normal uv doesn’t either, then a PEP that suggests that “installers” (in the general sense) should add support for emscripten wheels is misleading, because adding the support described in the PEP won’t mean that the installer now works for pyodide…
That is a fair point. I can add the extra information about what installers need to do to the PEP:
- Installed executables need a different shebang than pip’s own shebang. Maybe we could set this with a
PIP_HOST_RUNNERvariable. - evaluation of markers needs to be done with
os.name,sys.platform,sys.platlibdir,sys.implementation._multiarch,platform.system,platform.machine,_PYTHON_HOST_PLATFORM, and_PYTHON_SYSCONFIGDATA_NAMEfor the target platform packaging.tags.platform_tagsneeds the patch from Add compatibility for Pyodide-tagged wheels by hoodmane · Pull Request #804 · pypa/packaging · GitHub as already described in the PEP.
I could look into implementing this as branches of pip and uv and linking them from the PEP. The first two changes will be generally required to support cross-venvs. The question from my perspective is whether pip and uv are willing to support these things. If it is, I am willing to write a spec for it and implement it.
I’m going to ask the same question I asked earlier - is there any reason pyodide couldn’t use a “piwheels” type of solution, and have a supplementary package index that hosts emscripten wheels independently of PyPI? If hosting really is the “only problem”, then that should work even though there’s no standard tag, or installer support. And it can be done now , and act as a working solution while we understand better how emscripten support affects the wider packaging ecosystem (if at all).
Yes, this would certainly work. I do not personally want to maintain a hosting solution for Emscripten wheels, but we can continue with what we are currently doing with jsdelivr until someone else to do that. It is already possible to upload wheels to anaconda.org so perhaps someone could make an official Pyodide solution based on that.
It is certainly possible to do this but I would prefer to put the wheels on PyPI. It is easier as a package maintainer to build the wheels and put them all in the same place using the same security setup etc. The way that pyodide is set up makes it feel just like another platform in the build matrix.
I’m not sure why piwheels goes with a different approach but it might be because there you are installing into a full Linux distribution that can likely build things from source. The purpose of piwheels is just to be a centralised cache for the builds because it is potentially slow to build on a raspberry pi.
I might have oversimplified. The thing that works is specifically building wheels for targeted applications.
Are we engaging with the emscripten-forge folks? If that could also download PyPI pyodide wheels, that would be an important benefit from standardization. I believe it started with Pyodide as a base, so this seems like it might be really easy.
As I see it, the key feature here is “here is the official platform for CPython 3.x on Emscripten” so that everyone building CPython with Emscripten (Pyodide, emscripten-forge, maybe CPython itself if it ever reaches Tier 2 support) can follow, and everyone producing wheels (cibuildwheel, etc) can follow to ensure any of those builds can be used. Someone doesn’t have to follow it for specialty applications, but they need to if they want compatibility.
I think this shouldn’t be tied too much to the details of how it could be implemented in tools like pip and uv. Cross-compiling would be great, but it’s well out of scope (sorry, might not have been clear above that I wan’t requesting it to be part of this PEP!). I think this can be focused as “here’s how a reusable Emscripten platform can be defined on a yearly basis, tied to Python version”, without focusing on the how or even if pip/uv can download packages. The built-in package managers for these distributions like micropip and piplite would be able to access and install packages. This PEP would enable work to be started on adding support in pip/uv, but IMO that’s not something the PEP needs to worry about, any more that the iOS/Android PEPs (which added iOS/Android tags!) needed to worry about it.
They had to go this way since even though the tag was standardized, there were no manylinux containers until recently, making it hard to build (cibuildwheel couldn’t support it originally). This was sort of the opposite case (tag but hard to build, vs. no tag but easy to build). I’d expect it to proceed similarly to the pyodide built-in package set, with more packages starting to upload armv7 wheels over time and the custom index being used less.
(1) sounds like something pip would’t accept (by definition - “installers must do something different than pip does” isn’t something pip can manage to do 
)
(2) is about cross-platform installs, and is a much bigger issue, which I wouldn’t be willing to support as part of this PEP.
(3) is just “packaging (as one of the key implementations of the compatibility tag spec) must implement this PEP”, so isn’t anything new.
Speaking for pip (but only as one maintainer, this is not official pip policy), no, we wouldn’t support “cross-venvs” (whatever those are) as part of a PEP to add emscripten support. It would need to be a separate PEP that covered all cases of cross-platform handling (e.g., creating a Linux environment on Windows). And frankly, I don’t personally have the appetite for tackling something that big at the moment.
I think we may be talking past each other here. I’m not asking you to add requirements on installers (pip and uv) to the PEP. Rather, I’m pointing out that the PEP as it stands, and particularly its motivation, doesn’t need any changes to pip and uv - not least because they don’t work (unmodified) on pyodide (and from what you’ve said, this isn’t currently considered a problem). I don’t want the scope of the PEP to be expanded to include “add pyodide support to pip and uv (and any other installers)” - rather I want the PEP to stick clearly to the goals it does aim to achieve.
My understanding is that the only actual goal of the PEP is “get emscripten wheels hosted on PyPI”. If that’s the case, my questions (as a possible PEP delegate, not as a pip maintainer) are:
- Why is hosting on PyPI necessary, rather than on (say) a separate index the way piwheels works?
- Is this only about PyPI, or do other index implementations like devpi and artifactory need to be included? Do those other index implementations currently prohibit emscripten wheels?
- What’s the standard that currently prohibits emscripten wheels being published on PyPI? Or is this just a rule that PyPI implement themselves, and if so, why isn’t a feature request to relax that rule enough?
Actually, if this is the only goal of the PEP, and changes to installers are out of scope, then I wouldn’t normally be the PEP-delegate. As delegate for package index interface PEPs, that would be @dstufft. But let’s work out what the actual scope of the PEP is before worrying too much about who will make the final decision.
2 Likes
Ah, OK. Your message arrived while I was writing my previous post, if the PEP is trying to solve more than just “we can’t upload to PyPI” that will change some things I was saying. Although I’m not sure the PEP is clear what problems it is trying to address in that case…
I have no idea. Nor do I really understand where they would fit in. As a “consumer” of tag information, I guess (although not an “installer” as such).
Agreed. That’s why I’m uncomfortable with comments around things that “installers must (or should) do” - those are precisely details about how pip/uv would provide pyodide support, and currently pip doesn’t support pyodide as a platform[1].
Apart from these specific comments, thanks for your clarification. I think the biggest issue here is likely that the PEP is written without a real understanding of just how little the target audience knows about pyodide (and writing a coherent proposal while managing to not include reams of “what is pyodide” tutorial material is a genuinely hard problem to solve).
I don’t know about uv, but I suspect they don’t either ↩︎
2 Likes
I think this is correct. I probably know more about pyodide than most people reading here and I still know very little. The succinct explanation in the PEP would be sufficient perhaps if a reasonable number of people had a good understanding already but that isn’t the case. I’m not suggesting that the PEP itself should have a long and detailed explanation of everything about pyodide but the PEP authors should understand that more explanation needs to be given somewhere (e.g. here in discourse).
Relatedly, the OP does not link to any previous discussions or any other context. This PEP was a spin out from another thread but there is no link to that thread. The OP could also have links to other sources of information like general background and so on.
I think the biggest issue here is likely that the PEP is written without a real understanding of just how little the target audience knows about pyodide
I can reread this thread and think about what sorts of context would help. I think a difficulty that we may have is that I am not sure know what sort of context non-Pyodide-expert readers need, and non-Pyodide-expert readers don’t know what they are missing.
This PEP was a spin out from another thread but there is no link to that thread.
There is in fact a link to that thread: peps/peps/pep-0783.rst at main · python/peps · GitHub
Well, speaking for myself:
- I don’t know how to run a pyodide copy of Python
- I don’t even know how to describe the build (is it an executable? does it have a repl?)
- I don’t know whether it runs in a command line context where “running an installer” even makes sense.
- I’ve no idea what micropip is, what relationship it has to pip, or whether it’s even an “installer” in the sense that people used to pip/uv expect.
- I don’t know why you have both micropip and a patched copy of pip. I don’t know why a user would pick one over another.
- I don’t know if there’s any expectation that pip and/or uv would ever support pyodide natively.
- I don’t know what that would involve - my understanding is that pyodide lives in a sandboxed “browser context”, which means that things like the
subprocessmodule might not work (the docs say “Availability: not Android, not iOS, not WASI” - how does pyodide relate to those platforms?). Without asubprocessmodule, pip support seems unlikely (to put it mildly!) - I don’t know how pyodide users expect to install packages, whether they have a concept of “running a Python script” or whether everything is in terms of standalone applications built in Python, but not exposing the fact that Python is the implementation language. This is important because the packaging ecosystem has acknowledged weaknesses in the “building a standalone application” area, and I don’t want those to end up obscuring the question of what “pyodide support” means.
Does that help at all?
I could go and look all of this up, probably, but I don’t have a need, personally, for pyodide, so I lack the context of a real-world problem that I’d be using pyodide to solve. In fact, I don’t even know what sort of real-world problems pyodide is intended to solve. Is it for writing command line applications? For embedding Python code in webpages in place of javascript? Something else?
2 Likes
The non-Pyodide-expert readers literally don’t know what Pyodide is. They also don’t know what WASM and WASI are and what the differences are between those. They don’t know what emscripten is and they don’t make websites and don’t use JS or Node either. They have never heard of Pyodide before or if they did hear of it then all their information is out of date because it changed so much in the last few years.
The PEP possibly implies that Python people who don’t know what Pyodide is might need to do something to support it somehow but it is impossible for them to agree to that with so little understanding of what Pyodide even is. They have no idea what (if anything) is being asked of them now in the short term but are probably more concerned about what is perhaps implicitly being asked for in the long term future.
4 Likes
Does that help at all?
Yep! Thanks for the specific questions.
I could go and look all of this up, probably, but I don’t have a need, personally, for pyodide
I have no expectation that you would do that, but I appreciate you expressing the specific points of concern. Perhaps some of this should go into the PEP or be cross referenced from it.
- how to describe the build
The output of the Emscripten linker is a pair of files, a JavaScript file and a WebAssembly file. The JavaScript file contains implementations of system calls (for instance a file system), graphics apis, etc. The WebAssembly file contains all the compiled C code. Pyodide provides an additional loader file, which exports a JavaScript function called loadPyodide() which accepts various options for how the runtime should start up. It returns a JS abstraction over a Python runtime. (loadPyodide() can be called multiple times in the same JS thread and will produce multiple Python runtimes with completely independent memory spaces).
- whether it runs in a command line context where “running an installer” even makes sense.
There is a cli entrypoint. It mounts the entire native file system into the Emscripten file system (minus /dev, /proc, and /lib) and also all the native environment variables. Then it calls the normal Python main() function. It works like a drop in replacement for the normal Python cli, except that it is much slower and various modules don’t work.
- what micropip is, what relationship it has to pip, or whether it’s even an “installer” in the sense that people used to pip/uv expect.
Micropip is a low quality installer written specifically for use with Pyodide. Pyodide comes with a distribution of 264 builtin packages. Micropip will look for requirements there, or from pypi, or possibly from alternative simple indexes. It has a low quality resolution algorithm that won’t backtrack and is not deterministic.
Also, when used in a webpage the file system is transient. So packages have to be installed on every page load. Thus, it’s much preferable to resolve dependencies ahead of time for determinism and efficiency.
- why you have both micropip and a patched copy of pip, why a user would pick one over another.
Since the quality of implementation in micropip leaves a lot to be desired, it is naturally preferable to use pip when possible. Furthermore, we want to make it as easy as possible for people to use Pyodide with their existing CI pipelines. Supposing in CI they build their package, create a virtual environment, pip install the built package and its testing requirements, and then run pytest. We’d like to substitute the virtual environment for a Pyodide virtual environment and then have the same code work in the same way.
You can see it in action here (Nowadays numpy just uses cibuildwheel which knows how to do all this out of the box.)
- if there’s any expectation that pip and/or uv would ever support pyodide natively.
I asked the uv people and they said yes to supporting Pyodide. It seems like it would be a lot more work for pip. Some day it would be nice if pip could make cross-venvs but that seems like an issue to try to wrangle with in the future. Or never.
- I don’t know what that would involve
For uv, this PR is enough to get it correctly installing wheels for Pyodide if I use it like:
uv pip install --python /path/to/pyodide/python --extra-index-url file:/path/to/pyodide/simple-index --target target_directory
For pip it’s a bit more complicated. We’d like to run pip under a native Python, but then we have to hack that native Python so that sysconfig and os.name and all those things report the information from the Pyodide Python.
- pyodide lives in a sandboxed “browser context”, which means that things like the subprocess module might not work (the docs say “Availability: not Android, not iOS, not WASI” - how does Pyodide relate to those platforms?). Without a subprocess module, pip support seems unlikely (to put it mildly!)
Yeah, currently there is no subprocess support. We would be running pip in node, and node has support for subprocesses. So it should be possible to add a subprocess backend for Pyodide that makes subprocess work when run under node. It is my goal to implement this at some point but it is hard and has not yet been a high enough priority.
In the meantime, we can run pip in a native Python and make a cross-venv.
- I don’t know how pyodide users expect to install packages, whether they have a concept of “running a Python script” or whether everything is in terms of standalone applications built in Python, but not exposing the fact that Python is the implementation language. This is important because the packaging ecosystem has acknowledged weaknesses in the “building a standalone application” area, and I don’t want those to end up obscuring the question of what “pyodide support” means.
We don’t expect the Python packaging ecosystem to fix these problems. Pyodide helps a bit with these problems, but we are mostly still pushing this onto the downstream. Of course people might appreciate if things were easier but we only have so much energy to make things work well.
- what sort of real-world problems pyodide is intended to solve. Is it for writing command line applications? For embedding Python code in webpages in place of javascript? Something else?
Most generally, it is for running Python inside a JavaScript runtime. The primary use case is for embedding Python code in webpages in place of JavaScript. It also is often used in Node projects that need a small amount of Python code, sometimes it is more convenient than running a Python subprocess. It also allows sandboxed Python execution, so a bunch of the llm people are using it to execute junk that was spat out by an llm.
Also, I focus a lot on the use case of maintaining a package that supports Pyodide.
2 Likes
I think that the key distinction here is that micropip runs inside pyodide. You can call it from either Python or JS code inside the pyodide/JS runtime. If I understand correctly it is implemented in Python but shares no actual code with normal pip.
The most common usecase for Pyodide is to run Python within a webpage in the browser and in that situation micropip can be used to install things dynamically. For example in the SymPy Live website you can enter the Python code
import micropip
await micropip.install('colorama')
and then that will download and install colorama for use inside the ephemeral Python environment in the webpage session.
Although that is usually how Pyodide is used it is also possible to use Pyodide outside of the browser with Node. In this case the easiest way for a Python developer to set things up is to create the “cross-venv” which is a virtual environment that can be used much like a normal Python virtual environment but where the python executable runs Pyodide under Node (I think?).
The pyodide docs don’t give a clear explanation of this but here is a simple demonstration running on Linux (not sure if it is significant that I already have node v18 installed):
# First activate a normal CPython 3.12 venv
source .venv/bin/activate
# Create and activate a Pyodide cross-venv:
pip install pyodide-build
pyodide venv .venv-pyodide
source .venv-pyodide/bin/activate
Having created and activated the cross-venv you can run the emscripten Python:
$ python
Python 3.12.7 (main, Apr 4 2025, 20:01:14) [Clang 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b on emscripten
Type "help", "copyright", "credits" or "license" for more information.
>>>
Once you have activated the cross-venv it functions more or less like a normal venv. In particular you can run pip and install packages although it fails when I do it:
$ pip install colorama
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = '/.../.venv-pyodide/bin/python3.12-host'
isolated = 0
environment = 1
user site = 0
safe_path = 0
import site = 1
is in build tree = 0
stdlib dir = '/install/lib/python3.12'
sys._base_executable = '~/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/bin/python3.12'
sys.base_prefix = '/install'
sys.base_exec_prefix = '/install'
sys.platlibdir = 'lib'
sys.executable = '/.../.venv-pyodide/bin/python3.12-host'
sys.prefix = '/install'
sys.exec_prefix = '/install'
sys.path = [
'/install/lib/python312.zip',
'/install/lib/python3.12',
'/install/lib/python3.12/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x000070bd540f0740 (most recent call first):
<no Python frame>
Although it doesn’t work for me this pip is the patched pip that has been referred to. I think it doesn’t work because the patching is not compatible with the way that uv has installed the original CPython that I used.
Unlike micropip the patched pip does not actually run inside Pyodide. Rather it runs under the original CPython environment that was used to create the cross-venv but the pip is patched so that it downloads wasm wheels and installs them into the cross-venv rather than into pip’s own CPython environment.
1 Like
By the way, on the “what it can be used for”, there are a ton of uses. A few of them:
- Live Jupyter notebooks (fantastic for teaching) https://jupyterlite.readthedocs.io/en/stable/try/lab
- Interactive WebApps written in Python (using (pure) Python libraries from PyPI) Repo-Review - Scientific Python Development Guide
- In-browser REPL https://pyodide.org/en/stable/console.html
- Live docs https://awkward-array.org/doc/main/_static/try-it.html
- PyScript: replacing JS with Python for beginners https://pyscript.net
- Single binary deployment for all OS’s and architectures with NodeJS (WASI is even better than Emscripten for this one, but doesn’t have dynamic linking)
I assume what we want to do is capture enough of these sorts of descriptions in the PEP so the reader can get a feel for what using Emscripten Python, and specifically packaging for it, looks like.
1 Like
Although it doesn’t work for me this pip is the patched pip that has been referred to. I think it doesn’t work because the patching is not compatible with the way that uv has installed the original CPython that I used.
Actually there is an upstream problem here unrelated to the specific Pyodide use case:
cpython issue
uv issue
In the same spirit of trying to make it clearer what I don’t know…
I don’t see why I would need to know this. I don’t need to understand what C compilers output to know how native extensions get bundled in Python packages - all I need to know is that compiled code is not portable across machines (with some exceptions that I know can be described, but that’s enough).
With pyodide, I don’t even know what counts as a machine. Does it matter that I’m on Windows or Linux? Or that I’m on an x86 or ARM machine? I thought emscripten/WASI/??? ran on a virtual environment that’s something to do with Javascript?
I can’t work out what that means. Best I can imagine is that it means that you can write Javascript code somewhere (where? in a web page? in a .js file run via something like deno foo.js?) that does callPython("import sys; print(sys.executable)"). But what’s sys.executable going to refer to? Is this *actually* Python, or a highly restricted subset because concepts like sys.executable` aren’t meaningful in this context?
Sorry, I don’t know what a “cli entrypoint” means. Does it mean I get a command prompt and can run python somefile.py from there? Is that how users typically use pyodide? (It’s how many users use CPython, so my mental model is “yes, that’s typical use”).
That sounds like you’re installing dependencies from within a running Python process. In every other context I work with, that’s a bad idea, running the risk of confusion or errors when you install something after you first use it.
I’m not clear when it wouldn’t be possible to use pip, except if your installing into an already-running process, which is not something the packaging ecosystem really tries to support.
Again you lose me here as I don’t know what a “Pyodide virtual environment” is as distinct from any other virtual environment (python -m venv).
Other than using pyodide venv rather than python -m venv, that seems perfectly normal to me. So I don’t understand why you have a pyodide command rather than a “pyodide-flavoured” interpreter that would allow python -m venv to do what pyodide venv does.
What gives you that impression? What I’ve been saying so far isn’t that I think pyodide support would be hard for pip, but rather than I have no clue how to even begin to assess that question.
Ah. Why can’t you run pip under the pyodide Python? That’s what I’d expect, and in particular I’d strongly advise against using --target, because it doesn’t support things like upgrades and reinstalls, making it essentially just “create a new empty directory, pip install --target into it, discard once done”. I thought that’s what uv’s --target did, too, so don’t they have the same limitations?
Confused again. Note is a Javascript runtime, pip is Python code. How can node run a Python program?
If I read this correctly, you’re not actually after pip to support pyodide, rather you’re after pip to have robust cross-platform install capabilities, with one of the targets being pyodide (but with no intention of pyodide being a host platform).
OK, that’s a much more understandable statement, and if I got it right, then you’re correct, cross-platform support for pip is not an easy ask. There’s a lot of UI questions we haven’t answered yet, as well as a bunch of limitations with --target that we don’t consider acceptable for a full solution. If we fix the UI questions, pip install --target-platform=pyodide_platform_desc.txt --target xxx could be viable (with the proviso that xxx must be a brand new, empty directory) and shouldn’t involve anything more than any other platform description.
That wasn’t my point, exactly. My point is, if the workflow you’re targeting is “build an application”, then you need the sort of solutions that tools like pyinstaller, cx_freeze, or zipapps provide. Those tools aren’t under the umbrella of “Python packaging”, even though it’s arguable that they should be, and so there’s a lot of stuff that simply doesn’t have “packaging ecosystem” answers.
So <script language=python>print("Hello, from Python!")</script>? OK, that’s cool.
I guess if I tried to force this into some sort of mental model, I’d be looking at
- I have an application, I’ve written it as a script with some dependencies.
- I create a directory structure, put my dependencies in a
libdirectory and modifysys.pathin my main script to add thatlibdirectory. - I now zip everything up into a zipapp, which the user can then run.
For pyodide, “zipping up” is replaced with some magic to make a web thingy that I can add to a webpage using some incantation in the HTML.
The bad news is, almost none of that is covered by “packaging”, even in the normal CPython sense.
Yeah, that seems different. I’d assume a key thing here would be some sort of pyodide equivalent of pytest? (Maybe pytest itself, just made into a pyodide “magic thingy”?) But again, testing isn’t a packaging matter.
OK, my takeaway from this is that cross-platform environment building is key to pyodide. And currently we have no standards for that - specifically:
- No standard way of describing an environment (although PEP 739 might be relevant here).
- No robust support in tools for treating a standalone directory as an “environment” (and no standards saying how such support would work).
- No common terminology for a UI for describing environments.
That’s fine - I get that you’re not asking for this. All you want is a tag, and that’s certainly possible.
Pip won’t need to do anything to “support pyodide” in this model, as we currently only support cross-builds as far as is possible using the --platform, --implementation and --abi options. The question will be whether pyodide can be described by those values or not. I suspect not, but that’s just a guess.
How am I doing? Do I sound like I have more of a clue now, or have I managed to go completely off in the wrong direction?
It’s just normal pytest. Think of pyodide as being a platform much like Windows x86-64 vs manylinux aarch64. Pure Python packages work fine but anything with extension modules needs to be compiled specifically for the pyodide platform.
The main things that don’t work in pyodide are subprocesses and threads so e.g. pytest works but you can’t use pytest -n auto to run tests with multiple processes.
The SymPy CI job that runs pytest under pyodide is here:
2 Likes
I gave that a go (I’m on WIndows, Python 3.13.2, and have node 23.11.0 installed). Sadly, it failed at the pyodide venv step with an error:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Gustav\\AppData\\Local\\Temp\\tmpquun7mj0.tar'
Yeah, I got that from Hood’s message. I’m uncomfortable about scripts installing packages into the environment they are running under, and I doubt I’d ever be willing to describe that as “supported”. It’s not a packaging issue, though, it’s about the core import mechanism itself (which caches a bunch of stuff in ways that pretty much assume what’s on disk won’t change while things are running). I will accept that if you’re careful, it generally works, so as long as you don’t do “clever” stuff, and you’re willing to deal with any errors yourself.
All of them (except the last one) come down to “Embedding Python in web pages”. And the last one is essentially “Embedding Python in standalone JS”. Is that fair? Because if so, that’s a nice model for me to get my head round. It puts the pyodide tool (and the patched pip, and stuff like that) in the realm of “web development things”, which I understand the existence of, but treat as a dark art that should only be explored at the risk of your sanity (my web development is at the level of writing HTML and CSS in an editor).
I’d rather not, to be honest. I want the PEP to stand without needing a lot of background. IMO, what we should need is more along the lines of
We need a tag for pyodide. We need it because pyodide is a supported Python platform, and existing tags aren’t sufficient because pyodide has its own binary format. It will be defined like so-and-so, and tools will determine if a Python installation supports that tag like this. New tags will be introduced when X or Y happens. PyPI should accept wheels with the new tag.
To be frank, if the PEP had simply said that, I would probably never have started this whole discussion about “what does all this mean for pip” 
Edit: Don’t get me wrong, the discussion has been useful, and incredibly interesting. But I don’t think it’s that relevant to the PEP (same proviso as always here, “as long as I’ve not totally misunderstood something” 
)
How am I doing? Do I sound like I have more of a clue now, or have I managed to go completely off in the wrong direction?
Yes I think you are correctly understanding.
I don’t see why I would need to know this.
Agreed that you probably don’t need to know this. But you asked “how to describe the build” and this was an attempt to answer that. If you meant something else, I’m happy to try again.
With pyodide, I don’t even know what counts as a machine. Does it matter that I’m on Windows or Linux? Or that I’m on an x86 or ARM machine? I thought emscripten/WASI/??? ran on a virtual environment that’s something to do with Javascript?
The host environment is always Emscripten which so it’s always a posixy, linuxy environment. The system calls are all implemented in JavaScript. In the browser, there should be no differences between execution on Window, Linux, x86 or Arm. In Node, the native file system will behave differently depending on the native platform, but it is still mostly platform independent.
Best I can imagine is that it means that you can write Javascript code somewhere (where? in a web page? in a .js file run via something like deno foo.js?) that does
callPython("import sys; print(sys.executable)"). But what’ssys.executablegoing to refer to?
Yes this is right. In the browser, sys.executable is meaningless but in node it will point to a shell script that starts up a Pyodide Python interpreter.
Is this actually Python, or a highly restricted subset because concepts like
sys.executablearen’t meaningful in this context?
I would argue that it is actually Python, but there is a valid argument in the other direction. We are gradually increasing how normal we can be but it is hard.
Does it mean I get a command prompt and can run python somefile.py from there?
Yes.
Is that how users typically use Pyodide?
No, but it’s very useful mostly for package maintainers and maintainers of Pyodide itself. And for creating things like virtual environments. I think the most common goal is using Pyodide in a browser.
I’m not clear when it wouldn’t be possible to use pip, except if your installing into an already-running process, which is not something the packaging ecosystem really tries to support.
Yes, we are often installing into an already-running process. It isn’t as broken as you might think, but we would also like to do less of this.
Other than using pyodide venv rather than
python -m venv, that seems perfectly normal to me. So I don’t understand why you have a pyodide command rather than a “pyodide-flavoured” interpreter that would allowpython -m venvto do whatpyodide venvdoes.
That makes a lot of sense. I am not sure if there is a good reason not to do this, I’ll have to think about it.
Why can’t you run pip under the pyodide Python?
It uses urllib3 and subprocesses. urllib3 supports Pyodide in Node now using stack switching. subprocesses in Pyodide under Node could be supported but will take some work. I’m not sure if anything else is missing.
But it is also a lot slower than native Python.
I’d strongly advise against using
--target, because it doesn’t support things like upgrades and reinstalls
We don’t use --target, we make a normal virtual environment, put a pip.conf pointing to the index where we have Emscripten-platformed wheels, and patch venv/bin/pip so that it monkeypatches os.name, sys.platform, platform.system, sysconfig, … before calling pip._internal.cli.main.main(). And also, fixes up the shebang of installed cli entrypoints so they point to the pyodide python instead of the native python.
Considering how crazy this is, it works amazingly well at making cross-venvs.
Confused again. Note is a Javascript runtime, pip is Python code. How can node run a Python program?
I mean pip in Pyodide in Node, sorry for being confusing =)
I guess if I tried to force this into some sort of mental model, I’d be looking at
…
The bad news is, almost none of that is covered by “packaging”, even in the normal CPython sense.
Yep, that’s an accurate mental model.
I’d assume a key thing here would be some sort of pyodide equivalent of pytest? (Maybe pytest itself, just made into a pyodide “magic thingy”?)
All we need is to take the normal pytest and install it with pip into our cross-venv. So it gets a shebang that points to the Pyodide python.
OK, my takeaway from this is that cross-platform environment building is key to pyodide. And currently we have no standards for that
Exactly.
as we currently only support cross-builds as far as is possible using the --platform, --implementation and --abi options. The question will be whether pyodide can be described by those values or not.
It cannot be.