A number of stdlib modules don’t work in the browser as outlined in Compile Python to WebAssembly (WASM) — Unofficial Python Development (Victor's notes) documentation namely multiprocessing, sockets, threading in Pyodide (but not in CPython Emscripten node build). Some of these might work one day via some browser-specific implementation, others will never work.
The question is what should happen when users try to use these modules. For instance, let’s take the example of
multiprocessing. In @tiran 's wasm-python terminal importing multiprocessing raises an
ImportError. As far as I understand, that’s the standard behavior for unsupported functionality in a given runtime, and allows easy feature detection with
try: ... except ImportError:.
The problem is that a lot of these modules were assumed to be non-optional parts of the stdlib for a long time (because all mainstream Python distributions came with them). A lot of packages use them without handling the case where they are missing. If they are missing, because of top-level imports everywhere having this unhandled import error, will likely prevent the import of the package altogether even if e.g. multiprocessing is really not critical for core functionally.
And so in Pyodide what we have been doing lately is to include these modules even if they don’t work.
So for instance, one can import
multiprocessing but trying to create a new process would produce an exception. This allows for a large part of the package ecosystem to work without changes (and experiments with exposing
multiprocessing weren’t conclusive). But then it raises the question of what is the proper way to detect missing functionality (most recently in Dask in pyodide by ian-r-rose · Pull Request #9053 · dask/dask · GitHub).
So overall this thread aims to,
- get feedback on what the correct behavior should be for these stdlib modules when used in the browser or Node.js
- how to adjust the package ecosystem to handle that behavior (if necessary)
- and indirectly, to make sure that the fixes people contribute to make packages work with Pyodide would also benefit other CPython WASM builds.
Also @tiran (or someone else) could you please confirm that,
import platform if platform.system() == 'Emscripten': ...
is a recommended way to detect running in Emscripten and will work in the future (it does in your REPL now). Though there should probably be a way to differentiate between browser and non-browser runtimes? We should adapt the Pyodide FAQ accordingly. Thanks!