I’ve been trying to get C extensions working in the WASI build of Python. I started with a simple one that lives at the top-level, statically linked it, and used PyImport_AppendInittab in my C wrapper and it worked just fine. I’ve moved on to a more difficult test: numpy. I’ve been able to get numpy to compile to Wasm as static archives and I can link them with a static Python library, however, I’m having a problem importing numpy.core._multiarray_umath.
Importing that library seems to cause a cyclical import. Its module initialization function imports numpy.core._exceptions which causes some other imports to happen, eventually reaching numpy.core.multiarray which imports numpy.core._multiarray_umath again. In the standard Python build, this works fine and it just continues, but in the Wasi build it fails when it hits that import cycle. I’m not clear as to what happens that the standard build works that isn’t happening in the Wasi version (it likely has to do with static linking, not so much Wasi). If anyone has any tips, I’d greatly appreciate it.
That’s great to hear! I have been thinking that might be the best solution for WASI until we get some dynamic linking solution (I’m hoping the component model could get leveraged for this some day).
The code that handles built-in imports is in the BuiltinImporter class. There’s nothing specifically there that I can see which would suggest statically linking and using built-in imports would cause a failure that would work otherwise. Typically cyclical imports fail because someone is importing to a specific object in a module instead of the module itself since the import system sets a module in sys.modules ASAP specifically so any other imports will get the module from there and not trigger another import.
Any chance you can take your build and run it under a debugger to see if my hunch is correct? If it is then please open an issue at Issues · python/cpython · GitHub ? I would also then open an issue on NumPy letting them know.
I haven’t had much luck getting a debugger to work, so I’ve had to track things down the old fashioned way (printf…). The line where things go bad is https://github.com/python/cpython/blob/main/Python/import.c#L2547. The first time this is hit by importing numpy.core._multiarray_umath, it causes the parents to get imported (numpy.core and numpy) which causes the subsequent imports to come around to importing numpy.core._multiarray_umath again. The second time it hits that line in import.c, it fails.
I’ll keep trying to get a debugger on it, and maybe post a repo with my changes and build scripts so that you can try to reproduce it if you have time.
As far as the working C extension goes, it came together relatively easily. I even have exports from the host program that are called by the Python C extension, so it’s demonstrating communication both ways from the host.
Have you tried doing the same build to the native CPU of your machine so you can debug that way? In other words don’t build for WASI for the target but your machine’s own CPU, leaving everything else the same?
Can you provide a permalink? That line currently goes to a } in the middle of a function.
I’ve been trying to build a native version of Python and numpy, but I’m having trouble getting it to work for some reason that I haven’t figured out yet.
@kesmit I was hitting this last year and I just removed the assert and changed the if above it as in
- if range_start == range_end {
+ if range_start >= range_end {
continue;
}
- assert!(range_start < range_end);
Then, with a local build of wasmtime I was able to do some debugging. It’s not as good as printf, but did give me faster iteration cycles till I find the places worthy of a printf.
I have no idea what those ranges represent, just saying that it worked for me.
I did get a little further by getting a static x86 build working. I found out that there was a linker option that needed to be added. That got the x86 build working, but unfortunately, it didn’t completely fix the Wasm build (although it is getting a little further). I’m still digging…
I thought I would at least be able to edit Lib/importlib/_bootstrap.py with print statements, rebuild Python and they would show up, but that’s not even working so there’s something I don’t understand here.
@brettcannon There was an issue in the Python BuiltinImporter. In this case, the builtin extension was namespaced below numpy.core. The BuiltinImport is not expecting namespaced extensions and when it sees the path parameter supplied, it short-circuits and returns None. The simple fix I put in (that may not be perfect), is to check for a ‘.’ in the import name and to see if it is a builtin name before doing anything else.
class BuiltinImporter:
. . .
@classmethod
def find_spec(cls, fullname, path=None, target=None):
if '.' in fullname and _imp.is_builtin(fullname):
return spec_from_loader(fullname, cls, origin=cls._ORIGIN)
if path is not None:
return None
if _imp.is_builtin(fullname):
return spec_from_loader(fullname, cls, origin=cls._ORIGIN)
else:
return None
Here’s the next step: pandas. However, pandas has some much bigger problems than numpy. It only sorta works, and the build takes more hand-holding.
import platform
print(platform.platform())
import pandas as pd
df = pd.DataFrame([[1.234, 2, 'hi'], [3.14, 3, 'bye']], columns=['a', 'b', 'c'])
print('COLUMNS', df.columns)
#print('DTYPES', df.dtypes)
for i, row in enumerate(df.iterrows()):
print('ROW', i, row[1])
Result:
wasi-0.0.0-wasm32-32bit
COLUMNS Index(['a', 'b', 'c'], dtype='object')
ROW 0 a 1.234
b 2
c hi
Name: 0, dtype: object
ROW 1 a 3.14
b 3
c bye
Name: 1, dtype: object
Out of curiosity, are you adding pandas due to demand or because you just want to? I totally get NumPy as it’s so foundational for the sciences/numerics, but I would interested if pandas is there or if pure Python alternatives could also work in this instance to save the headache of trying to make it work.
Would you mind opening an issue at