Cannot import pyd from egg

Hi folks,

I have the following project files and I want to distribute them as one egg file.
Is it possible to import *.pyd module from *.egg file?

mylib/
    legacy/
        __init__.py
        _calc.cp310-win_amd64.pyd
    setup.py
    Makefile
    dist/
        mylib-0.0.0-py3.10.egg

The “legacy” package contains “_calc.xxx.pyd” file which has some c functions.

# legacy/__init__.py
from . import _calc as calc

I wrote setup.py as,

# setup.py
from setuptools import setup
setup(
    name = "mylib",
    package_dir = {'': "."},
    packages = ["legacy"],
    package_data = {'legacy': ['*.pyd']},
)

which builds “dist/mylib-0.0.0-py3.10.egg” using Makefile:

egg:
	py -3 setup.py bdist_egg

There’s no problem except that I can’t import _calc module from the egg.

# test.py in dist/
import sys
import os
sys.path.append(os.path.abspath("mylib-0.0.0-py3.10.egg"))
import legacy
# ImportError: cannot import name '_calc' from partially initialized module 'legacy'
# (most likely due to a circular import)
# (C:\usr\home\mylib\dist\mylib-0.0.0-py3.10.egg\legacy\__init__.py)

If I unzip the egg, it is no problem with importing.
Any advice would be appreciated.

I had always assumed that you have to install the egg with easy-install.

Also isn’t .egg format replaced by python wheels?

Also not sure if you can run .pyd aka .dll files from inside a .egg aka .zip file.

Loading extension modules from a ZIP archive is not supported in Windows. To load an extension module, the interpreter calls LoadLibraryExW(), which requires a real file on disk. We’d have to first find all of the dependent DLLs and then extract the PYD and DLLs from the archive into a temporary directory.

3 Likes

Thanks for the explanation. Maybe it would be helpful to add a suitable error message for this case of ImportError into Python?

Could not there be cases which would cause a different behaviour on Windows vs other platforms without raising an exception? (E.g. a different module of the same name would exist on the filesystem.)

Why not use a pyinstaller built single exe project?
It knows the trick of writing the dll to disk in a tmp folder so they load.

Thank you very much for the clear explanation!

It’s possible, but still not possible to import pyd from wheels.
As long as I read Wheel vs Egg — Python Packaging User Guide, I think the wheel seems to be recommended.

I think so, too. When I got the “…(most likely due to a circular import)…” error message, I thought there was something wrong with making the egg file.

IMHO, the pyinstaller tends to create huge and slow executables for the GUI applications.
I’ve used py2exe, but unfortunately, it doesn’t seem to support the recent version of Python.
I think providing an egg/whl as a plugin to your application can be fast and lightweight.

I wrote my own python to exe tool that is on PyPi win-app-packager.
I take the app files it creates and then install them using Inno setup Inno Setup
I use this for PyQt based projects.
The resulting app runs with no slow downs.

I tried to replace py2exe with PyInstaller almost ten years ago and was disappointed by its huge size and extremely long startup time.
Today, I tried PyInstaller again and found that the compile time is a bit long (~10 minutes), but the size is small enough (~50MB) and the startup time is greatly improved.

Many thanks for the information!

2 Likes