My understanding is that pyInstaller doesn’t work at the package level at all, but at the individual import module level. So it scans the source code looking to see what specific components of (say) scipy are used, and only includes them in the application. That gives significant space savings (especially with a large library like scipy) but it’s fragile in the face of dynamic imports and similar runtime tricks for loading modules.
I believe there are heuristics that handle conditional imports in well-known libraries, but that doesn’t help with actual application code. As I said above, my (very limited) experience is that it’s a good approach for pyInstaller’s target audience, but it doesn’t fit well for some use cases. I’ve not tried the approach mentioned above of customising things via the Analysis API, so it may be that offers a way of working round this.
The key thing is that as I understand it, pyInstaller scans the application environment and extracts the necessary files from that. It doesn’t work with wheels, or any other packaging distribution files, at all.
The above may be inaccurate - whenever I’ve tried to read the pyInstaller documentation, which hasn’t been for some time, I’ve found it hard to follow precisely because its perspective is based around freezing an environment, and not around packages and dependencies, etc, like I’m used to. So the terminology and context is very different to what I am used to.
Last I checked, PyOxidiser just uses wheels. But it also rewrites all your (probably not zip safe) packages and extension modules into one big file which brings in a new world of ways to go wrong so I wouldn’t consider the problem of a packaging tool based on wheels or metadata to be solved.
You can apply the same sort of idea there though and just say that you give a complete list of which import packages from site-packages should be included. Then pyinstaller or other tools could be used to help create that list and you could maybe make some manual changes.
Alternatively if you have the output of pip freeze and the directory of wheels then you can generate a venv with the installed packages. Then you just need a way to tell pyinstaller to skip import scanning and include the whole of site-packages.
Late to this. I have not traditionally lurked in the packaging channels.
Yes - minus the installer icon, since this produces a “scie” which is self-installing, you can use Pex: pex [the package] -c [console script entry point] --scie eager -o my-self-installing-native-executable. If you don’t have a console script entry point you can use -m module or -e package.module:function.
For example:
:; pex f2-commander -c f2 --scie eager -o f2
:; file f2
f2: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f1f01ca2ad165fed27f8304d4b2fad02dcacdffe, stripped
:; zipinfo -1 f2 | grep METADATA
warning [f2]: 31530344 extra bytes at beginning or within zipfile
(attempting to process anyway)
.deps/Send2Trash-1.8.3-py3-none-any.whl/Send2Trash-1.8.3.dist-info/METADATA
.deps/f2_commander-0.1.0-py3-none-any.whl/f2_commander-0.1.0.dist-info/METADATA
.deps/humanize-4.11.0-py3-none-any.whl/humanize-4.11.0.dist-info/METADATA
.deps/linkify_it_py-2.0.3-py3-none-any.whl/linkify_it_py-2.0.3.dist-info/METADATA
.deps/markdown_it_py-3.0.0-py3-none-any.whl/markdown_it_py-3.0.0.dist-info/METADATA
.deps/mdit_py_plugins-0.4.2-py3-none-any.whl/mdit_py_plugins-0.4.2.dist-info/METADATA
.deps/mdurl-0.1.2-py3-none-any.whl/mdurl-0.1.2.dist-info/METADATA
.deps/platformdirs-4.3.6-py3-none-any.whl/platformdirs-4.3.6.dist-info/METADATA
.deps/pygments-2.18.0-py3-none-any.whl/pygments-2.18.0.dist-info/METADATA
.deps/python_dotenv-1.0.1-py3-none-any.whl/python_dotenv-1.0.1.dist-info/METADATA
.deps/rich-13.9.2-py3-none-any.whl/rich-13.9.2.dist-info/METADATA
.deps/textual-0.83.0-py3-none-any.whl/textual-0.83.0.dist-info/METADATA
.deps/typing_extensions-4.12.2-py3-none-any.whl/typing_extensions-4.12.2.dist-info/METADATA
.deps/uc_micro_py-1.0.3-py3-none-any.whl/uc_micro_py-1.0.3.dist-info/METADATA
Although all the machinery existed for this since November 2022 using the scie jump directly and using the higher level science tool since May 2023 (See Science · GitHub), Pex only incorporated that machinery for convenience in 2.11.0 in July. Maybe this helps.
It does not. They is me (or has been for the last 5 years or so) and I do plan to do so. I have in fact been torturing myself using Windows for the last 2 years to force this issue (I prefer Linux). My bar has been high though, full green CI to land the 1st support, and that has been slow!
That said, the science executable does support Windows and is written in Python and uses shiv for the moment since it does support Windows. That dogfood is here for example: lift/lift.toml at main · a-scie/lift · GitHub
A few more steps than just pex --scie eager, but not too hard either.
PyApp will support this in future. Currently you can (among other options) supply a premade distribution or a requirements.txt-style file for runtime installation. The binaries for Hatch now use a premade distribution so installation is just a single download and unpack operation.
PyApp supports Windows, I forget if you’ve tried that yet.
I have. I consider it to be a fundamentally different approach (that isn’t what I want from an “tool that builds a standalone application from Python code”). But we’ve had this debate before and it’s likely fruitless to repeat it.
Pynsist (disclaimer: I’m plugging my own tool ) has a few different ways to get dependencies, but the recommended option where possible is to give it a fully resolved list of packages from PyPI (like the output of pip freeze), which it downloads as wheels for you. It’s also possible to point it at extra wheel files in a local directory.
Caveats:
It treats your own application code rather differently, expecting this as an importable package in the source folder, not a wheel.
It only makes Windows installers, so if you’re making a cross platform application you still need to figure out something else for Linux & Mac.
For Linux, depending on the target audience, I might make a Flatpak (with this helper tool), or just give people instructions for pip(x).
It builds old fashioned executable installers, not the newer .msi packages (or have those too been replaced now?)
I’m not a very good maintainer of Pynsist, since I don’t actually use Windows myself.
) has a few different ways to get dependencies, but the recommended option where possible is to give it a fully resolved list of packages from PyPI (like the output of 