PyEmpaq is a simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.
With PyEmpaq you can convert any Python project into a single .pyz file with all the project’s content packed inside.
That single file is everything that needs to be distributed. When the final user executes it, the original project will be expanded, its dependencies installed in a virtualenv, and then executed. Note that no special permissions or privileges are required, as everything happens in the user environment.
Both the packaging and the execution are fully multiplatorm. This means that you can pack a project in Linux, Windows, Mac or whatever, and it will run ok in Linux, Windows, Mac or whatever. The only requirement is Python to be already installed.
What’s new in this version?
Added include and exclude configuration options to have full control on what is inside the packed file.
Reuse the existing project directory only if it has a complete previous installation.
Support the declaration of a minimum Python version to run the packed project.
Expose the .pyz path to the project being run.
Now used logging to expose messages to developer and users, with different verbosity levels.
Set up the proper PATH in the end command environment.
Added a -V/--version option to just print the version and exit.
Run tests (unit and integration) in Linux, MacOS and Windows.
I’ve had a vague niggling idea for a few years to build something almost exactly like this, so thank you for just doing it I’ll definitely check it out.
Take my following assertions regarding those two tools with a grain of salt because I’ve never really used them, but here’s my comparison.
Shiv would create a fully self-contained Python zip app with all their dependencies included, which means that it ships not only your project but also the dependencies! this makes the resulting .pyz NOT multiplatform (unless your project and ALL your dependencies are pure Python…).
On the other hand, PyEmpaq will create the virtualenv and install the needed dependencies the first time the .pyz is executed in the destination machine. This allows for example to pack a fully graphical desktop program that works with Qt, like this example.
A side detail of this is that we normally include dependencies in our programs, and those dependencies have sub-dependencies, etc. How sure are you that the whole set of dependencies and sub dependencies have a license that allows you to distribute them? Shiv will distribute them, PyEmpaq will not!
Pex looks like it also includes the dependencies in the resulting file, but goes an extra mile pointing to the Python that was used to build the thing. So for example I built a .pex with Py3.10 and moved the file to a machine that has only 3.9, and it didn’t work, and both were Ubuntus! No multiplatform at all, no even mention to pack in Linux (once!) and run in Windows (or everywhere), as PyEmpaq allows to do.
Everything said, it looks like both tools are used to “handle dependencies” (like fades), while PyEmpaq is to pack and distribute your project (and it will handle the dependencies because are needed by your project, but it’s not the focus).
Please correct me if I’m wrong with the analysis I did for those tools, and let me know if you have any questions!
@facundo Thanks for the details. This made me realize this does not do what I had understood on the first read. I had indeed assumed that dependencies were included in the .pyz file.
Assuming I package a project that has dependencies. Will running the .pyz need access to the package index server on the first run only, or on each run?
I think that the way I would describe this tool is that it produces single-file and light-weight installers (not self-contained, in the sense that it will need to download dependencies from the package index server). This is an interesting concept. Maybe it could learn some features from pipx (or integrate with it), so that once the program is installed the .pyz file is not needed anymore (until maybe it is time to uninstall).
I am trying it, but I am encountering an issue:
$ python3.11 Something.pyz
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/tmp/tmp.EQhCLfHNpy/Something.pyz/__main__.py", line 20, in <module>
ModuleNotFoundError: No module named 'packaging'
I filed a bug report on the ticket tracker. Seems like the same way appdirs is installed in the venv directory inside the .pyz output file, packaging should be installed as well. Also I think it is worth considering a migration from appdirs to platformdirs.
One of the advantages of keeping the .pyz around is it fits people’s expectations of “download and run”. If running it is able to transparently reuse cached installs, nobody ever has to really know about the install step - it’ll only occur on the first run. It could even be used for automatic updates (ClickOnce is a similar kind of approach for .NET).
Regarding when the .pyz access the package index server, it will happen only the first time, unless you provide another version of the same .pyz. IOW, if you pack it and send it to some user, the user can run it 100 times and only the first time will create a virtualenv and install stuff there. But, if you do changes in your code, re-pack it, and send the .pyz again to the user, it will do the virtualenv + installation again (even if the .pyz has the same name).
Regarding the failure, I already fixed it and released the fix in 0.3.1. Also commented in the issue you opened.
@sinoroc@steve.dower regarding installers… yes, it would be neat. I already have an issue opened to somewhat provide installers, just to throw the concept there and start collecting ideas.
What worries me is that one hand the project itself would need to be prepared for that (e.g. providing an icon) and on the other hand it will vary a lot in the different systems, which would be a challenge.
In any case, yes, you’re right that it transparently reuse cached installs.
This looks awesome (disclaimer: I’ve not tried it yet). Two thoughts that came to mind for me:
On Windows, .pyz files aren’t completely “first class citizens”, and an application shipped as an executable file (.exe) is typically a lot more user friendly. It’s pretty easy to prepend an executable shim to a .pyz file (distlib and installer both have such wrappers, I believe both of them originate from @vsajip’s simple_launcher project). Maybe an option to do this could be added?
One issue with caching virtual environments is cache management. How does the user handle that? Questions like how to influence where the cache is located, how to purge out of date entries from the cache, and what will happen if the cache entry goes stale (for example, if I uninstall Python 3.11 and install 3.12, all the virtual environments will be broken) are the sort of things I’m thinking of.
Version 0.3.1 works for me now. Thanks for the prompt fix.
As already said, the uninstall process and in general the management of the cache containing the created virtual environments (~/.local/share/pyempaq in my case) might be an issue long-term that is worth thinking about now. It is the same with tools like pip, pipx, Poetry, docker, and so on. Things get cached, but there is not much visibility that there are a bunch of things that one might not use anymore although they are taking storage space. “Not much visibility”, because there is no icon on the desktop to remind you that you have this or this installed.
Maybe it would be a good idea to build a tool for the user (not for the packager), that would give commands like list, and uninstall. Maybe this tool should be inside the .pyz and it can be activated with a flag, for example something like PYEMPAQ_TOOL=1 python Something.pyz cache-list-all.
Anyway, I think this is a clever idea, it has potential. For example, pipx is a very good concept, it makes it super easy to install all kinds of CLI tools, but how do you first get pipx installed? It is always the same bootstrap conundrum. It seems to be python -m pip install --user pipx that is recommended, which is good, but I feel like having pipx in a virtual environment would be better, because pipx has its own dependencies. Maybe a PyEmpaq-style .pyz-based solution can play a role here.
I also like that it can help distribute simple (or not so simple) scripts that do not have proper packaging (no pyproject.toml, not on PyPI).
Yes, it would be great if .pyz would be associated with Python, we’re not there yet. Regarding including an executable inside, it would kill the multiplatformcity of the packed file, which is something that I want to stretch as much as I can. If you put the executable inside, then you’re packing for a specific platform, and that moment you could integrate the dependencies just inside (targeting that platform), etc… this project is not about that.
Cache management is something to consider, yes; some thoughts/replies about it:
the user could influence where the cache could be located, yes, we could add an environment variable for that… but I don’t know if it’s a real need.
there’s nothing planned to “purge old entries”… but hey, disks are big nowadays
you’re right about changing the Python version… I opened a new issue for that.
All that said, note that the main idea of PyEmpaq is to solve majority of needs for casual packers (like I did a game and want to send it to all my friends, no caring their platforms); for super big projects with lot of people working there they could pay the cost of fine tuning and delivering different binaries for different platforms, etc.
@sinoroc regarding your cache handling ideas, having the tool inside each project looks weird; we could provide other tool, though.
For the record, I don’t expect casual end users running it like PYEMPAQ_TOOL=1 python Something.pyz cache-list-all.
Note that currently there’s one environment variable that would put the unpacker in verbose (PYEMPAQ_DEBUG), so you can find where the project is installed easily:
You’re right about distributing simple and not so simple scripts. But let’s take a step further. What about distributing whole programs? A desktop utility, for example, have a lot of code, yes, but also images, other media, etc, and packing it “as a library” and distributing it through PyPI is cumbersome. And also you tell your friend to use pipx to try your game or something, it’s… not user friendly. With PyEmpaq you send the file by mail or Telegram, your friend is playing it two minutes later.
The focus of PyEmpaq is making the life simple for the majority of casual users. Maybe I should write something about this in the README.
The .pyz files already contain the code to “install”, so to me it seems only logical to have at least the “uninstall” code in there: PYEMPAQ_ACTION=uninstall python Something.pyz.
If I remember correctly (I do not use Windows these days), it is a matter of adding pyz to the PATHEXT environment variable. Not sure why the installer for Python does not do that. Maybe it is not something that installers on Windows are allowed to do.
Anyway, I think I have seen people distributing zipapps with the .py file extension. The assumption being that .py files are associated with a Python interpreter and double-clicking those files would start the program. I do not know if it is really the case, if it is a viable solution. I guess many people have the .py extension associated with some code editor.
That’s not always sufficient, which is why I made the suggestion. It works a lot of the time, so it may be sufficient for the use cases PyEmpaq is aimed at, but there are annoying edge cases.
The launcher installer creates a file association (sources), so if you install py.exe then double-clicking .pyz files (or typing myapp.pyz in Command Prompt) should launch it.
Modifying PATHEXT only allows you to omit the .pyz in terminals. MSI-based installers can modify the environment, but it’s not super reliable. Particularly for PATHEXT, I’d rather it be part of the launcher rather than the main installer, because we know that the launcher is only ever installed once.[1]
One of the unreliable bits is that environment variables are “uninstalled” by calculating the insertion and then finding and removing it. So if you have two installers touching the same key with the same value… yeah, you can probably guess from there. ↩︎
Without PATHEXT, PowerShell starts up a separate console window rather than running in the existing console. I don’t know why this is, but it means that in the default configuration, running .pyz files from a Powershell prompt is essentially useless - you have to set PATHEXT to get sane behaviour.
The CMD shell doesn’t exhibit this behaviour, which may be why there aren’t many complaints about it (anyone who chooses to use Powershell as their default shell is probably capable of setting PATHEXT).
I’ll definitely check it out.
