I wrote a summary of situation on my Mac, but Discourse crashes when I tried to submit, and now my Safari tab hangs whenever I open discuss.python.org
My conclusion was that pipsi and pipx are the most popular tools, but they are more workarounds than solutions, and donāt really solve the root problem of standalone apps being deployed like libraries.
āHow much effort would be involved in convincing the authors of <insert list of standalone tools here> to make this their official distribution method?ā
Any solution that doesnāt get widely adopted is probably useless, IMO. Thereās a definite snowball effect involved. We should also look at why tools like py2exe, cx_Freeze and pyinstaller arenāt more popular.
Agreed pipsi and pipx are workarounds. Managing the virtualenvs, particularly when you dump and reinstall your base Python as often as I do, make them a PITA in practice.
My personal list of tools that Iād like to have available as standalone includes: black, flake8, invoke, devpi-server, pew, pipenv, pylint, pytest, rwt, sqlite-utils, tox, virtualenv, youtube-dl and cowsay (gotta have some fun ones in there ) I have all of them bundled up using shiv at the moment, so (a) I know they work standalone, and (b) I know they are useful to have as standalone apps.
Iāve heard talk about Briefcase, but never tried it (or an app bundled using it). So, maybe? Having to alter setup.py means itās hard to experiment with it āfrom the outsideā - you need to check out and modify the project if you want to build a test executable. And if I, as a 3rd party, can make my own standalone app just by saying something like make_standalone pipenv, then itās a lot easier for early adopters to try out a tool like this and offer feedback.
But from a glance at the quickstart, it looks like the setup.py changes for a project are simple enough, so maybe Iāll give it a try.
Oh, one other thought. The converse of this is āhow much effort would be involved in making this solution so popular/easy to use, that vendors of IDEs like VS Code and PyCharm donāt even consider expecting you to install your linter/formatter/whatever in your Python environment, but just assume youāll have it as a standalone app?ā
I donāt think PyPI distribution is going away in the foreseeable future (if ever at all) unless we outright ban it. It is, however, very possible to convince people to distribute standalones alongside with PyPI. People already request most of these project for alternative packaging (Homebrew is a popular one), so I think it is viable for maintainers to accept as long as we can get a moderate user base and a somewhat āofficialā status.
Some ideas:
A central place to put packages on, like PyPI, and an āofficialā way to install, like pip. Maintainers donāt want to host releases, and users want to be told exactly what to do.
It needs to be dead simple. No extra or very little extra configuration. For a setuptools project, just use whatever available from setup.py and make everything else optional.
Dead simple to release, like Twine to PyPI.
I donāt know exactly how much effort that would be, but assuming the packaging part can be worked out (that would be the most problematic, I imagine), tooling and infrastructures might be easier since there are already related works to take ideas from.
The big advantage of a standalone app deployment is properly integrating your dependencies. Things that I normally do in mine include flattening the directory hierarchy, zipping as many dependencies as possible, combining DLLs into a single directory, and replacing python.exe with my own (or adding mine alongside, for the sake of libraries that require starting new processes). Capturing this with existing tooling is really quite difficult. However, itās also the best way to deal with DLL Hell (on Windows, maybe others?) as experienced by scipy et al.
Iāve seen varying experiences with Briefcase. Some have been excellent, others have hit their own edge cases. I canāt decide whether itās best to get behind it or to learn from it and push in a new direction (similarly for zipapp and the wild variety of packing tools).
Iāve got a few teams at work that Iām working with who need this, though, so I expect Iāll come up with a concrete set of requirements sooner or later. It will be Windows-centric, unfortunately, but I suspect the required approaches will also have to be fairly platform specific and weāll mainly want to align on configuration rather than implementation.
This gets a bit tricky as that gets us into the binary executable distribution game which is historically the apt/Homebrew/etc. type realm. Iām not saying we couldnāt do it specifically for Python apps, but it is a new arena.
Iām starting to think that something along these lines and/or moving towards static compilation with C extensions + freezing is the way to go (but maybe the single binary solution is something to target after since we all know that compilation of some popular libraries is not exactly easy ). But coming up with tooling that would allow sending someone a zip where you say āunzip this and then run/double-click this thingā would be fantastic and also help in the educational world where they have been dying for a way to have e.g. a fifth grader share his game with his friends easily.
Performance. Less files to decompress on install, files to index, which can matter on some file systems/platforms.
Nice! We are migrating all of our pexes to shivs. Zipapps are great in environments where you know you have the Python executable, stdlib, etc. and can install all your dependencies. But thereās also a usecase for distribution of everything, including the interpreter, in a single file. There are snaps and xars, freezing of various flavors, etc. There will always be tradeoffs in size, complexity, duplication of dependencies (sort of the anti-distro problem). Iām not sure any particular solution will be general enough to include with Python. There are just too many competing requirements for a single solution to win, IMHO.
The biggest problem with shivs is the usual one - not being āproperā exes means that they donāt work seamlessly on Windows. See my essay here on why exes are the only real solution.
Speaking of *nix, is there an established way to build a ārelocatableā CPython like Windowsās embedded distribution? Itād be extremely helpful if weāre going the flat directory direction, but I have yet to find a definitive solution.
From a Unix perspective, why are pipsi and pipx considered āworkaroundsā? I havenāt looked at their implementations, but the basic idea of installing an app and its dependencies into a separate directory somewhere, and then linking its launcher script to a directory on $PATH seems perfectly legitimate to me.
I think the idea of standalone applications, with dependencies isolated from any other software which might be installed on the system, often gets mixed up with the idea of single file applications, with everything bundled into one binary. There are other reasons you might want a single file application, but you donāt need one to achieve a standalone application. E.g. Pynsist produces installers which aim for a standalone application as far as practical, but itās still many files in a directory.
Technically, I think the biggest limitation for standalone applications is how you include the interpreter. Tools like pyenv and conda provide ways to programmatically get a specified interpreter version in a specified path, but theyāre not (or not yet?) the kind of de-facto standard people build higher level tools around. Python.org provides the āembeddableā builds for Windows, but they donāt include the MSVC runtime DLLs, so they donāt work out of the box on all systems.
Thereās no way to install multiple versions of the same application. This is why I always go back to managing my own virtual environments.
No real way to ensure two installations are the same. Dependencies of Python packages have open specifiers like >=3.0 (or they should), so in (admittedly rare) situations you have the privilege of hunting down weird bugs caused by subtal package combinations.
Essentially all problems come from the fact that these tools are built on PyPI, and PyPI (at least as it currently stands) is fundamentally built for installing packages to work together, and an inappropriate platform to deploy applications. This is my definition of a workaround: It look the part, but is built on sand, and eventually breakd down if you poke hard enough.
Note that a standalone application does not mean a single executable (the other way is true). Most Windows and Mac applications consist of mutiple files extracted into a directory, but they all qualify as standalone. The fundamental attribute is whether those files can work on their own, as an entity.
Specifically regarding the package combination problem, various package managers solve this in different ways.
Curation (e.g. APT, RPM). A group of people hand-pick a set and promise the combination works, and you trust them.
Locking. The maintainer supplies to the index server the preferred versions they want. I believe this is how Ruby solves this? A poor analogy for Python is to supply both pyproject.toml and Pipfile.lock in your package, and pyproject.toml is consulted by pip install, and Pipfile.lock pipx install.
Distinct packaging formats. Youād submit sdist and wheel to be used as a library, and another package with dependencies vendored to be used as a standalone application.
And all of them (maybe more) have their tradeoffs.
I have a few random thoughts that may be useful for this conversationā¦
I think a common deficiency with many Python standalone application tools is that they require having a Python distribution on the system. For a ājust worksā experience, I think it is important for app distributions to provide the runtime. In this case a Python distribution. This is a āhard problemā because building portable binaries is hard. Extra so for CPython because of limitations in its build system and 3rd party library dependencies. (I have the python-build-standalone project (https://github.com/indygreg/python-build-standalone) to make this easier.) Tools like Briefcase punt on this problem and delegate to a shebang or a Python launcher.
One of my discoveries with PyOxidizer is the significant performance speedup from importing modules from memory (https://gregoryszorc.com/blog/2018/12/28/faster-in-memory-python-module-importing/ and https://gregoryszorc.com/blog/2019/01/06/pyoxidizer-support-for-windows/). Unfortunately, importing modules from something that isnāt a real filesystem can confuse all kinds of Python packages which assume <module>.__file__ isā¦ a filesystem path. There may need to be some PEP work to formalize that __file__ and __path__ are abstract concepts not bound to traditional filesystems. Maybe allow them to be URIs or something, Iām not sure. I think the importing abstractions with the decoupling of finders and loaders are mostly good and flexible enough. But they do seem to be rather traditional filesystem focused. And code in the wild reflects this. Auditing tools should probably be taught to look for __file__ abuse and to steer people towards using the resources API (importlib.abc.ResourceReader) if they arenāt doing so already.
Something else that I would find extremely useful for PyOxidizer (and would be useful for any standalone packaging tool) is to make it easier for these tools to find all required resources (namely package dependencies) for packaging. Today, tools tend to do things like invoke setup.py to populate a virtualenv then consume that. Or they inject themselves into the setup.py process to gather the information they need. It would be super useful to get a build system metadata dump from the packaging tool so app packaging tools could consume this and do useful things. I think some of the recent work around abstracting the build process can help here. But Iām not too familiar with that work. Keep in mind that tools like PyOxidizer need to know extensively about C extensions and binary dependencies so they can be compiled in a portable manner. Complex setup.py scripts that invent their own build steps which can do complicated things undermine standalone application packaging.
Another feature that would be useful is a standard mechanism for declaring app packaging details. Today, various tools have to invent their own setup.py extensions or config files for packaging. It would be useful to have a standard grammar such that most applications/libraries could define the settings once and be packaged using N tools. This may not get the long tail of specialized applications (like say Mercurial). But Iād like to think it would handle 95% of applications.
Iām trying to make https://github.com/indygreg/python-build-standalone and PyOxidizer loosely coupled. The output of python-build-standalone contains a machine-readable file describing the Python distribution and its settings (see the README.rst in that repository). This allows PyOxidizer to theoretically consume any Python distribution (CPython, PyPy, etc). It might be worth standardizing such a format for packaging Python distributions themselves so Python app packaging tools can achieve greater flexibility in the Python distribution they use. e.g. if all you need to do is point the packaging tool at a URL for a Python distribution in a standard format, with a trivial change you could replace CPython with PyPy.
I know Iām forgetting a few things. But hopefully this mini brain dump is useful to the discussion.
A community member familiar with āConstructorā asked me to chime in here. Constructor is a tool for creating installers that are self-contained. I donāt think it really fits the use case declared here, but maybe for some related use cases it works. Unlike pipenv or something like that, youād ship an artifact that would contain all the files necessary to create an environment on the other end, rather than a list of dependencies to install. Constructor requires conda to run, but does not assume that conda ends up in the created installer.
Weāve experimented with nuitka and pyinstaller for getting a standalone conda executable, but ultimately shared libraries were not possible to bundle into a single standalone executable. Nuitka produced a nice folder that had our bundled executable along with a bunch of the core python stack. It was pretty close to being a standalone executable, but would have taken a lot more work to figure out static linking and such to produce a single exe. Having better tools for this would be extremely helpful.
) I have all of them bundled up using shiv at the moment, so (a) I know they work standalone, and (b) I know they are useful to have as standalone apps.
