AppImages of manylinux Pythons

This is work in progress.

I am producing AppImages of the manylinux /opt/python installs by relocating them from the Docker images. These AppImages can be used as (almost) isolated and self contained Python runtimes.

The corresponding GitHub project is python-appimage. The Python runtimes are available from the releases.

This is interesting, thanks for sharing! You might want to cross-post at Making Python relocatable on Linux as well.

Can you give us a summary of the current issues and why you say “(almost) isolated”?

For those of us that are unfamiliar with AppImages, can you give us some detail about that format? What is required to support running Python from AppImages?

Hi @niess, following up here, are you able to answer my questions above? Thanks!

Hello @dustin, sorry for the delay.

Concerning AppImage one can find an overview over there. Note that it is Linux only. In practice it allows to compress a binary and its deps as a single executable file that is extracted and mounted at runtime to a temporary folder. It requires fuse to work and will fail on WSL1. However, as a fallback one can also extract the image to a fixed directory and then run its content. This mode works on WSL1 as well and has no specific requirements as far as I know. Actually I find this second mode more convenient for my use case, e.g. in order to pip install extra packages. Note that I could use a simple compressed tarball as well in this case, instead of an AppImage.

You can try one of the Python AppImages as following:

wget https://github.com/niess/python-appimage/releases/download/\
python3.8/python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage
chmod +x python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage

./python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage

In order to extract the content and work from it one can do:

./python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage --appimage-extract
mv squashfs-root python3.8.3
ln -s python3.8/AppRun my-python

Then a package can be installed from PyPI as ./my-python -m pip install the-package. The package is installed to the extracted folder, i.e. python3.8.3 in this case.

I don’t have much issues currently for my use case. I use it in order to distribute a ready to go Python env to my Physicists colleagues. They can also easily deploy it to a production server in order to process/produce some data. Some other people use it in order to distribute python based apps as a single exec file.

Note that there is no real isolation with the AppImages that I am producing. Basically it just runs from a folder where “everything” is packaged. The hard work is actually for the developer to tweak its app such that it can run that way. Basically it requires to be relocatable. So Indeed it is related to the topic that you pointed at.

For Python to run that way I had to:

  • Go through all C extension modules and package their relevant binary deps (shared libs) in the AppImage. This is done by copying the library and setting a relative RPATH using $ORIGIN Some shared libs, e.g. libc are blacklisted during this process, according to the AppImage exclusion list.

  • I do the same for the Python binary itself, i.e. bundle its shared libs and set a relative RPATH using $ORIGIN.

  • Bundle shared data for TCl/Tk.

  • Set a relative shebang for the pip script.

The details can be found in the relocate_python function python-appimage package (sorry I can’t post the link apparently).

In addition, I bundle a dedicated sitecustomize.py which removes system paths from the package search path, if the AppImage Python is not located there. This is to isolate the AppImage Python from the system one. Otherwise it would see system packages, e.g. installed as sudo pip install anything. This script also patches the shebangs of scripts installed with pip to use a relative location instead of an absolute one.