Multi-arch venv: sound or bad idea?

Packaging
1

What do you think about a virtual env that’s suitable for multiple architecture simultaneously?

An example:

venv/lib/python3.13/site-packages/yaml/
  __init__.py
  _yaml.cpython-313-aarch64-linux-gnu.so*
  _yaml.cpython-313-x86_64-linux-gnu.so*
  _yaml.cpython-313-riscv64-linux-gnu.so*
2

How would you even get there? Do you have some fat-binary Python installation? Or separate Python installations that have python -m venv venv-ed into the same location?

I’d have thought that multi-python-version venvs would come first (although I can’t imagine how that would work either).

3

For now I’d build it manually.

The idea is to build something like a portable application that uses system Python and stdlib; the app brings own source code and deps for all platforms.

4

I love it, and would like to see it done more often. I’ve already helped a few projects to distribute pre-built libraries like this (they are added to PYTHONPATH manually when launching, rather than being installed from a wheel). Works for multiple versions too.

Unfortunately though, it doesn’t really help users. Only distributors (or people building apps, like those mentioned above).

A tool to merge separate wheels into a “fat” install (not necessarily a venv - a regular directory would be more common in my experience) would be used.

1 Like
5

It would be a fantastic idea, but unfortunately it’s not going to work at the moment. ABI3 extension modules don’t have platform and arch information in their file names. The suffix is just ‘.abi3.so’. Packages like cryptography, psutil, and a bunch of Rust / PyO3-based projects have ABI3 binaries.

>>> import importlib.machinery
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cpython-312-x86_64-linux-gnu.so', '.abi3.so', '.so']

Several packages include auxiliary shared libraries without platform identifiers, too. The most famous example is PyTorch:

torch/_C.cpython-311-x86_64-linux-gnu.so
torch/lib/libaoti_custom_ops.so
torch/lib/libbackend_with_compiler.so
torch/lib/libc10.so
torch/lib/libc10_cuda.so
torch/lib/libc10d_cuda_test.so
torch/lib/libcaffe2_nvrtc.so
torch/lib/libjitbackend_test.so
torch/lib/libshm.so
torch/lib/libtorch.so
torch/lib/libtorch_cpu.so
torch/lib/libtorch_cuda.so
torch/lib/libtorch_cuda_linalg.so
torch/lib/libtorch_global_deps.so
torch/lib/libtorch_python.so
torch/lib/libtorchbind_test.so
3 Likes
6

We can fix this upstream pretty easily. Adding .abi3-x86_64-linux-gnu.so ahead of the existing .abi3.so shouldn’t break anything (certainly won’t break anything that’s following The Rules :wink: ).

Getting build backends to produce the correct outputs is a bigger ask, but with the newer range of backends coming out it’s more possible than ever.

At least on Windows it’s safe to simply rename an extension module to add/remove the ABI tag. Does that work on Linux too?

1 Like
7

Extension modules and libraries are not the only issue. There is nothing preventing a pure Python project to distribute Python modules with the same name but different content on different platforms. For example, the os.path module loads a different implementation at runtime based on the system where it is imported, but the same could have been implemented distributing a different Lib/os/path.py on Windows and on POSIX systems.

I’m not aware of any project doing this, but it is not forbidden or explicitly discouraged.

8

Just a terminology thing, but I wouldn’t call that a virtual environment but a directory of code that works cross-platform. It isn’t disposable and it’s specifically relocatable which virtual environments aren’t.

2 Likes
9

Not exactly what you’re looking for, but if you aim to distribute a Python program to all platforms no matter the dependencies, you may want to take a look to PyEmpaq.

10

How does the PyEmpaq handle the issues that others have outlined?

  • deps included for one os but not the other?
  • shared objects that don’t include arch tag?
  • transient shared objects like libpcre2-8.so.0?
11

PyEmpaq will create a virtualenv in the final environment where you’re running the packed software, pip install all the needed dependencies there, and run the “payload” software.

So, no “deps included” (just the requirements to run), and no shared/transient objects, pip will install whatever it corresponds in the final system.

1 Like