A mess with SOABI tags on musllinux

Opening a discourse thread b/c I’m not even sure which all parts of the ecosystem need to know about this and fix stuff. But musllinux wheels are currently kind of busted, in a surprising way.

This was discovered because coveragepy started distributing musllinux wheels, and ran into a weird issue where some Python builds would load the extension module, and some wouldn’t:

It turns out that the issue is that the files in the wheel are named like tracer.cpython-39-x86_64-linux-gnu.so. That gnu in the SOABI tag means “glibc”, which is clearly wrong. But it turns out that many CPython builds on musl still think this it the correct SOABI tag. And others – in particular, the official Python packages in the latest alpine release – have a correct x86_64-linux-musl tag. (To check the SOABI tag, use sysconfig.get_config_vars("SOABI").) And the SOABI tag is used both when building extensions, and when loading them – so the Python you build with has to have the same SOABI as the Python you import on, or else it can’t find the file.

I think what where we need to end up is everyone agreeing on using the correct musl tag. But to get there we probably need changes in:

  • CPython (if they haven’t happened already? I don’t know whether the correct tag on latest alpine is b/c of a CPython fix or a downstream distro patch)
  • the musllinux build image (which currently generates -gnu tagged extensions)
    • and also every other alpine-based python docker image
  • auditwheel to make sure everything stays fixed?
  • maybe pip to avoid installing musllinux wheels on broken CPythons, while we’re waiting for images to get updated?

It’s messy, and I’ve hit my limit on this project, so hopefully someone else can take up coordinating things from here. But I did want to get the word out there.

Alpine has patched SOABI (bpo-43112.patch\python3\main - aports - Alpine packages build scripts) and raised a bug with CPython (Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker).

The way this manifests in practice is that currently anyone building a non-abi3 wheel using musllinux will have it fail to load on Alpine 3.14’s distribution-provided Python 3.9. (abi3 wheels may not work either; cryptography’s specific use of abi3 is through cffi and pyo3, which could be different, I haven’t confirmed.)

Alpine 3.12 and 3.13’s Python 3.8 do not have this patch and work with -gnu SOABI (which is consistent with upstream Python and therefore the musllinux images).

1 Like


Let me ask something, because we have released psycopg 3 binary packages for musl linux a few days ago and I’d like to know if everything is working well.

As far as I can see, it does. The libraries have a -gnu suffix but they work ok on Alpine Linux 3.14 with Python 3.10.

$ docker run --rm -ti python:3-alpine sh
/ # cat /etc/os-release 
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.14"
/ # pip install psycopg[binary]
Collecting psycopg[binary]
  Downloading psycopg-3.0.4-py3-none-any.whl (140 kB)
     |████████████████████████████████| 140 kB 2.3 MB/s 
Collecting psycopg-binary==3.0.4
  Downloading psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 9.7 MB/s 
Installing collected packages: psycopg-binary, psycopg
Successfully installed psycopg-3.0.4 psycopg-binary-3.0.4

/ # python 
Python 3.10.0 (default, Oct 26 2021, 23:20:47) [GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg._cmodule._psycopg
<module 'psycopg_binary._psycopg' from '/usr/local/lib/python3.10/site-packages/psycopg_binary/_psycopg.cpython-310-x86_64-linux-gnu.so'>

I understand it shouldn’t have worked? Why would psycopg work and cryptography not?

Tested with the python:3.9-alpine image right now, which worked too.


…ah, I understand. The python:3.X-alpine have Python built by Python and have a -gnu suffix. The Python obtained by alpine:latest via apk add python3 has a -musl suffix and indeed the binary package doesn’t work:

/ # python3
Python 3.9.5 (default, May 12 2021, 20:44:22) 
[GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
Traceback (most recent call last):
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: cannot import name 'pq' from 'psycopg_binary' (/usr/lib/python3.9/site-packages/psycopg_binary/__init__.py)
- couldn't import psycopg 'python' implementation: libpq library not found

Now I know better what’s going on, but I’m properly confused about how to proceed in regards to which package to distribute. I understand that

  • the packages currently released now work on current python-alpine docker images but will break as soon as BPO 43112 will be fixed.
  • after it’s fixed, it will be possible to build new packages using newly released cibuildwheel and related images, but the new packages will not work on python:3.10.0-alpine, python:3.9.9-alpine and previous ones
  • we can’t release both the images because they have exactly the same tag.

Uhm… what’s the best course of action?

It is worth noticing that the problem is only in the name of the library: creating a symlink to the existing library is enough to import the module (and I tested it works):

(.venv) piro@baloo:~/dev/psycopg3$ d run --rm -ti --volume `pwd`:/src --workdir /src -e PSYCOPG_TEST_DSN -e PGHOST= -e PGUSER=`whoami` alpine sh

/src # apk update
OK: 14942 distinct packages available

/src # apk add python3 py3-pip tzdata
OK: 80 MiB in 53 packages
/src # pip3 install psycopg[binary]

/src # pip3 install -U pip
Successfully installed pip-21.3.1

/src # pip install psycopg[binary]
Successfully installed psycopg-3.0.4 psycopg-binary-3.0.4

/src # cd /usr/lib/python3.9/site-packages/psycopg_binary
/usr/lib/python3.9/site-packages/psycopg_binary # ln -s pq.cpython-39-x86_64-linux-gnu.so pq.cpython-39-x86_64-linux-musl.so 
/usr/lib/python3.9/site-packages/psycopg_binary # ln -s _psycopg.cpython-39-x86_64-linux-gnu.so _psycopg.cpython-39-x86_64-linux-mus
/usr/lib/python3.9/site-packages/psycopg_binary # ls -l
total 1584
-rw-r--r--    1 root     root           421 Nov 23 00:11 __init__.py
drwxr-xr-x    2 root     root          4096 Nov 23 00:11 __pycache__
-rw-r--r--    1 root     root       1267896 Nov 23 00:11 _psycopg.cpython-39-x86_64-linux-gnu.so
lrwxrwxrwx    1 root     root            39 Nov 23 00:13 _psycopg.cpython-39-x86_64-linux-musl.so -> _psycopg.cpython-39-x86_64-linux-gnu.so
-rw-r--r--    1 root     root          2596 Nov 23 00:11 _psycopg.pyi
-rw-r--r--    1 root     root        332640 Nov 23 00:11 pq.cpython-39-x86_64-linux-gnu.so
lrwxrwxrwx    1 root     root            33 Nov 23 00:13 pq.cpython-39-x86_64-linux-musl.so -> pq.cpython-39-x86_64-linux-gnu.so
-rw-r--r--    1 root     root             0 Nov 23 00:11 py.typed
-rw-r--r--    1 root     root           251 Nov 23 00:11 version.py

/usr/lib/python3.9/site-packages/psycopg_binary # cd -

/src # python3
Python 3.9.5 (default, May 12 2021, 20:44:22) 
[GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg._cmodule._psycopg
<module 'psycopg_binary._psycopg' from '/usr/lib/python3.9/site-packages/psycopg_binary/_psycopg.cpython-39-x86_64-linux-musl.so'>

/src # pip install -e ./psycopg[test]
Obtaining file:///src/psycopg
Successfully installed attrs-21.2.0 coverage-6.1.2 importlib-metadata-4.8.2 iniconfig-1.1.1 mypy-0.910 mypy-extensions-0.4.3 pluggy-1.0.0 pproxy-2.7.8 psycopg-3.0.5.dev0 py-1.11.0 pytest-6.2.5 pytest-asyncio-0.16.0 pytest-cov-3.0.0 pytest-randomly-3.10.2 tenacity-8.0.1 tomli-1.2.2 typing-extensions-4.0.0 zipp-3.6.0

/src # pytest -m not\ slow
======================================================= test session starts ========================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
Using --randomly-seed=4128270608
libpq wrapper implementation: binary
libpq used: 120009
libpq compiled: 120009
rootdir: /src, configfile: pyproject.toml, testpaths: tests
plugins: randomly-3.10.2, asyncio-0.16.0, cov-3.0.0
collected 3376 items / 347 deselected / 1 skipped / 3028 selected                                                                  

tests/types/test_composite.py .................................................                                              [  1%]
tests/types/test_bool.py ...............                                                                                     [  2%]

maybe we should fix manylinux/cibuildwheel to create a symlink for “the other tag”, whichever it is?

1 Like

I think (I’ve opened a tracking issue in cibuildwheel, Tracking issue for broken musllinux wheels · Issue #934 · pypa/cibuildwheel · GitHub) that likely this should be handled by auditwheel. It could correct -gnu to -musl, and then provide compatibility symlinks. On fixing a bug of this magnitude, it would have been nice if CPython could have accepted both extensions for now, and only been strict about Python 3.11+.


Doesn’t look like there’s anything we can do in cibuildwheel, for now. We build with stock CPython via manylinux. If these wheels don’t work on Alpine Python, then that unfortunately feels like an incompatibility introduced by Alpine, not CPython or the PyPA toolchain. While I agree that the -gnu suffix is wrong, it should probably be fixed on a CPython version bump, so we maintain wheel compatibility.

Can this problem be solved in Alpine by doing new releases of their Python packages that don’t include this patch?

Had a call with @henryiii and we are fixing this in Alpine by adding a fallback for -gnu if -musl is not found. That should fix the immediate problem with alpine 3.14 (and now also 3.15). But it also means that vanilla built python 3.9 cannot use distro packages with python modules (eg apk add py3-foobar). But I guess that distro packages are not recommended in general, so its probably ok.

I wrongly assumed that Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker would be fixed sooner than later, in python 3.10. Would be nice if this could be properly fixed in 3.11.


I’ve assigned the PR to myself to help close it out. We can discuss the PR on GitHub where I have left a question for you.