Opening a discourse thread b/c I’m not even sure which all parts of the ecosystem need to know about this and fix stuff. But musllinux wheels are currently kind of busted, in a surprising way.
This was discovered because coveragepy started distributing musllinux wheels, and ran into a weird issue where some Python builds would load the extension module, and some wouldn’t:
It turns out that the issue is that the files in the wheel are named like tracer.cpython-39-x86_64-linux-gnu.so. That gnu in the SOABI tag means “glibc”, which is clearly wrong. But it turns out that many CPython builds on musl still think this it the correct SOABI tag. And others – in particular, the official Python packages in the latest alpine release – have a correct x86_64-linux-musl tag. (To check the SOABI tag, use sysconfig.get_config_vars("SOABI").) And the SOABI tag is used both when building extensions, and when loading them – so the Python you build with has to have the same SOABI as the Python you import on, or else it can’t find the file.
I think what where we need to end up is everyone agreeing on using the correct musl tag. But to get there we probably need changes in:
CPython (if they haven’t happened already? I don’t know whether the correct tag on latest alpine is b/c of a CPython fix or a downstream distro patch)
the musllinux build image (which currently generates -gnu tagged extensions)
and also every other alpine-based python docker image
auditwheel to make sure everything stays fixed?
maybe pip to avoid installing musllinux wheels on broken CPythons, while we’re waiting for images to get updated?
It’s messy, and I’ve hit my limit on this project, so hopefully someone else can take up coordinating things from here. But I did want to get the word out there.
The way this manifests in practice is that currently anyone building a non-abi3 wheel using musllinux will have it fail to load on Alpine 3.14’s distribution-provided Python 3.9. (abi3 wheels may not work either; cryptography’s specific use of abi3 is through cffi and pyo3, which could be different, I haven’t confirmed.)
Alpine 3.12 and 3.13’s Python 3.8 do not have this patch and work with -gnu SOABI (which is consistent with upstream Python and therefore the musllinux images).
Let me ask something, because we have released psycopg 3 binary packages for musl linux a few days ago and I’d like to know if everything is working well.
As far as I can see, it does. The libraries have a -gnu suffix but they work ok on Alpine Linux 3.14 with Python 3.10.
$ docker run --rm -ti python:3-alpine sh
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.2
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ # pip install psycopg[binary]
Collecting psycopg[binary]
Downloading psycopg-3.0.4-py3-none-any.whl (140 kB)
|████████████████████████████████| 140 kB 2.3 MB/s
Collecting psycopg-binary==3.0.4
Downloading psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl (2.2 MB)
|████████████████████████████████| 2.2 MB 9.7 MB/s
Installing collected packages: psycopg-binary, psycopg
Successfully installed psycopg-3.0.4 psycopg-binary-3.0.4
/ # python
Python 3.10.0 (default, Oct 26 2021, 23:20:47) [GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg._cmodule._psycopg
<module 'psycopg_binary._psycopg' from '/usr/local/lib/python3.10/site-packages/psycopg_binary/_psycopg.cpython-310-x86_64-linux-gnu.so'>
>>>
I understand it shouldn’t have worked? Why would psycopg work and cryptography not?
Tested with the python:3.9-alpine image right now, which worked too.
…ah, I understand. The python:3.X-alpine have Python built by Python and have a -gnu suffix. The Python obtained by alpine:latest via apk add python3 has a -musl suffix and indeed the binary package doesn’t work:
/ # python3
Python 3.9.5 (default, May 12 2021, 20:44:22)
[GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
Traceback (most recent call last):
...
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: cannot import name 'pq' from 'psycopg_binary' (/usr/lib/python3.9/site-packages/psycopg_binary/__init__.py)
- couldn't import psycopg 'python' implementation: libpq library not found
Now I know better what’s going on, but I’m properly confused about how to proceed in regards to which package to distribute. I understand that
the packages currently released now work on current python-alpine docker images but will break as soon as BPO 43112 will be fixed.
after it’s fixed, it will be possible to build new packages using newly released cibuildwheel and related images, but the new packages will not work on python:3.10.0-alpine, python:3.9.9-alpine and previous ones
we can’t release both the images because they have exactly the same tag.
Uhm… what’s the best course of action?
It is worth noticing that the problem is only in the name of the library: creating a symlink to the existing library is enough to import the module (and I tested it works):
I think (I’ve opened a tracking issue in cibuildwheel, Tracking issue for broken musllinux wheels · Issue #934 · pypa/cibuildwheel · GitHub) that likely this should be handled by auditwheel. It could correct -gnu to -musl, and then provide compatibility symlinks. On fixing a bug of this magnitude, it would have been nice if CPython could have accepted both extensions for now, and only been strict about Python 3.11+.
Doesn’t look like there’s anything we can do in cibuildwheel, for now. We build with stock CPython via manylinux. If these wheels don’t work on Alpine Python, then that unfortunately feels like an incompatibility introduced by Alpine, not CPython or the PyPA toolchain. While I agree that the -gnu suffix is wrong, it should probably be fixed on a CPython version bump, so we maintain wheel compatibility.
Can this problem be solved in Alpine by doing new releases of their Python packages that don’t include this patch?
Had a call with @henryiii and we are fixing this in Alpine by adding a fallback for -gnu if -musl is not found. That should fix the immediate problem with alpine 3.14 (and now also 3.15). But it also means that vanilla built python 3.9 cannot use distro packages with python modules (eg apk add py3-foobar). But I guess that distro packages are not recommended in general, so its probably ok.