Wheels for musl (Alpine)

Thank you for working on this!

Correct.

I think Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker needs to be acknowledged and fixed first. It will be difficult to add support for binary wheels unless upstream python recognizes that the musl vs glibc ABI is more than the calling convention.

we could have manylinux_glibc_$VERSION and manylinux_alpine_$VERSION.

I don’t think manylinux_alpine_$VERSION linux makes sense. It makes much more sense to do manylinux_musl_$VERSION. I dont think we should use alpine in there at all, since there are more musl libc based distros out there like void linux, Gentoo, adelie linux, openwrt and sabotage linux.

I don’t think it needs to be fixed, there are other ways to detect whether a Python executable is linked against musl instead of glibc. Although it’d certainly make things easier if Python can encode that information at compile time.

As mentioned above, manylinux_musl_$VERSION is also not appropriate here, given the decision on PEP 600. An appropriate platform tag should not contain the string manylinux at all.

1 Like

I also have a way to detect musl libc at runtime in a PR for find_library. However I think it would be much nicer if it could be detected/configured at compile time.

It has to be based on something, right? My current intuition on the matter is that it would be the easiest to point at some particular version of Alpine and say: this is what you need to limit yourself to - in the same way manylinux does with CentOS. Is there a better candidate for it than Alpine?

The PEP effectively says “manylinux == manylinux_glibc, but the latter is confusing so let’s not do that.” However, this is based on the historical details and so-far exclusive relevance of that tag.

If indeed the various distros that @ncopa mentions are similar enough to warrant having a many* standard for, then I think those previous considerations have much less weight vis-à-vis having a broadly usable image (rather than an alpine-specific one). Whether it’s manymusl or manylinux_musl is then only bikeshedding (as long as it doesn’t interfere with existing packages and infrastructure).

I believe what was meant was that the manylinux_alpine name is bad. musl is a libc used across quite a few Linux distros. Using alpine (the distro) as a base is reasonable, but naming the specification as <something>_alpine isn’t.

They should be. I am basing myself off of the manylinux2014 specification, to try and see what is similar enough between these environments.

Supported architectures are all covered by musl:

x86_64
i686
aarch64
armv7l
ppc64
ppc64le
s390x <-- Void Linux doesn't provide this, but Alpine and Gentoo do

Regarding libraries:

# provided by modern GCC
libgcc_s.so.1
libstdc++.so.6
# provided by libc.so on musl, but all functionality is available
libm.so.6
libdl.so.2
librt.so.1
libc.so.6
libnsl.so.1
libutil.so.1
libpthread.so.0
libresolv.so.2 <-- this is the last one provided by libc.so
# provided by Xorg and related projects, haven't changed ABI in a long time
libX11.so.6
libXext.so.6
libXrender.so.1
libICE.so.6
libSM.so.6
# provided by glvnd, very stable
libGL.so.1
# provided by GLib, also very stable
libgobject-2.0.so.0
libgthread-2.0.so.0
libglib-2.0.so.0

Adding OpenSSL libraries to this list might even be possible (depending on how the OpenSSL 3 release is going to work), given that Alpine and Gentoo default to it, and Void Linux is moving to it as well. But since manylinux2014 doesn’t include it, there’s no reason for us to include it either.

Creating a package list for them should be pretty simple, once a PEP rolls out. Since musl doesn’t support symbol versioning, that part isn’t a concern to us.

Given that this is going to be a modern standard, caring only about recent Python 3 versions also seems reasonable to me.

All that said, I believe the biggest concern is the changes between musl 1.1.x and 1.2.x, specifically for 32bit devices. I would say this matters, since armv7l is likely (?) to be a supported platform. musl 1.2.0 implemented the time64 transition, which changed time_t to be 64 bits on all architectures. While ABI compatibility was maintained with older binaries, this change means that any binary built on musl>=1.2.0 is unlikely to work with musl<1.2.0, and, more importantly, libraries that use time_t somewhere in their external API can end up subtly miscompiled. Therefore, I would argue for this new standard, in the interest of future proofing (we all want things to work post 2038, I think :P), to use some musl>=1.2.0.

2 Likes

Do you happen to know when Alpine transitioned to musl 1.2.x? Making the version requirement sounds like a good technical decision, but it wouldn’t be very helpful if most people out there can’t use it.

Should be alpine 3.13: Alpine 3.13.0 released | Alpine Linux

Released last month :fearful: So it’ll probably not help many people unless we take this very slowly (which is also not a good thing).

Maybe for an initial release we should try and stick with 1.1.x (not sure which one to pick, would appreciate input from @ncopa), then? Having manymusl_major_minor_arch like it’s done for glibc doesn’t sound too terrible.

Since musl has already gone through the pain of implementing time64, it would be nice to take advantage of this work, but if using an older musl might enable everything to just work on older alpine versions which are likely to still be in use, it can be a worthy tradeoff. That said, armv7l and i686 (which I forgot to mention at previous comments) are still valuable test platforms, due to all the ARM SBCs out there using 32-bit userland (which is unfortunately already left out of CI testsuites for many projects).

Thanks for the list @enriconr, that’s already a first big step!

At the time of the manylinux2010 saga, the 32bit images were also an issue and were dropped for the initial roll-out. It then took roughly half a year for them to be added.

What I’m trying to say is that it’s probably a much more feasible goal to aim for 64bit only initially (without closing the door to 32bit), and to not block the whole effort on some ancient hardware.

In a similar vein (and in particular with the groundwork of the “perennial” manylinux PEP 600), it’s completely possible to build these images out of order, so a presumptive manymusl_1_2 could come before manymusl_1_1.

I’m thinking though that a putative manymusl proposal might not be usable in the similar forward-compatible manner as manylinux. Here’s a comment from the creator/maintainer of musl about ABI compatibility, and how musl is taking a slightly different path than glibc (ABI stability is guaranteed, but that’s not the same as “guaranteed it will work”)

This is different from the glibc approach, which is to use symbol
versioning to attempt to retain “bug-compatibility” with the version
of glibc the application was linked with. Such a system forces new
application binaries that want to be able to run on systems with old
glibc to link against the old glibc, and thereby get the buggy
behaviors even if they’re running on a system without the bugs. Myself
and most of the musl community I’m aware of consider this entirely
unreasonable, and that’s why musl doesn’t do it.

I should note that this approach is in part what makes musl attractive. Since they are able to avoid the binary bloat that comes with keeping old versions of interfaces around, they remain a very tiny library while still having lots of implemented functionality.

That said, I see what you mean. For an example of what glibc does, one need only look at memcpy(3):

In glibc 2.14, a versioned symbol was added so that old binaries (i.e., those linked against glibc versions earlier than 2.14) employed a memcpy () implementation that safely handles the overlapping buffers case (by providing an “older” memcpy () implementation that was aliased to memmove(3)).

While in this case it is a cheap alias, there are many cases which result in a lot of code duplication. musl has ended up carrying some “bloat” on 32bit systems now, due to the time64 changes, but it is strictly for ABI compat, not for bug compatibility.

Overall, given that musl’s stance on this is part of what brings users to it, I would argue that manymusl should adopt it. Binary wheels are guaranteed to not throw “symbol lookup error” during dynamic linking or have weird ABI incompatibility, but bug compatibility is guaranteed only if you’re running the same musl version as specified in manymusl_major_minor_... (musl versions with 3 fields, actually, so we’d need some other naming).

@ericonr do you happen to have any source or reference for musl’s versioning scheme? I cannot find it in their wiki nor the “Official manual” section.

My reading of Rich’s post is that generally, you can be pretty confident that it works to build against musl version X and run on version X+1, with the only exception being if the library was somehow depending on some bug in version X that got fixed in X+1. I see why Rich emphasizes that exception, because Rich is a very precise guy and musl does put less effort into bug-for-bug compatibility than glibc, but really this is the same caveat that applies to every upgrade process ever, and I don’t think we need to worry about it being a major issue for musl.

The other thing he’s pointing out is that in many cases, you can build against musl X+1 and actually get a binary that runs on version X (!). Unfortunately, this isn’t super useful to us at the spec level: “many cases” is not the same as “all cases”, so we can’t just assume that building on any version of musl will produce a binary that works on all versions of musl. We’ll still need some kind of monotonic “version” counter to use in the wheel tag. (Though auditwheel could potentially be smart enough to detect cases where a binary built against musl X+1 will work on musl X, and tag the wheel appropriately.)

AFAIK, the biggest challenge to doing this is that we don’t yet have a concrete proposal for how to reliably figure out what version of musl is running. The best option I’ve seen is to first manually parse the python binary’s ELF header to see if the string musl appears and what the path to libc.so is, and then invoking libc.so as a child process and parsing the text it prints in the output. This is like… incredibly janky, but probably doable? Also, it will fail if the python binary itself was statically linked – but maybe that’s OK? If python was statically linked against musl, then I think it might not be able to load extension modules that are dynamically linked against musl anyway?

The other option would be to give up on using musl itself as the clock, and instead use the distro. It’s very easy to figure out whether you’re running on alpine and what version you have: just check the standardized os-release file. The downside is that then wheels would be specific to alpine, and wouldn’t automatically install on other musl-based linuxes. The upside would be that wheels could potentially depend on features that alpine provides beyond just musl… though I’m not sure whether there are any, since alpine is so minimalist.

As a general piece of advice: I’d recommend whoever picks this up to pick one of these options and run with it. manylinux happened because we were ruthless about getting something workable into users’ hands, and made whatever compromises we had to to accomplish that. This is the sort of problem where you can spend forever debating and going off on tangents, and it doesn’t help. If you can get something that allows people using the python:alpine docker image to install wheels, then that’s worth shipping, even if it doesn’t solve every other problem.

7 Likes

All versions, from 0.5.0 to 1.2.2 have followed x.y.z as a version scheme. A quick talk on IRC seems to imply that to be scheme for all planned releases as well.

From what I understood, Rich has suggested kind of the same thing; and it makes sense, after all, compiled modules are unlikely to use newly introduced functions, which means that if auditwheel keeps a database of symbols, figuring out the minimum musl version would be doable. I am not 100% comfortable with this approach, however. Still, such a workaround would be a good idea because otherwise “you’d mostly end up imposing gratuitous later version dependency than needed”.

An alternative approach could be allowing the user to manually override the manymusl version in the wheel to allow installation in their system, but I guess this is unlikely to help in container usage.

Yes, statically linked binaries with musl just get an error when they attempt to call dlopen. That said, I’d rather see Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker fixed than having to manually parse any ELF at all - the downside being that this would take a while to propagate across python versions, from what I understand.

While unfortunate for users of other musl based distributions, this would work for the most pressing case here, which is python stuff on alpine containers.

All that said, I think sticking with 64-bit platforms for now is the best thing to do. Since Rust hard codes information taken from platform headers instead of being able to read them, they can’t immediately adapt to the time64 changes. See musl: Change `time_t` definition on 32-bit targets according to "time64" · Issue #1848 · rust-lang/libc · GitHub

Would it be worth having a stop-gap alpine platform tag which is not supposed to be future-proof? Are the future maintenance requirements too great?

As long as auditwheel can automatically switch from generating alpine wheels to manymusl wheels, there would be no friction anywhere in the packaging workflow, right?

Maybe for an initial release we should try and stick with 1.1.x (not sure which one to pick, would appreciate input from @ncopa), then? Having manymusl_major_minor_arch like it’s done for glibc doesn’t sound too terrible.

I’d rather start with musl 1.2.x and add 1.1 later if needed.

As a general piece of advice: I’d recommend whoever picks this up to pick one of these options and run with it. manylinux happened because we were ruthless about getting something workable into users’ hands, and made whatever compromises we had to to accomplish that. This is the sort of problem where you can spend forever debating and going off on tangents, and it doesn’t help. If you can get something that allows people using the python:alpine docker image to install wheels, then that’s worth shipping, even if it doesn’t solve every other problem.

Then what I would prefer to see is that we start with musllinux (not alpine), and don’t bother with the version number for now. Assume musl 1.2.x with time64 and only support that. But even before that I’d like to be able at compile time tell that this is musl.

Yes, statically linked binaries with musl just get an error when they attempt to call dlopen. That said, I’d rather see Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker fixed than having to manually parse any ELF at all - the downside being that this would take a while to propagate across python versions, from what I understand.

I agree that Issue 43112 should be fixed first.