Wheels for musl (Alpine)

Would it be worth having a stop-gap alpine platform tag which is not supposed to be future-proof? Are the future maintenance requirements too great?

As long as auditwheel can automatically switch from generating alpine wheels to manymusl wheels, there would be no friction anywhere in the packaging workflow, right?

Maybe for an initial release we should try and stick with 1.1.x (not sure which one to pick, would appreciate input from @ncopa), then? Having manymusl_major_minor_arch like it’s done for glibc doesn’t sound too terrible.

I’d rather start with musl 1.2.x and add 1.1 later if needed.

As a general piece of advice: I’d recommend whoever picks this up to pick one of these options and run with it. manylinux happened because we were ruthless about getting something workable into users’ hands, and made whatever compromises we had to to accomplish that. This is the sort of problem where you can spend forever debating and going off on tangents, and it doesn’t help. If you can get something that allows people using the python:alpine docker image to install wheels, then that’s worth shipping, even if it doesn’t solve every other problem.

Then what I would prefer to see is that we start with musllinux (not alpine), and don’t bother with the version number for now. Assume musl 1.2.x with time64 and only support that. But even before that I’d like to be able at compile time tell that this is musl.

Yes, statically linked binaries with musl just get an error when they attempt to call dlopen. That said, I’d rather see Issue 43112: SOABI on Linux does not distinguish between GNU libc and musl libc - Python tracker fixed than having to manually parse any ELF at all - the downside being that this would take a while to propagate across python versions, from what I understand.

I agree that Issue 43112 should be fixed first.

Based on our manylinux experience it will simply be too costly time-wise. We ended up with perennial manylinux specifically because updating the manylinux versions took so long every time.

Now that I’m on a computer with docker, I took a quick peek at the popular alpine image, and it appears that it ships with literally nothing except musl, openssl, libz, and libtls-standalone (which appears to be some funky thing forked from part of libressl):

❯ docker run --rm -it alpine ls /lib /usr/lib
/lib:
apk                    libc.musl-x86_64.so.1  libz.so.1.2.11
firmware               libcrypto.so.1.1       mdev
ld-musl-x86_64.so.1    libssl.so.1.1          modules-load.d
libapk.so.3.12.0       libz.so.1              sysctl.d

/usr/lib:
engines-1.1                 libtls-standalone.so.1
libcrypto.so.1.1            libtls-standalone.so.1.0.0
libssl.so.1.1               modules-load.d

OpenSSL has traditionally been really bad at ABI compatibility, and I doubt many packages are using this libtls-standalone thing since it appears to be unique to alpine. And musl and libz both have extremely stable ABIs and it’s safe to assume they’re available on every musl-based distribution.

So my updated suggestion is:

  • Write a quick PEP for musllinux_X_Y tags, where the tag means that the wheel should work on any real-world distro that uses musl X.Y or later. You can cite the perennial manylinux PEP for most of the details. This PEP could honestly be like 3 paragraphs long. Maybe less.
  • Update pip to understand when it’s running under musl X.Y or later. This will require grossness but it’s doable. Maybe later something like bpo-43112 will make it less gross, but worry about that some other time.
  • Update auditwheel to understand musllinux_X_Y tags
  • Maybe provide some standard musllinux docker images, though tbh these would probably just be the standard python:alpine images + gcc + auditwheel, so the need is a lot less urgent than for manylinux
5 Likes

This sounds exceptionally sensible to me! For pyca/cryptography we’d be able to start producing wheels as soon as auditwheel added support.

2 Likes

I am still a bit worried about time64 mixing with Rust, which is quite relevant, because, as I understand it, the main motivation for this post was using Rust in Python wheels without forcing containers to also include the Rust toolchain. On 32 bit devices, any ABI boundary between Rust and C code which uses types which depend on time_t (so struct timespec, struct stat, …) should by all rights be completely broken - note that using these types from within Rust will still be completely fine, since the libc ABI is completely functional.

For that reason, if we are starting with musl 1.2, it shouldn’t cover 32-bit devices, in my opinion. If someone from the Rust side (maybe someone has a contact they could ping?) could leave their thoughts about this here, that would be of great help. If the risk is considered low enough, we could go forward with 32-bit devices as well, I guess.

Possibly a rust toolchain as well? Luckily setuptools_rust already works around rustc's behavior of defaulting to static linkage when using musl, so building dynamic wheels should work out of the box.

Isn’t it the other way around? A musllinux_1_2_i686 tag promises to only run on 32-bit with musl 1.2+, which is correct from what I can tell—The runtime it’s incompatible with is musl 1.1. The problematic tag would be musllinux_1_1_i686, which (by the definition of PEP 600) should be compatible with all musl 1.1+ (32-bit), which may not be correct.

One way to work around the problem would be to add an additional contraint. The rules laid out by PEP 600 all still hold, except that if a project published both musllinux_1_1 and musllinux_1_2 wheels on the same index, it does not need to promise the 1_1 one will work on musl 1.2+. Practically installers should always prefer the 1_2 one if both are present (pip already does).

I’ve made a draft here: Pre-PEP: Platform Tag for Linux Distributions Using Musl

1 Like

I think the time_t thing might be a red herring? It only affects parts of the Python C API that use time_t, which appears to be nothing except a few underscore-prefixed functions. I’m not sure any C extensions use them at all, and auditwheel should be able to detect the rare problematic cases in any case.

I don’t mind only support 64 bit initially, even if python itself can support 32 bit devices with musl 1.2+, but yeah, anything using rust won’t work til rust is fixed. Using rust is a cost/benefit decision and I think they are fine with not having 32 bit as they only focus on the bigger architectures.

The problem, as I understand, is that Rust is broken with musl 1.2+ and probably hard to fix. It is not a problem for python to worry about IMHO.

No. musllinux_1_1_i686 will be compatible with all future musl tags. The issue with 1.2 I’m referring to is showcased in GitHub - ericonr/rust-time64, where the resulting binary can simply be entirely miscompiled.

I’m not sure how you would detect this at all? The mismatch is in the type definition, not in linkage. And the issue can appear for any module that uses Rust and C code, for example from an external library that’s linked statically into the bundle, not only when talking directly to the Python C API.

The project I linked above shows the issue I’m worried about, but I have no idea how to actually fix it.

It’s kinda towards the bottom of GitHub - ericonr/rust-time64 (though you also mentioned it in musl: Change `time_t` definition on 32-bit targets according to "time64" · Issue #1848 · rust-lang/libc · GitHub), but IIUC, that issue should only affect 32bit targets, right?

I had the impression that people in this thread were broadly in agreement that 32bit can be explicitly descoped for the first round of getting something working, and could still be added later (once all blockers have been resolved, and people are interested enough to do the work).

Yes, definitely. I just want to make the issue with Rust + musl 1.2 on 32-bit systems very clear, if anyone is planning on using Rust there. Some of the comments seemed dismissive of the issue.

Hmm OK, so I completely mischaracterised the issue. So, if I understand correctly, the real issue is not really about interoperability of libraries using musl in general, but specifically if the software (for example, Rust) is implemented incorrectly (in some sense)? Maybe I should remove that part from the PEP draft then.

I don’t think people are generally dismissive of the issue, but are trying to focus on the most immediate topic by pointing out the issue does not really need to affect the wheel tag. Many comments here seem to be aimed more specifically to solve the “cryptography on Alpine containers” situation, which needs a lot of things to happen. A wheel tag is merely the first step to the solution, and has nothing to do with Rust. The time64 issue is still not insignificant, but it’s actually counter-productive to include it in the wheel tag discussion, because it raises unnecessary questions we don’t need to get into (why do we choose to only target a very new musl version? why do we exclude 32-bit?) It is actually easier to specify a generic tag that contains unworkable combinations, because, well, nobody is going to use it anyway.

3 Likes

Exactly :slight_smile:

Specifically, the whole thing works fine for languages that use C headers, since the situation is completely solved there. The ugliness only appears when you combine things using C headers and things that hardcoded their own understanding of the ABI (which was correct in isolation).

That’s fair. I’m very much a new comer to this forum, so thank you for pointing this out plainly. In that case, I will try to document these concerns when we have a place to put documentation for the musllinux tag.

No problem at all! Your input is valuable to the conversation and definite helped me coming up with the PEP draft. I’m looking forward to hearing about your opinions more in the steps after the musllinux tag, when we actually try to make the build toolchain available for projects :slightly_smiling_face:

2 Likes

Is anybody working on a PEP and implementation? It’s been over a month since last reply.

I’m still looking for a sponsor for my draft