Based on our manylinux experience it will simply be too costly time-wise. We ended up with perennial manylinux specifically because updating the manylinux versions took so long every time.
Now that I’m on a computer with docker, I took a quick peek at the popular
alpine image, and it appears that it ships with literally nothing except musl, openssl, libz, and libtls-standalone (which appears to be some funky thing forked from part of libressl):
❯ docker run --rm -it alpine ls /lib /usr/lib /lib: apk libc.musl-x86_64.so.1 libz.so.1.2.11 firmware libcrypto.so.1.1 mdev ld-musl-x86_64.so.1 libssl.so.1.1 modules-load.d libapk.so.3.12.0 libz.so.1 sysctl.d /usr/lib: engines-1.1 libtls-standalone.so.1 libcrypto.so.1.1 libtls-standalone.so.1.0.0 libssl.so.1.1 modules-load.d
OpenSSL has traditionally been really bad at ABI compatibility, and I doubt many packages are using this libtls-standalone thing since it appears to be unique to alpine. And musl and libz both have extremely stable ABIs and it’s safe to assume they’re available on every musl-based distribution.
So my updated suggestion is:
- Write a quick PEP for
musllinux_X_Ytags, where the tag means that the wheel should work on any real-world distro that uses musl X.Y or later. You can cite the perennial manylinux PEP for most of the details. This PEP could honestly be like 3 paragraphs long. Maybe less.
- Update pip to understand when it’s running under musl X.Y or later. This will require grossness but it’s doable. Maybe later something like bpo-43112 will make it less gross, but worry about that some other time.
- Update auditwheel to understand
- Maybe provide some standard musllinux docker images, though tbh these would probably just be the standard
python:alpineimages + gcc + auditwheel, so the need is a lot less urgent than for manylinux
This sounds exceptionally sensible to me! For pyca/cryptography we’d be able to start producing wheels as soon as
auditwheel added support.
I am still a bit worried about time64 mixing with Rust, which is quite relevant, because, as I understand it, the main motivation for this post was using Rust in Python wheels without forcing containers to also include the Rust toolchain. On 32 bit devices, any ABI boundary between Rust and C code which uses types which depend on
struct stat, …) should by all rights be completely broken - note that using these types from within Rust will still be completely fine, since the libc ABI is completely functional.
For that reason, if we are starting with musl 1.2, it shouldn’t cover 32-bit devices, in my opinion. If someone from the Rust side (maybe someone has a contact they could ping?) could leave their thoughts about this here, that would be of great help. If the risk is considered low enough, we could go forward with 32-bit devices as well, I guess.
Possibly a rust toolchain as well? Luckily
setuptools_rust already works around
rustc's behavior of defaulting to static linkage when using musl, so building dynamic wheels should work out of the box.
Isn’t it the other way around? A
musllinux_1_2_i686 tag promises to only run on 32-bit with musl 1.2+, which is correct from what I can tell—The runtime it’s incompatible with is musl 1.1. The problematic tag would be
musllinux_1_1_i686, which (by the definition of PEP 600) should be compatible with all musl 1.1+ (32-bit), which may not be correct.
One way to work around the problem would be to add an additional contraint. The rules laid out by PEP 600 all still hold, except that if a project published both
musllinux_1_2 wheels on the same index, it does not need to promise the
1_1 one will work on musl 1.2+. Practically installers should always prefer the
1_2 one if both are present (pip already does).
I’ve made a draft here: Pre-PEP: Platform Tag for Linux Distributions Using Musl
I think the
time_t thing might be a red herring? It only affects parts of the Python C API that use
time_t, which appears to be nothing except a few underscore-prefixed functions. I’m not sure any C extensions use them at all, and auditwheel should be able to detect the rare problematic cases in any case.
I don’t mind only support 64 bit initially, even if python itself can support 32 bit devices with musl 1.2+, but yeah, anything using rust won’t work til rust is fixed. Using rust is a cost/benefit decision and I think they are fine with not having 32 bit as they only focus on the bigger architectures.
The problem, as I understand, is that Rust is broken with musl 1.2+ and probably hard to fix. It is not a problem for python to worry about IMHO.
musllinux_1_1_i686 will be compatible with all future musl tags. The issue with 1.2 I’m referring to is showcased in GitHub - ericonr/rust-time64, where the resulting binary can simply be entirely miscompiled.
I’m not sure how you would detect this at all? The mismatch is in the type definition, not in linkage. And the issue can appear for any module that uses Rust and C code, for example from an external library that’s linked statically into the bundle, not only when talking directly to the Python C API.
The project I linked above shows the issue I’m worried about, but I have no idea how to actually fix it.
It’s kinda towards the bottom of GitHub - ericonr/rust-time64 (though you also mentioned it in musl: Change `time_t` definition on 32-bit targets according to "time64" · Issue #1848 · rust-lang/libc · GitHub), but IIUC, that issue should only affect 32bit targets, right?
I had the impression that people in this thread were broadly in agreement that 32bit can be explicitly descoped for the first round of getting something working, and could still be added later (once all blockers have been resolved, and people are interested enough to do the work).
Yes, definitely. I just want to make the issue with Rust + musl 1.2 on 32-bit systems very clear, if anyone is planning on using Rust there. Some of the comments seemed dismissive of the issue.
Hmm OK, so I completely mischaracterised the issue. So, if I understand correctly, the real issue is not really about interoperability of libraries using musl in general, but specifically if the software (for example, Rust) is implemented incorrectly (in some sense)? Maybe I should remove that part from the PEP draft then.
I don’t think people are generally dismissive of the issue, but are trying to focus on the most immediate topic by pointing out the issue does not really need to affect the wheel tag. Many comments here seem to be aimed more specifically to solve the “cryptography on Alpine containers” situation, which needs a lot of things to happen. A wheel tag is merely the first step to the solution, and has nothing to do with Rust. The time64 issue is still not insignificant, but it’s actually counter-productive to include it in the wheel tag discussion, because it raises unnecessary questions we don’t need to get into (why do we choose to only target a very new musl version? why do we exclude 32-bit?) It is actually easier to specify a generic tag that contains unworkable combinations, because, well, nobody is going to use it anyway.
Specifically, the whole thing works fine for languages that use C headers, since the situation is completely solved there. The ugliness only appears when you combine things using C headers and things that hardcoded their own understanding of the ABI (which was correct in isolation).
That’s fair. I’m very much a new comer to this forum, so thank you for pointing this out plainly. In that case, I will try to document these concerns when we have a place to put documentation for the
No problem at all! Your input is valuable to the conversation and definite helped me coming up with the PEP draft. I’m looking forward to hearing about your opinions more in the steps after the
musllinux tag, when we actually try to make the build toolchain available for projects
Is anybody working on a PEP and implementation? It’s been over a month since last reply.
I’m still looking for a sponsor for my draft
I missed the fact that you already wrote a PEP. Excellent!
The PEP looks good to me. I just miss two points
- a statement similar to PEP 600 -- Future 'manylinux' Platform Tags for Portable Linux Built Distributions | Python.org about PyPI support for manymusl binaries.
- a link or statement regarding musl ABI stability. Does MUSL promise that binaries compiled with musl ABI
TL;DR: it’s not exactly like glibc, but yes, for all (non-buggy) intents and purposes. that promise exists.
Nathaniel’s comment is great (as usual). It clearly shows that he works in academics.
I like to see this information included in the document. A PEP should mention relevant and useful pieces of information. ABI stability is an important assumption and relevant for implementers of pip and warehouse (PyPI).
I’ll try to find some time and write these into the draft. Although I’d really like a better reference than a short email on the backwards compatibility issue.