Wheels for musl (Alpine)

I don’t mind only support 64 bit initially, even if python itself can support 32 bit devices with musl 1.2+, but yeah, anything using rust won’t work til rust is fixed. Using rust is a cost/benefit decision and I think they are fine with not having 32 bit as they only focus on the bigger architectures.

The problem, as I understand, is that Rust is broken with musl 1.2+ and probably hard to fix. It is not a problem for python to worry about IMHO.

No. musllinux_1_1_i686 will be compatible with all future musl tags. The issue with 1.2 I’m referring to is showcased in GitHub - ericonr/rust-time64, where the resulting binary can simply be entirely miscompiled.

I’m not sure how you would detect this at all? The mismatch is in the type definition, not in linkage. And the issue can appear for any module that uses Rust and C code, for example from an external library that’s linked statically into the bundle, not only when talking directly to the Python C API.

The project I linked above shows the issue I’m worried about, but I have no idea how to actually fix it.

It’s kinda towards the bottom of GitHub - ericonr/rust-time64 (though you also mentioned it in musl: Change `time_t` definition on 32-bit targets according to "time64" · Issue #1848 · rust-lang/libc · GitHub), but IIUC, that issue should only affect 32bit targets, right?

I had the impression that people in this thread were broadly in agreement that 32bit can be explicitly descoped for the first round of getting something working, and could still be added later (once all blockers have been resolved, and people are interested enough to do the work).

Yes, definitely. I just want to make the issue with Rust + musl 1.2 on 32-bit systems very clear, if anyone is planning on using Rust there. Some of the comments seemed dismissive of the issue.

Hmm OK, so I completely mischaracterised the issue. So, if I understand correctly, the real issue is not really about interoperability of libraries using musl in general, but specifically if the software (for example, Rust) is implemented incorrectly (in some sense)? Maybe I should remove that part from the PEP draft then.

I don’t think people are generally dismissive of the issue, but are trying to focus on the most immediate topic by pointing out the issue does not really need to affect the wheel tag. Many comments here seem to be aimed more specifically to solve the “cryptography on Alpine containers” situation, which needs a lot of things to happen. A wheel tag is merely the first step to the solution, and has nothing to do with Rust. The time64 issue is still not insignificant, but it’s actually counter-productive to include it in the wheel tag discussion, because it raises unnecessary questions we don’t need to get into (why do we choose to only target a very new musl version? why do we exclude 32-bit?) It is actually easier to specify a generic tag that contains unworkable combinations, because, well, nobody is going to use it anyway.

3 Likes

Exactly :slight_smile:

Specifically, the whole thing works fine for languages that use C headers, since the situation is completely solved there. The ugliness only appears when you combine things using C headers and things that hardcoded their own understanding of the ABI (which was correct in isolation).

That’s fair. I’m very much a new comer to this forum, so thank you for pointing this out plainly. In that case, I will try to document these concerns when we have a place to put documentation for the musllinux tag.

No problem at all! Your input is valuable to the conversation and definite helped me coming up with the PEP draft. I’m looking forward to hearing about your opinions more in the steps after the musllinux tag, when we actually try to make the build toolchain available for projects :slightly_smiling_face:

2 Likes

Is anybody working on a PEP and implementation? It’s been over a month since last reply.

I’m still looking for a sponsor for my draft

I missed the fact that you already wrote a PEP. Excellent!

The PEP looks good to me. I just miss two points

Nathaniel had a pretty extensive response on that topic to a comment I made.

TL;DR: it’s not exactly like glibc, but yes, for all (non-buggy) intents and purposes. that promise exists.

Nathaniel’s comment is great (as usual). It clearly shows that he works in academics. :slight_smile:

I like to see this information included in the document. A PEP should mention relevant and useful pieces of information. ABI stability is an important assumption and relevant for implementers of pip and warehouse (PyPI).

I’ll try to find some time and write these into the draft. Although I’d really like a better reference than a short email on the backwards compatibility issue.

1 Like

I can sponsor.

2 Likes

musl libc - Compatibility doesn’t mention a musl policy directly (and I didn’t find much searching the docs/wiki), but it does link to this comparison, which merely says “Backwards Compatibility: yes”.

I’ve updated the document and submitted it to New PEP: Platform Tag for Linux Distributions Using Musl by uranusjr · Pull Request #1876 · python/peps · GitHub

PEP 656 -- Platform Tag for Linux Distributions Using Musl | Python.org is live!

2 Likes

In the PEP there is some code to pull out the musl version. It doesn’t work for me. Is there a better way?

os_name, sep, arch = sysconfig.get_platform().partition("-")
assert os_name == "linux" and sep, "Not a Linux"
arch.replace("-", "_")
return f"musllinux_{musl_version[0]}_{musl_version[1]}_{arch}"

When I run that code in an Alpine container, I do not get a valid arch value:

docker run -it --rm python:3.7.2-alpine3.9 /bin/sh
...
/ # python
Python 3.7.2 (default, Mar  7 2019, 22:40:59) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'

Not that glibc is any better, here is the dance needed to find it (either via os.confstr('"CS_GNU_LIBC_VERSION") or using ctypes to call gnu_get_libc_version())

Matti

That code’s actually expected to be supplied with the MUSL version. I don’t see any code in the PEP that explains how, for a given installation, you determine what the MUSL version that you’re running on is. I think the PEP needs to include that.

It looks like PEP 600 (manylinux) doesn’t cover the equivalent question for glibc directly - it just refers to an undefined get_system_glibc_version function, but the explanation of how to write that is in PEP 513

As there are no previous MUSL peps, we probably need to make sure that we pick up any other dependencies like this that might have come from basing PEP 656 on PEP 600…

I just opened PEP 656 musllinux support by uranusjr · Pull Request #411 · pypa/packaging · GitHub which includes an approach to fetch musl presence and the version in use by invoking ldd. Maybe I can update the PEP to include this as a reference implementation?