I don’t mind only support 64 bit initially, even if python itself can support 32 bit devices with musl 1.2+, but yeah, anything using rust won’t work til rust is fixed. Using rust is a cost/benefit decision and I think they are fine with not having 32 bit as they only focus on the bigger architectures.
The problem, as I understand, is that Rust is broken with musl 1.2+ and probably hard to fix. It is not a problem for python to worry about IMHO.
No. musllinux_1_1_i686 will be compatible with all future musl tags. The issue with 1.2 I’m referring to is showcased in GitHub - ericonr/rust-time64, where the resulting binary can simply be entirely miscompiled.
I’m not sure how you would detect this at all? The mismatch is in the type definition, not in linkage. And the issue can appear for any module that uses Rust and C code, for example from an external library that’s linked statically into the bundle, not only when talking directly to the Python C API.
The project I linked above shows the issue I’m worried about, but I have no idea how to actually fix it.
I had the impression that people in this thread were broadly in agreement that 32bit can be explicitly descoped for the first round of getting something working, and could still be added later (once all blockers have been resolved, and people are interested enough to do the work).
Yes, definitely. I just want to make the issue with Rust + musl 1.2 on 32-bit systems very clear, if anyone is planning on using Rust there. Some of the comments seemed dismissive of the issue.
Hmm OK, so I completely mischaracterised the issue. So, if I understand correctly, the real issue is not really about interoperability of libraries using musl in general, but specifically if the software (for example, Rust) is implemented incorrectly (in some sense)? Maybe I should remove that part from the PEP draft then.
I don’t think people are generally dismissive of the issue, but are trying to focus on the most immediate topic by pointing out the issue does not really need to affect the wheel tag. Many comments here seem to be aimed more specifically to solve the “cryptography on Alpine containers” situation, which needs a lot of things to happen. A wheel tag is merely the first step to the solution, and has nothing to do with Rust. The time64 issue is still not insignificant, but it’s actually counter-productive to include it in the wheel tag discussion, because it raises unnecessary questions we don’t need to get into (why do we choose to only target a very new musl version? why do we exclude 32-bit?) It is actually easier to specify a generic tag that contains unworkable combinations, because, well, nobody is going to use it anyway.
Specifically, the whole thing works fine for languages that use C headers, since the situation is completely solved there. The ugliness only appears when you combine things using C headers and things that hardcoded their own understanding of the ABI (which was correct in isolation).
That’s fair. I’m very much a new comer to this forum, so thank you for pointing this out plainly. In that case, I will try to document these concerns when we have a place to put documentation for the musllinux tag.
No problem at all! Your input is valuable to the conversation and definite helped me coming up with the PEP draft. I’m looking forward to hearing about your opinions more in the steps after the musllinux tag, when we actually try to make the build toolchain available for projects
Nathaniel’s comment is great (as usual). It clearly shows that he works in academics.
I like to see this information included in the document. A PEP should mention relevant and useful pieces of information. ABI stability is an important assumption and relevant for implementers of pip and warehouse (PyPI).
I’ll try to find some time and write these into the draft. Although I’d really like a better reference than a short email on the backwards compatibility issue.
musl libc - Compatibility doesn’t mention a musl policy directly (and I didn’t find much searching the docs/wiki), but it does link to this comparison, which merely says “Backwards Compatibility: yes”.
In the PEP there is some code to pull out the musl version. It doesn’t work for me. Is there a better way?
os_name, sep, arch = sysconfig.get_platform().partition("-")
assert os_name == "linux" and sep, "Not a Linux"
arch.replace("-", "_")
return f"musllinux_{musl_version[0]}_{musl_version[1]}_{arch}"
When I run that code in an Alpine container, I do not get a valid arch value:
docker run -it --rm python:3.7.2-alpine3.9 /bin/sh
...
/ # python
Python 3.7.2 (default, Mar 7 2019, 22:40:59)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'
Not that glibc is any better, here is the dance needed to find it (either via os.confstr('"CS_GNU_LIBC_VERSION") or using ctypes to call gnu_get_libc_version())
That code’s actually expected to be supplied with the MUSL version. I don’t see any code in the PEP that explains how, for a given installation, you determine what the MUSL version that you’re running on is. I think the PEP needs to include that.
It looks like PEP 600 (manylinux) doesn’t cover the equivalent question for glibc directly - it just refers to an undefined get_system_glibc_version function, but the explanation of how to write that is in PEP 513
As there are no previous MUSL peps, we probably need to make sure that we pick up any other dependencies like this that might have come from basing PEP 656 on PEP 600…