Wheels for musl (Alpine)

My reading of Rich’s post is that generally, you can be pretty confident that it works to build against musl version X and run on version X+1, with the only exception being if the library was somehow depending on some bug in version X that got fixed in X+1. I see why Rich emphasizes that exception, because Rich is a very precise guy and musl does put less effort into bug-for-bug compatibility than glibc, but really this is the same caveat that applies to every upgrade process ever, and I don’t think we need to worry about it being a major issue for musl.

The other thing he’s pointing out is that in many cases, you can build against musl X+1 and actually get a binary that runs on version X (!). Unfortunately, this isn’t super useful to us at the spec level: “many cases” is not the same as “all cases”, so we can’t just assume that building on any version of musl will produce a binary that works on all versions of musl. We’ll still need some kind of monotonic “version” counter to use in the wheel tag. (Though auditwheel could potentially be smart enough to detect cases where a binary built against musl X+1 will work on musl X, and tag the wheel appropriately.)

AFAIK, the biggest challenge to doing this is that we don’t yet have a concrete proposal for how to reliably figure out what version of musl is running. The best option I’ve seen is to first manually parse the python binary’s ELF header to see if the string musl appears and what the path to libc.so is, and then invoking libc.so as a child process and parsing the text it prints in the output. This is like… incredibly janky, but probably doable? Also, it will fail if the python binary itself was statically linked – but maybe that’s OK? If python was statically linked against musl, then I think it might not be able to load extension modules that are dynamically linked against musl anyway?

The other option would be to give up on using musl itself as the clock, and instead use the distro. It’s very easy to figure out whether you’re running on alpine and what version you have: just check the standardized os-release file. The downside is that then wheels would be specific to alpine, and wouldn’t automatically install on other musl-based linuxes. The upside would be that wheels could potentially depend on features that alpine provides beyond just musl… though I’m not sure whether there are any, since alpine is so minimalist.

As a general piece of advice: I’d recommend whoever picks this up to pick one of these options and run with it. manylinux happened because we were ruthless about getting something workable into users’ hands, and made whatever compromises we had to to accomplish that. This is the sort of problem where you can spend forever debating and going off on tangents, and it doesn’t help. If you can get something that allows people using the python:alpine docker image to install wheels, then that’s worth shipping, even if it doesn’t solve every other problem.

7 Likes