There’s been a lot of discussion over on the “next manylinux specification” thread, and while most of it has been specifically about the two proposals under consideration, there seem to be a couple of underlying issues. These are not directly related to which proposal is more appropriate, but are more about the wider questions around managing manylinux in general, so I thought they should be split out into a separate topic. There’s nothing that needs a “pronouncement” here, so my interest is mainly in clearing the water a bit - I feel that the “next manylinux” discussions would be easier at times if we all understood the background problems/constraints a bit better.
I’m going to be quoting from the topic “The next manylinux specification” here - there’s extra context in that thread, but I think that this needs to be a new thread to keep the discussions focused.
Where we stand on manylinux2010
If I understand the manylinux2010 tracking issue, we haven’t yet fully released manylinux2010 for use. It’s nearly there, but there are some final issues to be addressed. Also, the Fedora 30 change to libcrypt probably requires some additional work.
I don’t know much about manylinux development, but I get the impression that (as with many other Python packaging projects!) resource is extremely tight. I don’t know to what extent that’s the reason that shipping manylinux2010 is where it is, but I think that before we dive into manylinux2014 it’s important (not just for the sake of the Python community, but also, and possibly even more so, for the sake of the manylinux maintainers) to ensure that we have addressed the underlying issues that caused manylinux2010 development to be so painful.
From the initial post in “the next manylinux specification”:
One obvious question that comes to mind for me is whether the individuals interested in helping with manylinux2014, are currently working on the backlog of work for manylinux2010 (or on any other areas of auditwheel/manylinux development). Getting involved in a project takes some startup time, as well as assistance from existing project maintainers, so it seems like helping out now, would be a good way to get up to speed in readiness for manylinux2014. Otherwise, I’d be a bit concerned that we’d get a “blip” of effort getting the next manylinux up and running (and the manylinux maintainers would be under pressure reviewing and accepting PRs, etc), but it would then disappear, and we’d be back to the same situation we’re now in.
The perennial manylinux proposal has the benefit of trying to address the root cause here, which (if I’m reading the situation correctly) is “too much work to do”. But I’m concerned that it’s only chipping at the edges (largely because the bulk of the work simply can’t be avoided). And if resource genuinely is the issue, then short of getting more people participating, the only real solution is to avoid promising to deliver more than we’re capable of doing - even if that means we have to acknowledge that some problems will remain unsolved until we have the resource to address them.
The C++ ABI issue
This issue seems to me to be fundamentally about the fact that code built with C++ has strict libstdc++ compatibility requirements. Both manylinux2010 and manylinux1 require linking with libstdc++.so.6, so in theory that’s covered. But C++ has some pretty “interesting” rules here (@pitrou mentioned static local initialisation, which triggers compiler-generated code that runs before main()) and they need to be checked.
That pretty much says to me that there’s a toolchain-related issue involved here, and we need to ensure that auditwheel checks for that and disallows incompatible toolchains. In a perennial manylinux world, that’s “just” an auditwheel change, with whatever docs we agree are needed being updated to match the implementation. But in a manylinux2014 world, this is something that needs to be covered in the PEP if we’re to have a spec that ensures we don’t get such crashes in the future.
(This is possibly something that should be retrofitted to manylinux2010 and maybe even manylinux1 as well).
Summary
It seems to me that there’s a reasonable amount of work that can be done right now, to finish off the manylinux2010 release, and to address the C++ compatibility issues. These aren’t dependent in any way that I can see on the approval of a new manylinux spec - and in fact, approving a new spec at this point would risk diverting effort away from the existing specs. (Indeed, the initial post in “The next manylinux specification” suggests adding more architectures to the supported list, which I would assume involves a non-trivial amount of extra work, over and above the current load).
I’m not sure what the solution is here - I’m neither part of the auditwheel project, nor am I one of the people offering to help with implementing manylinux2014 - but maybe if we have the discussion we can come up with some sort of workable approach?