Where we are on manylinux2010 (and how that relates to manylinux2014)

There’s been a lot of discussion over on the “next manylinux specification” thread, and while most of it has been specifically about the two proposals under consideration, there seem to be a couple of underlying issues. These are not directly related to which proposal is more appropriate, but are more about the wider questions around managing manylinux in general, so I thought they should be split out into a separate topic. There’s nothing that needs a “pronouncement” here, so my interest is mainly in clearing the water a bit - I feel that the “next manylinux” discussions would be easier at times if we all understood the background problems/constraints a bit better.

I’m going to be quoting from the topic “The next manylinux specification” here - there’s extra context in that thread, but I think that this needs to be a new thread to keep the discussions focused.

Where we stand on manylinux2010

If I understand the manylinux2010 tracking issue, we haven’t yet fully released manylinux2010 for use. It’s nearly there, but there are some final issues to be addressed. Also, the Fedora 30 change to libcrypt probably requires some additional work.

I don’t know much about manylinux development, but I get the impression that (as with many other Python packaging projects!) resource is extremely tight. I don’t know to what extent that’s the reason that shipping manylinux2010 is where it is, but I think that before we dive into manylinux2014 it’s important (not just for the sake of the Python community, but also, and possibly even more so, for the sake of the manylinux maintainers) to ensure that we have addressed the underlying issues that caused manylinux2010 development to be so painful.

From the initial post in “the next manylinux specification”:

One obvious question that comes to mind for me is whether the individuals interested in helping with manylinux2014, are currently working on the backlog of work for manylinux2010 (or on any other areas of auditwheel/manylinux development). Getting involved in a project takes some startup time, as well as assistance from existing project maintainers, so it seems like helping out now, would be a good way to get up to speed in readiness for manylinux2014. Otherwise, I’d be a bit concerned that we’d get a “blip” of effort getting the next manylinux up and running (and the manylinux maintainers would be under pressure reviewing and accepting PRs, etc), but it would then disappear, and we’d be back to the same situation we’re now in.

The perennial manylinux proposal has the benefit of trying to address the root cause here, which (if I’m reading the situation correctly) is “too much work to do”. But I’m concerned that it’s only chipping at the edges (largely because the bulk of the work simply can’t be avoided). And if resource genuinely is the issue, then short of getting more people participating, the only real solution is to avoid promising to deliver more than we’re capable of doing - even if that means we have to acknowledge that some problems will remain unsolved until we have the resource to address them.

The C++ ABI issue

This issue seems to me to be fundamentally about the fact that code built with C++ has strict libstdc++ compatibility requirements. Both manylinux2010 and manylinux1 require linking with libstdc++.so.6, so in theory that’s covered. But C++ has some pretty “interesting” rules here (@pitrou mentioned static local initialisation, which triggers compiler-generated code that runs before main()) and they need to be checked.

That pretty much says to me that there’s a toolchain-related issue involved here, and we need to ensure that auditwheel checks for that and disallows incompatible toolchains. In a perennial manylinux world, that’s “just” an auditwheel change, with whatever docs we agree are needed being updated to match the implementation. But in a manylinux2014 world, this is something that needs to be covered in the PEP if we’re to have a spec that ensures we don’t get such crashes in the future.

(This is possibly something that should be retrofitted to manylinux2010 and maybe even manylinux1 as well).

Summary

It seems to me that there’s a reasonable amount of work that can be done right now, to finish off the manylinux2010 release, and to address the C++ compatibility issues. These aren’t dependent in any way that I can see on the approval of a new manylinux spec - and in fact, approving a new spec at this point would risk diverting effort away from the existing specs. (Indeed, the initial post in “The next manylinux specification” suggests adding more architectures to the supported list, which I would assume involves a non-trivial amount of extra work, over and above the current load).

I’m not sure what the solution is here - I’m neither part of the auditwheel project, nor am I one of the people offering to help with implementing manylinux2014 - but maybe if we have the discussion we can come up with some sort of workable approach?

AFAIK manylinux2010 is basically working now. We managed to produce manylinux2010 wheels for PyArrow (and it’s not exactly a simple package when it comes to native dependencies, see Dockerfile).

One issue is hit is a possible bug where auditwheel repair doesn’t correctly rewrite internal DLL dependencies, but it seems it’s not manylinux2010-specific.

I guess my main point there was simply that the people looking to contribute could probably do so even before the next manylinux spec gets signed off, even if it’s “only” on administrative issues like reviewing the tracker issue for 2010 that’s still open and tidying up whatever’s needed to close it.

From the perspective of someone who’s done a little bit of packaging for PyPI (only a couple of packages, but one of them has compiled code with complicated library and data dependencies), I think the remaining incomplete items from the tracking issue will be really important for takeup. Here they are, for reference:

Nominate a timeline and/or “percentage of PyPI downloads by manylinux2010 compatible installers” for switching the default build environment and auditwheel output to manylinux2010
Reword the “not yet widely supported” caveat from the platform compatibility tagging page to instead specify the minimum required versions of auditwheel to emit manylinux2010 packages, and pip and pipenv to install them
Option to target manylinux2010 instead of manylinux1 in [various build utilities]

I’m also seriously missing a packager-oriented explanation of what you get and what you give up when you switch from 1 to 2010. I vaguely understand that you get somewhat newer compilers and shell tools (which?) and a build environment that doesn’t have to be hacked to run on modern host environments and is still receiving security patches (at least for a little while); is there more? I also vaguely understand that there is some set of old Linux distributions on which 1 wheels will work but 2010 wheels won’t; knowing just how old those are and how widespread they still are would really help people decide whether their package’s users will be inconvenienced by a change. And is there any other reason not to switch?

1 Like

It took me some time to find this thread.
I just want to say @pf_moore I understand your concerns, and we have been internally looking for ways to get us moving to help more with manylinux 2010 and manylinux 2014.
To show our commitment to these standards, and wanting to work together with the community, we have been working to get all our binaries manylinux 2010 compliant. as of last week, we were able to build and push all manylinux 2010 compliant binaries for TensorFlow nightly builds to pypi, including our GPU packages. all our releases from now on (except for security patches on 1.13 or older releases) will be manylinux2010 from now on.

As a part of this work, Manuel Klimek from Google has helped us by creating a toolchain that can be deployed on any linux distribution, and can produce manylinux compliant packages on ubuntu.
About how old the distributions that cannot run manylinux 2010, I think the most straightforward answer to this would be “any linux distro release that is before 2010” potentially won’t be able to run manylinux 2010 binaries?

I think with Manylinux 2010, as you pointed out the benefits mostly are newer toolchains, on OSs that are not EOL. But manylinux 2014 also introduces new platforms like arm and ppc to pypi. That potentially can be another motivation for packagers who want to provide packages for raspberry pi’s, or IBM.

2 Likes

Yannick Jadoul of cibuildwheel has questions on the manylinux2010 rollout issue, including:

  • should they make the manylinux2010 image the default one?
  • how widely has pip 19 been adopted?
  • is 32-bit officially deprecated for manylinux wheels?
  • is there a preference regarding uploading both manylinux1- and manylinux2010-compatible wheels to PyPI?

More details and nuance in the GitHub comment.

Also: I’ve created a rollout tracking issue for manylinux2014 support – please feel free to comment there, especially with tasks that need to happen but aren’t yet in the checklist, so we can improve the plan.