The next manylinux specification

I’ve documented a bit the wheel building of pyarrow at https://uwekorn.com/2019/09/15/how-we-build-apache-arrows-manylinux-wheels.html

The most interesting for this thread here is that I have a minimal reproducer of the cause of the import pyarrow; import tensorflow segmentation fault. TL;DR it boils down to Ubuntu 14.04 gcc-4.8 and manylinux1-devtoolset gcc 4.8 producing different implementations for void std::__once_call_impl<std::_Bind_simple<void (*())()> >(). Hope this is enough information for others to dive more into this problem.

4 Likes

Of course the problem is not only that these implementations are different, but they are incompatible (probably rely on different static variables). Which is why the backtrace of the crash looks like the CPU jumps to address 0 at the point where it should jump to the actual function wrapped in std::call_once.

However, I’d like to stress again that our (PyArrow’s) problem is not merely the Tensorflow incompatibility (this was primarily due to Tensorflow violating the manylinux1 spec) but the fragility of a build chain where we have to rebuild and bundle all dependencies manually with little assistance from existing tools.

For the record, our build chain for manylinux2010 isn’t significantly different from our manylinux1 build chain. In manylinux2014, we might end up being able to pull more dependencies from the build system instead of building them from scratch. But of course it remains to be seen whether we build manylinux2014 wheels at all. We’ll need a volunteer to take up.

Great detective work! That’s definitely a big step towards figuring this out. But, the biggest mystery is still unresolved: each python extension loads in its own ELF namespace, so even if two extensions have conflicting symbols, it shouldn’t matter, because they can’t see each other’s symbols. But somehow in the tensorflow/pyarrow setup, something is breaking this isolation. How is that happening?

There’s a possible problem with your reproducer script: your main program that calls dlopen is written in C++, while in the real CPython case, the main program that calls dlopen is written in C. The reason this could be a problem is that when the main program is written in C++, it basically LD_PRELOADs all the system’s libstdc++ symbols on top of every module you dlopen. (Yes, this is super weird. I didn’t design it…) When the main program is written in C, this doesn’t happen. So probably the next step is to rewrite the reproducer in C and see if the same problem occurs or not.

Right, procuring all the libraries that you want to vendor into your wheel is definitely a challenge – CC @sumanah for an example of a challenge facing scientific users. One of the ideas that’s come up in the past is to use conda-forge as a source of vendorable, prebuilt binaries, e.g. by setting up a conda env and then building the wheel and running auditwheel there. I’m not sure exactly which manylinux version conda is targeting right now – it might be one of the new ones we need perennial manylinux to define – so someone would need to figure that out too. Otherwise though, this is getting into a different topic entirely from manylinux PEP definitions, so we should probably continue the discussion in a different thread.

I took extra care that the main script is in C and compiled with gcc instead of g++, so this should map to the CPython case.

@uwe, when you say that PyArrow 0.14.0 fixed the issue, you’re mistaken. On Ubuntu 18.04 with Python 3.7 I can do:

$ pip install pyarrow==0.14.0 tensorflow==1.13.1
$ python -c "import pyarrow, tensorflow"
Erreur de segmentation

Here is the stack trace in this case:

#0  0x000000000000003f in ?? ()
#1  0x00007ffff77d4827 in __pthread_once_slow (
    once_control=0x7fffbad5c1a8 <tensorflow::port::(anonymous namespace)::cpuid_once_flag>, 
    init_routine=0x7fffc6831830 <__once_proxy>) at pthread_once.c:116
#2  0x00007fffba67249a in void std::call_once<void (&)()>(std::once_flag&, void (&)()) ()
   from /home/antoine/t/venv/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so
#3  0x00007fffba6724de in tensorflow::port::TestCPUFeature(tensorflow::port::CPUFeature) ()
   from /home/antoine/t/venv/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so
#4  0x00007fffba308f35 in _GLOBAL__sub_I_cpu_feature_guard.cc ()
   from /home/antoine/t/venv/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so
#5  0x00007ffff7de5733 in call_init (env=0xb7d250, argv=0x7fffffffd8f8, argc=3, l=<optimized out>)
    at dl-init.c:72
#6  _dl_init (main_map=main_map@entry=0xfd1fc0, argc=3, argv=0x7fffffffd8f8, env=0xb7d250)
    at dl-init.c:119
#7  0x00007ffff7dea1ff in dl_open_worker (a=a@entry=0x7fffffff7dc0) at dl-open.c:522
#8  0x00007ffff7b4b2df in __GI__dl_catch_exception (exception=0x7fffffff7da0, 
    operate=0x7ffff7de9dc0 <dl_open_worker>, args=0x7fffffff7dc0) at dl-error-skeleton.c:196
#9  0x00007ffff7de97ca in _dl_open (
    file=0x7fffc570eb78 "/home/antoine/t/venv/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so", mode=-2147483646, caller_dlopen=0x641d20 <_PyImport_FindSharedFuncptr+112>, 
    nsid=<optimized out>, argc=3, argv=<optimized out>, env=0xb7d250) at dl-open.c:605
#10 0x00007ffff75c1f96 in dlopen_doit (a=a@entry=0x7fffffff7ff0) at dlopen.c:66
#11 0x00007ffff7b4b2df in __GI__dl_catch_exception (exception=exception@entry=0x7fffffff7f90, 
    operate=0x7ffff75c1f40 <dlopen_doit>, args=0x7fffffff7ff0) at dl-error-skeleton.c:196
#12 0x00007ffff7b4b36f in __GI__dl_catch_error (objname=0xb80af0, errstring=0xb80af8, 
    mallocedp=0xb80ae8, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:215
#13 0x00007ffff75c2735 in _dlerror_run (operate=operate@entry=0x7ffff75c1f40 <dlopen_doit>, 
    args=args@entry=0x7fffffff7ff0) at dlerror.c:162
#14 0x00007ffff75c2051 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#15 0x0000000000641d20 in _PyImport_FindSharedFuncptr ()
#16 0x000000000062517e in _PyImport_LoadDynamicModuleWithSpec ()
[... snip ...]

@njs Note the stack traces involve pthread_once and/or __pthread_once_slow, which are part of the pthreads library and probably loaded by CPython at startup.

Oh, you’re totally right, I misread. Hmm. Would it be easy to share the .sos you built, since building them is kind of complicated?

Right, but those are coming from glibc, and everyone is supposed to be sharing a single copy of them. Our working hypothesis is that pyarrow has a vendored copy of std::once that was automatically inserted by the manylinux compilers (as part of their hacks to make it possible to use new C++ features on systems with an old version of libstdc++), and that tensorflow is using the version of std::once from the system’s copy of libstdc++, and somehow these two copies of std::once are interfering with each other. None of that applies to pthread_once.

(noting that I’ve flagged a post, as a request to our Discourse admins, to separate the “why do we have crashes” investigation into a separate topic on this category)

Where/how is this done? (I don’t see any references to dlmopen in the CPython codebase)

I didn’t know dlmopen, but I’m discovering this interesting snippet in the dlopen / dlmopen man page:

The dlmopen() function also can be used to provide better isolation than the RTLD_LOCAL flag. In particular, shared objects loaded with RTLD_LOCAL may be promoted to RTLD_GLOBAL if they are dependencies of another shared object loaded with RTLD_GLOBAL.

I just meant the namespacing you get from RTLD_LOCAL.

9 posts were split to a new topic: Moving forward with the next manylinux specification

Any update on this? It’s been a bit over a month since you made that comment, and manylinux2014 is moving along. I don’t want to get into a debate over when the rollout will be considered to “have happened” but hopefully you at least have a better idea of when you expect a proposal to be ready (in ballpark terms if nothing else).

We tried a couple of avenues for finding funding quickly, but none of them panned out. We are going to have to write a grant proposal to someone. Due to other commitments and travel, the earliest we can start on that is going to be November, and I can’t say when the funding will actually materialize.

Having thought about this some more in the interim, though, I now agree with @njs that, if the ultimate solution for mysterious C++-related crashes requires changes to the manylinux specifications, those changes will be orthogonal to the changes from manylinux20xx to perennial, and therefore this outstanding bug should not be a blocker for perennial.

I still think the perennial PEP should be deferred until after the manylinux2014 transition is complete and we have seen all of the fallout from it. “Complete” means something like “more than 66% of all wheels downloaded from production PyPI daily, to Linux, containing compiled code, are manylinux2014 wheels.” This is entirely because of process-related unknown unknowns; I don’t think we can be confident of being able to tell whether perennial is completely specified until we see the manylinux1 → manylinux2014 transition play out in production.

Why would there be such a transition?

If a wheel currently published with manylinux1 works fine, won’t “upgrading” to manylinux2014 just reduce the number of users who can install it?

I certainly had the impression that wheels should be built/distributed under the earliest possible tag, and the newer tags are to help support those that currently won’t work with the earlier ones.

2 Likes

The more important driver for a transition, AIUI, is that the build environments for the older tags are based on versions of CentOS that are either past (ml1) or nearing (ml2010) their end-of-life date.

At some point in this very long thread it was suggested that the new build environments should tag wheels with the older tags when they don’t need any newer library features (it should be possible to detect this automatically). I don’t know if that actually got implemented. Assuming it did, though. that would indeed be a reason why my suggested definition of “complete” wouldn’t work. Let me try again:

I claim we won’t have enough information to judge whether the perennial proposal makes sense until both of the following are true:

  • A supermajority of the daily connections to production PyPI, by pip running on Linux, were a version of pip that understands the manylinux2014 tag
  • A supermajority of the wheels on production PyPI that contain compiled code for Linux have had an upload, with a “final release” version number, that was compiled in the manylinux2014 build environment

Once those are both true, we will need to canvass the community of people who have built wheels in the manylinux2014 build environment, and the community of people who have downloaded wheels for use on Linux, to find out whether there were any unexpected problems arising from the transition that need addressing by changes to the perennial process. (Probably we’ll hear about some problems in the form of bug reports on pip, auditwheel, the build environment, and specific compiled-code packages, but I don’t think we’ll discover all of the problems if we don’t do some outreach.)

+1 to that!

Now that manylinux_2_28 is a reality, would it be time to start talking about the next manylinux version? Would that be manylinux_2_34, using glibc 2.34, based on a version 9 BaseOS? (maybe AlmaLinux 9?)

Currently multiple popular releases like Fedora 35 and 36 and Ubuntu 21.10 and 22.04 LTS already support glibc 2.34+. Also many rolling releases like Gentoo, Manjaro Stable and openSUSE Tumbleweed support it. Especially for Ubuntu 22.04 LTS a new manylinux image could be very useful, it’s also used a lot in CI.

I think the discussion for the base image of 2_28 showed that it’s good to stay based on RHEL and its derivatives (if only already for the devtoolset backports!). They should all be ABI-compatible anyway, though there’s slight variations (see OP of that discussion). Provided that rhubi 9 comes with a current devtoolset, I think that would be the most attractive option.

It would also continue the pattern so far:

manylinux glibc RHEL
manylinux1 2.5 5
manylinux2010 2.12 6
manylinux2014 2.17 7
manylinux_2_28 2.28 8
manylinux_2_34 2.34 9

There’s a pretty large glibc-gap between RHEL 7 & 8, but the debian-based 2_24 was struggling with adoption particularly due to the lack of modern compilers, and is almost EOL, so I’m not counting it.

All that being said, I think there’s really no rush for this. The motivating/constraining factor here is not CI usage, but how many users have a new enough glibc (& pip) to consume manylinux_2_34 wheels. The answer is about 10% of python 3.10 users, and microscopic amounts of all other python users. That is not a reasonable user base (currently) to justify the maintenance effort for most projects to publish such wheels (in constrast, glibc 2.28+ is quite-to-very widespread for everything but python 3.7, which is starting to drop off the radar anyway, cf. e.g. NEP29).

4 Likes

Thanks for this interesting statistic!

This number is now growing quite steadily, to 24% currently. Considering the time it will take to consensus, implementation and adaptation, I don’t think it’s too early to start discussing this,

1 Like