The next manylinux specification

[zwol (Zack Weinberg)]

@njs

The manylinux2014 PEP is literally a copy/paste of the manylinux2010 PEP with minor updates, which in turn was literally a copy/paste of the manylinux1 PEP with minor updates, and PEP 600 is simply replacing the copy/paste with a generic template.
…
In this case, PEP 600 is very conservative and low risk to start with, and I think it’s very unlikely that going through a third roll-out is going to reveal some huge new issue that we missed in the first two roll-outs.

I think I finally understand why you think perennial is done and I don’t.

Perennial aims to be the final PEP on the subject of what makes a “manylinux” wheel. But it only covers the same subject matter that the earlier PEPs (for manylinux1, -2010, and -2014) covered. You think that’s all that it needs to cover. But I think it also needs to cover the process of updating everyone involved from version X to version X+1. And it is exactly that process that has never yet successfully been completed. The ml2010 and ml2014 PEPs have been published, but most everyone is still using ml1!

My position is that until we carry out one of those migrations successfully, we won’t even know what the missing pieces of the perennial design are, and therefore accepting the PEP would be premature.

@njs

If you’re right and it turns out that tying manylinux versions to glibc version numbers really is unworkable, then we would need to tear up all the existing manylinux PEPs and start over.

I think I may have misstated my concern here. Whether the version number of the tag is based on glibc version numbers, years, or whatever doesn’t prevent someone (the PEP writers, under the old process; the auditwheel and build-image maintainers, under perennial) from specifying that official “manylinuxWHATEVER” wheels are to be built using glibc vX, g++ vY, etc. And that certainly ought to be enough to deal with the actual C++ binary compatibility problems we have observed to date. (I reserve the right to retract this statement based on the conclusions of the root-cause analysis for the pyarrow+tensorflow crash.)

The problem is what happens if someone builds unofficial wheels in their custom lash-up environment that has, I dunno, glibc 2.12 but the very newest C++ compiler, and calls that a manylinux_2_12 wheel and ships it to their fanbase - probably outside PyPI - and then PyPA takes the blame when it doesn’t work properly. You know, exactly like what happened with TensorFlow and manylinux1.

With year-based versioning we have a nontechnical defense against this kind of behavior: we can say “that’s not really a manylinux2010 wheel.” I fear it will be harder to make that argument with version numbers that are explicitly tied to a glibc version number but not to anything else.