The next manylinux specification

In principle, I agree. My understanding is that auditwheel is too heavyweight to do that easily.

What concerns me most, though, is that no-one seems to be pushing for the incorrectly tagged wheels to be removed from PyPI. In particular, I think the manylinux people should be more concerned about the impact this might have on the credibility of manylinux as a way of ensuring compatibility. Sure “practicality beats purity” and all that, but if we, as the authors of the standards, aren’t trying to make sure they are followed, what’s the point?

Anyway, this is off-topic. Let’s kill the digression here, and let this topic return to discussing how to define the next manylinux spec. If anyone wants to discuss the process of enforcing tag standards, they should start a separate topic (and to be honest, I’ve said all I want to say on the matter, so I won’t be participating).

3 Likes

If anyone wants to discuss enforcing these standards, the right place to do it would be on the issue I linked to in the OP: “Run auditwheel on new manylinux uploads, reject if it fails”.

1 Like

We haven’t had any additional comments here in a few days, so I wanted to attempt to summarize the relevant discussion and talk about next steps:

(Please feel free to correct me if I’m missing, misconstruing or misunderstanding anything that was said)

“Perennial manylinux”:

  • There is some confusion here about how only specifying the libc provider / libc version determines valid libraries to link against
  • Saying “if it works for your users, it’s valid” might be too vague, and we would still need to explicitly define install-time checks to determine compatibility
  • Generally it would be “if auditwheel (or another tool which does publication-time checking) allows it”, but we would need to point to such a tool in the specification
  • If the glibc provider/version is only being used as a heuristic, it’s confusing to use it in the tag name
  • It might be better instead make individual “heuristics” which can explicitly specify symbol version bounds for relevant libraries
  • This sounds a lot like manylinux!
  • Additional heuristics would also allow us to distro/VFX-specific heuristics
  • It seems like folks are on board with allowing additional heuristics if possible

manylinux2014:

  • Generally seems like folks value the detailed specification for binary compatibility such a proposal would provide
  • Doesn’t seem to be any complaints about adding the additional PowerPC/ARM architectures that CentOS 7 supports.
    • Unclear whether the proposed ARM architectures would be compatible w/ modern Raspbian

A combined proposal

If we can accept that manylinux is the “de-facto” glibc heuristic, perhaps we can continue to use this, with some additional flexibility, while remaining open to defining additional heuristics in the future.

@njs suggested a tag like:

manylinux_12_x86_64 (= installs on systems with glibc 2.12+)

This allows us to let the glibc version “slide” along the manylinux specification, but still has the issue of vaguely-specified symbol version bounds for relevant libraries.

What if we used a heuristic like:

manylinux2014_2_12_x86_64 (= installs on systems with glibc 2.12+, backed by a manylinux2014 spec)

We could explicitly specify some things (CXXABI, GLIBCXX, GCC, and externally provided libraries) in a manylinux2014 PEP, while still using the glibc as the “heuristic version identifier”. This means we don’t need a new PEP to use a different glibc version.

At a certain point in time, the glibc version will become too new compared to (CXXABI, GLIBCXX, GCC, etc) and we would still need to write a new manylinuxNNNN specification.

Would this work? Does it make sense? Would it be a good balance between the two ideas?

1 Like

This is really the worst of both worlds IMO. The goal isn’t to let people use arbitrary glibc versions; the goal is to let us migrate to new broadly-useful manylinux profiles without having to write new PEPs or add more hard-coded logic to pip, like how it works on Windows/macOS. We don’t care about the glibc version per se. We only care about it because it’s (1) a convenient shorthand to refer to a given “era” of linux distros, (2) it’s one that pip can mechanically compute on any system.

We can bikeshed over the exact spelling of the tag. The only technical requirement to achieve our goal is that the tag should be a fixed string with slots to put in glibc version and platform. And we can bikeshed over exactly what the PEP says, e.g. whether it defines auditwheel as “normative” or “not normative, but strongly recommended”. Those don’t really matter. But there shouldn’t be any 2014 anywhere, that’s just making life difficult for ourselves and doesn’t provide any benefits.

1 Like

OK, fair enough. But how else can we strike a balance between having this flexibility, and also satisfying maintainers desire to determine if they can build compatible distributions without just saying “auditwheel will tell you if you did it right or not”?

This sounds like we’re just pushing the “specification” into auditwheel, where it would be a) more difficult for interested parties to determine what the rules are, exactly, and b) more work for auditwheel maintainers (who are few and far between). It just wouldn’t require a PEP.

I didn’t see this question get answered specifically. Somewhere, whether it’s in a PEP or not, we need to specify bounds for these libraries, right? With “perennial manylinux”, at what point would we update them, and how? Or are they just completely unspecified, and we just let incompatibilities like this happen:

Everything I’m proposing is about standardizing the things we’re already doing, and that we know work. Auditwheel is already the most accurate, useful, and up-to-date record of how to build compatible wheels on Linux, because it’s what maintainers actually use. I’m pretty sure there have been fixes to the spec in auditwheel that we forgot to merge back into PEP 513, or where the PEP 513 update came later as an afterthought.

In practice, the specification already lives in auditwheel, so there’s no additional maintenance burden. It’s true that reading the rules would require reading some code, but in practice (a) the code is guaranteed to be accurate and up to date, unlike the PEP, (b) you’re still better off than on Windows or macOS, where the rules aren’t written down at all…

A manylinux_glibc_2_19_x86_64 tag would mean that the wheel is compatible with all mainstream distros using glibc 2.19+ on x86-64.

Ubuntu 14.04 is a mainstream distro using glibc 2.19+ on x86-64.

Therefore, to get a manylinux_glibc_2_19_x86_64 tag, you need to be compatible with Ubuntu 14.04.

Therefore, a random wheel built on Debian 8 probably does not qualify for the manylinux_glibc_2_19_x86_64 tag.

A random wheel built on Ubuntu 14.04 might qualify for the tag, but to be certain we’d have to first do some research to check whether there were any distros shipping glibc 2.19 with even older versions of gcc. And of course we’d also need to vendor any necessary libraries, etc.

Hi all,
Thank you Dustin for starting the discussion on this. And I a m sorry for joining the discussion very late.

One other idea I would like to float for consideration for “perennial manylinux” is, moving the ABI and other requirements from the package name to a “metadata” file. From TensorFlow or a similar library’s point of view, there are more requirements we may want to specify. Therefore, packing all of these into the name can be difficult. What I had in mind is for a certain “tag”, pip will check this metadata files, which stores key-value pairs that are requirements for the pip package. From what I can see, we can pack much more information into such a file, and resort to as little heuristics as possible.
I can certainly see a few potential issues around this. One is, we need to be able to generate the key-value pairs to be used in a deterministic way on many different systems. And how to introduce new keys to be used in this metadata file can be difficult to control. Another issue I see is, it may require a lot of changes in current tooling to land such a change.

While we were discussing within TensorFlow, we realized that a perennial manylinux tag can be a quite open-ended discussion. Diversity of the Linux distributions make this a quite difficult problem. That is why we wanted to propose manylinux2014 as a quick stop gap solution to address immediate issues, as we thought addressing the perennial manylinux question until CentOS 6 is EOL may be difficult.

I’m assuming you mean things like, fine-grained descriptions of what system libraries you want to have installed, the presence/absence of CUDA hardware/libraries, that kind of thing?

That’s an important discussion, but an extremely complicated one. In fact I think it’s fair to say that the reason we didn’t have Linux wheels at all for a ~decade is that every time we tried, we got bogged down in those kinds of discussions :-). Manylinux succeeded by being relentlessly pragmatic, and declaring all that stuff out-of-scope. I think we should stick to that for perennial manylinux.

That doesn’t mean we shouldn’t talk about extensions like this. Just, let’s split it out into a separate discussion. We can always define new tags; we aren’t in danger of running out of strings.

1 Like

These, and then more. For TF, which generation GPU is installed, which CPU family is available, how many CPU cores the system has can all be important.

I understand your concern. I will focus on the discussion here first, to ensure that we finalize the more pressing needs first.

To contribute to the original discussion, I am concerned about the diversity in the linux distributions. I am afraid it may be impossible to fully asses what gcc versions get packaged with which glibc versions.
While trying to provide flexibility, the problem may become more difficult, extending the timeline of such a proposal. So how about extending the approach taken so far by the existing manylinux standards. What if the perennial manylinux just picked a distro, such as CentOS, and just created a tag such as manylinux-<centos_version>-<arch>?
This would be a much easier solution to implement, IMHO. It is modeled off of the existing manylinux tags, and while it may not be immediately clear which glibc versions this allows, I would argue that it is not a regression from the existing tags.

I agree that this should be kept out of scope for this discussion. But for later consideration, I’d point out that it’s not at all clear how an installer (which may be pip, or may be some other tool) could establish what values the target environment supports.

As a simple example, if the metadata says “needs 4 CPU cores, and an i7 or better”, how would pip find that information out? Keep in mind that pip must only use pure Python code, to avoid the problem of having to distribute pip as platform-specific binary wheels.

Also, please keep in mind that every time a new criterion is added, the library that does this detection (and it should be a library, so that installers other than pip can also use it) would need updating. Again, it’s not at all clear to me how the spec and reference library management would be handled in practice.

1 Like

It doesn’t work, because there’s no way for pip on some random non-centos platform to tell whether the current platform is compatible with some random centos version. (Especially future centos versions.)

You’re right that using glibc as our “clock” shifts some burden to people building wheels, since they have to figure out how to produce wheels that are compatible with all systems that have that glibc version. But this is fundamentally unavoidable. Fundamentally we need some way for pip to calculate a “signature” from an arbitrary system, and then people building wheels have to find a way to produce wheels that are compatible with all systems that have that signature.

In practice, it’s not a problem, because auditwheel and the docker images aggregate the necessary knowledge about what does and doesn’t work, so people building wheels can just use them and get working wheels. (Or if they end up with wheels that don’t work, then they file a bug on auditwheel, and it gets fixed for everyone.)

Not really. “How to produce wheels that are compatible with all systems that have that glibc version” can be described in the manylinux specification instead (and/or implemented in official build images).

I’m not sure why shifting the work to multiple people (who will have to redo it independently, and may make mistakes in the process) is a desirable idea.

I must have been unclear because you’ve completely misunderstood my post.

Here’s a better way to express my point: there’s a tradeoff between doing work at install time versus doing work at build time. If your wheel tag is centos_<version>, like @gunan was suggesting, then that makes build time very simple – build on centos, done, no auditwheel or fancy docker images needed. But, it means that at install time we have to somehow figure out whether an arbitrary system is ABI-compatible with an arbitrary centos version, which is complex and impractical. The manylinux approach OTOH makes install time very simple – just check the glibc version. But, it makes build time more complicated. Your build process has to – in some way or another – incorporate knowledge of all the library versions that shipped with all the distros that shipped with a certain glibc version, and this is what @gunan is concerned about.

In practice it’s fine, auditwheel aggregates that knowledge, and then people use auditwheel, so individuals don’t have to redo it from scratch, and we know from experience that this is practical and works well. But it’s true that building is more complex than if you could just grab the latest centos and go to town. My point is that this build-time complexity is unavoidable if you want to make broadly-usable wheels, and it’s futile to try to come up with tagging schemes that avoid it.

Right… so, are we discussing the current manylinux approach (which is fine, modulo the fact that the latest spec may require outdated dependencies :wink: ), or the “perennial manylinux” proposal? I realize I may have misread this discussion.

When I wrote this:

…I meant the overall strategy that manylinux has been using all along, and that I think perennial manylinux should use too.

Not sure if that answers your question or not.

As I was reading the wheel spec I was wondering if it would be a good idea to have some more Classifiers which would describe which glibc(or others) are supported?
This can be useful for users to learn about wheel if we don’t to want to add it to the wheel file name.

Another option maybe to use details of environment markers https://www.python.org/dev/peps/pep-0508/#id23 also mentioned here https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension.
I am hoping we first agree to get more metadata into the spec.We then decide later what to do with it.for example how to enforce glibc etc.

I did not see pipenv mentioned in “brief history of pypa” .Who would push pipenv to be compatible with pip if we either have more tags or classifiers ?

@dustin, do you feel like you have enough to go on to start drafting a PEP? As a first pass I’d probably go with the manylinux_${glibc minor}_${platform} version from this post, and for the description of what a wheel needs to comply, explicitly walk through an example like the Ubuntu/Debian stuff here, and explicitly say that we expect auditwheel to provide profiles for some well-chosen glibc versions and prioritize those and that’s what people should use to build these things. Then of course we can bikeshed the details, but that’s easier with a draft :slight_smile:

1 Like

I can draft a PEP, but I have serious concerns about what the perennial manylinux proposal will require from maintainers, especially considering Elana Hashman just stepped down as a PyPA maintainer:

What are your concerns? It’s true that manylinux maintenance has struggled a lot due to lack of resources, and that will presumably continue to be true (unless by some chance you happen to know any large companies that have an interest in manylinux being well-maintained). But part of the goal of perennial-manylinux is to reduce the maintenance burden compared to traditional-manylinux.

As I understand it, a new manylinux tag currently requires: a PEP, updates to pip, PyPI, and auditwheel, and new build images (not strictly required, but I’m not aware of any manylinux2010 images created without the build images). With this proposal, no PEP would be needed, nor updates to pip or PyPI, but it would still need a new profile in auditwheel and a new build image.

I have some concern that we’re solving the wrong problem. Auditwheel and especially the build images are the hard parts where maintainers are in short supply, and they would still need updating. A new PEP takes some effort, but the framework can be copy-pasted, the research into library versions would need to happen for auditwheel anyway, and having a PEP gives us a definitive description of what the tag means. The PyPI changes are trivial, just adding the new tags to a whitelist. The pip changes appear to have lingered in a PR for a few months on this occasion, but I don’t think there’s anything that should prevent them being merged quickly in general.

I don’t think ‘perennial manylinux’ would make anything worse, although perhaps there’s more room for confusion over what exactly a given tag means. But a big part of the stated motivation is how long it’s taken for manylinux2010 to be ready for use, and I’m not convinced that this will produce a significant improvement.

2 Likes