The next manylinux specification

Hi all,
Thank you Dustin for starting the discussion on this. And I a m sorry for joining the discussion very late.

One other idea I would like to float for consideration for “perennial manylinux” is, moving the ABI and other requirements from the package name to a “metadata” file. From TensorFlow or a similar library’s point of view, there are more requirements we may want to specify. Therefore, packing all of these into the name can be difficult. What I had in mind is for a certain “tag”, pip will check this metadata files, which stores key-value pairs that are requirements for the pip package. From what I can see, we can pack much more information into such a file, and resort to as little heuristics as possible.
I can certainly see a few potential issues around this. One is, we need to be able to generate the key-value pairs to be used in a deterministic way on many different systems. And how to introduce new keys to be used in this metadata file can be difficult to control. Another issue I see is, it may require a lot of changes in current tooling to land such a change.

While we were discussing within TensorFlow, we realized that a perennial manylinux tag can be a quite open-ended discussion. Diversity of the Linux distributions make this a quite difficult problem. That is why we wanted to propose manylinux2014 as a quick stop gap solution to address immediate issues, as we thought addressing the perennial manylinux question until CentOS 6 is EOL may be difficult.

I’m assuming you mean things like, fine-grained descriptions of what system libraries you want to have installed, the presence/absence of CUDA hardware/libraries, that kind of thing?

That’s an important discussion, but an extremely complicated one. In fact I think it’s fair to say that the reason we didn’t have Linux wheels at all for a ~decade is that every time we tried, we got bogged down in those kinds of discussions :-). Manylinux succeeded by being relentlessly pragmatic, and declaring all that stuff out-of-scope. I think we should stick to that for perennial manylinux.

That doesn’t mean we shouldn’t talk about extensions like this. Just, let’s split it out into a separate discussion. We can always define new tags; we aren’t in danger of running out of strings.

1 Like

These, and then more. For TF, which generation GPU is installed, which CPU family is available, how many CPU cores the system has can all be important.

I understand your concern. I will focus on the discussion here first, to ensure that we finalize the more pressing needs first.

To contribute to the original discussion, I am concerned about the diversity in the linux distributions. I am afraid it may be impossible to fully asses what gcc versions get packaged with which glibc versions.
While trying to provide flexibility, the problem may become more difficult, extending the timeline of such a proposal. So how about extending the approach taken so far by the existing manylinux standards. What if the perennial manylinux just picked a distro, such as CentOS, and just created a tag such as manylinux-<centos_version>-<arch>?
This would be a much easier solution to implement, IMHO. It is modeled off of the existing manylinux tags, and while it may not be immediately clear which glibc versions this allows, I would argue that it is not a regression from the existing tags.

I agree that this should be kept out of scope for this discussion. But for later consideration, I’d point out that it’s not at all clear how an installer (which may be pip, or may be some other tool) could establish what values the target environment supports.

As a simple example, if the metadata says “needs 4 CPU cores, and an i7 or better”, how would pip find that information out? Keep in mind that pip must only use pure Python code, to avoid the problem of having to distribute pip as platform-specific binary wheels.

Also, please keep in mind that every time a new criterion is added, the library that does this detection (and it should be a library, so that installers other than pip can also use it) would need updating. Again, it’s not at all clear to me how the spec and reference library management would be handled in practice.

1 Like

It doesn’t work, because there’s no way for pip on some random non-centos platform to tell whether the current platform is compatible with some random centos version. (Especially future centos versions.)

You’re right that using glibc as our “clock” shifts some burden to people building wheels, since they have to figure out how to produce wheels that are compatible with all systems that have that glibc version. But this is fundamentally unavoidable. Fundamentally we need some way for pip to calculate a “signature” from an arbitrary system, and then people building wheels have to find a way to produce wheels that are compatible with all systems that have that signature.

In practice, it’s not a problem, because auditwheel and the docker images aggregate the necessary knowledge about what does and doesn’t work, so people building wheels can just use them and get working wheels. (Or if they end up with wheels that don’t work, then they file a bug on auditwheel, and it gets fixed for everyone.)

Not really. “How to produce wheels that are compatible with all systems that have that glibc version” can be described in the manylinux specification instead (and/or implemented in official build images).

I’m not sure why shifting the work to multiple people (who will have to redo it independently, and may make mistakes in the process) is a desirable idea.

I must have been unclear because you’ve completely misunderstood my post.

Here’s a better way to express my point: there’s a tradeoff between doing work at install time versus doing work at build time. If your wheel tag is centos_<version>, like @gunan was suggesting, then that makes build time very simple – build on centos, done, no auditwheel or fancy docker images needed. But, it means that at install time we have to somehow figure out whether an arbitrary system is ABI-compatible with an arbitrary centos version, which is complex and impractical. The manylinux approach OTOH makes install time very simple – just check the glibc version. But, it makes build time more complicated. Your build process has to – in some way or another – incorporate knowledge of all the library versions that shipped with all the distros that shipped with a certain glibc version, and this is what @gunan is concerned about.

In practice it’s fine, auditwheel aggregates that knowledge, and then people use auditwheel, so individuals don’t have to redo it from scratch, and we know from experience that this is practical and works well. But it’s true that building is more complex than if you could just grab the latest centos and go to town. My point is that this build-time complexity is unavoidable if you want to make broadly-usable wheels, and it’s futile to try to come up with tagging schemes that avoid it.

Right… so, are we discussing the current manylinux approach (which is fine, modulo the fact that the latest spec may require outdated dependencies :wink: ), or the “perennial manylinux” proposal? I realize I may have misread this discussion.

When I wrote this:

…I meant the overall strategy that manylinux has been using all along, and that I think perennial manylinux should use too.

Not sure if that answers your question or not.

As I was reading the wheel spec I was wondering if it would be a good idea to have some more Classifiers which would describe which glibc(or others) are supported?
This can be useful for users to learn about wheel if we don’t to want to add it to the wheel file name.

Another option maybe to use details of environment markers also mentioned here
I am hoping we first agree to get more metadata into the spec.We then decide later what to do with it.for example how to enforce glibc etc.

I did not see pipenv mentioned in “brief history of pypa” .Who would push pipenv to be compatible with pip if we either have more tags or classifiers ?

@dustin, do you feel like you have enough to go on to start drafting a PEP? As a first pass I’d probably go with the manylinux_${glibc minor}_${platform} version from this post, and for the description of what a wheel needs to comply, explicitly walk through an example like the Ubuntu/Debian stuff here, and explicitly say that we expect auditwheel to provide profiles for some well-chosen glibc versions and prioritize those and that’s what people should use to build these things. Then of course we can bikeshed the details, but that’s easier with a draft :slight_smile:

1 Like

I can draft a PEP, but I have serious concerns about what the perennial manylinux proposal will require from maintainers, especially considering Elana Hashman just stepped down as a PyPA maintainer:

What are your concerns? It’s true that manylinux maintenance has struggled a lot due to lack of resources, and that will presumably continue to be true (unless by some chance you happen to know any large companies that have an interest in manylinux being well-maintained). But part of the goal of perennial-manylinux is to reduce the maintenance burden compared to traditional-manylinux.

As I understand it, a new manylinux tag currently requires: a PEP, updates to pip, PyPI, and auditwheel, and new build images (not strictly required, but I’m not aware of any manylinux2010 images created without the build images). With this proposal, no PEP would be needed, nor updates to pip or PyPI, but it would still need a new profile in auditwheel and a new build image.

I have some concern that we’re solving the wrong problem. Auditwheel and especially the build images are the hard parts where maintainers are in short supply, and they would still need updating. A new PEP takes some effort, but the framework can be copy-pasted, the research into library versions would need to happen for auditwheel anyway, and having a PEP gives us a definitive description of what the tag means. The PyPI changes are trivial, just adding the new tags to a whitelist. The pip changes appear to have lingered in a PR for a few months on this occasion, but I don’t think there’s anything that should prevent them being merged quickly in general.

I don’t think ‘perennial manylinux’ would make anything worse, although perhaps there’s more room for confusion over what exactly a given tag means. But a big part of the stated motivation is how long it’s taken for manylinux2010 to be ready for use, and I’m not convinced that this will produce a significant improvement.


I agree that auditwheel and the build images are the hardest part (especially this time, when we had to figure out how to cope with the kernel breaking compatibility so new systems can’t easily run CentOS 6 docker images!). That’s because they’re the parts that require actual work, not just bureaucratic overhead. But that also means that there’s not a lot we can do to fix that part. The bureaucratic overhead is the part we can get rid of.

We’re also going to be waiting some unknown amount of time before the new pip becomes ubiquitous across the install base. Many projects will still have to provide manylinux1 wheels until that happens, to avoid effectively regressing folks who haven’t upgraded pip recently.

Yeah, I’m not expecting miracles. But it’s still strictly better than doing a manylinux2014.

Also, cross-project coordination like this IME a lot harder to find volunteers for then incremental improvements to a single project. Even if the absolute amount of work involved in writing a PEP or updating warehouse isn’t large, there’s still significant cognitive overhead in figuring out what needs to happen, chasing people down in multiple projects, etc. And there’s a threshold effect: if someone has 10 units of energy they can contribute, but a piece of work needs 15 to complete, then they’ll probably contribute 0. (see also)

Fair enough. I agree that there is value in reducing the amount of cross-project coordination required.

Thinking about it, I think I’m also wary about the wider perception of this move. We’ve had manylinux1, then we changed to calendar based versioning for manylinux2010, and now we’re thinking of going to tags based on glibc versions. So we haven’t yet managed to keep any naming scheme for more than one round.

Perhaps that’s just something we have to swallow, and hope that the glibc-based scheme lasts longer. But it doesn’t do much for the impression that Python packaging is a confusing and unpredictable world.


As a concrete example of how having to update pip for new manylinux releases causes problems, here’s a project where they’re making manylinux2010 wheels, but then calling them manylinux1, because they’re impatient with users upgrading pip:

1 Like

Has anyone proposed discussing manylinux at the packaging mini-summit yet?

@njs has added a broad “binary packaging” suggestion.

we provide a “manylinux1” package while in reality it is a “manylinux10” so we modify the package meta using sed

From the GitHub issue @njs linked to.