The next manylinux specification

Packaging
81

I wrote up a draft because I wanted to keep some movement towards further manylinux profiles, but I don’t have a strong preference between doing manylinux2014 and going straight to ‘perennial’ manylinux.

Since my doubting post, I’m more convinced that there’s value in not having to wait for new versions of pip to spread before a new manylinux flavour is viable. But I’d also quite like to have two profiles using the same versioning scheme before we change again.

82

This is a good point. It seems to me there are two essentially independent issues here:

  1. Defining and signing off “manylinux-next”, so that people can start work on implementing it.
  2. The higher level question of “how do we remove roadblocks so that the process of updating manylinux is smoother?”

The tension is between agreeing something now, so the people implementing “manylinux-next” aren’t held up by the standards process, and making sure we don’t agree something that just perpetuates the problem.

As someone who’s not involved in the manylinux implementation work, can I ask - what’s the level of urgency on getting the next manylinux spec agreed? Specifically:

  1. Since the original post here, do we have a manylinux2010 build environment? More generally, is manylinux2010 “complete” yet?
  2. Are people waiting to start work on “manylinux-next”, and blocked by the lack of a definition (as opposed to, say, wanting a nice long lie down now that 2010’s done with :slight_smile:)?
  3. Is the expectation that manylinux-next development will be smoother than manylinux2010 (i.e., were the delays and problems with 2010 one-off issues, or are they inherent in the process)?

I ask because it seems to me that manylinux2014 is likely to be easier to finalise quickly than perennial manylinux, but I don’t have a feel for whether “get something out quickly” is an important goal.

83

Yes, at least for x64 architectures, we have a build image published which people can use. Supporting x86 seemed to be more challenging, so we decided to pursue the 64-bit image separately. In principle, we’re still open to an x86 image if someone does the work, but it represents a pretty small fraction of users now.

84

My understanding of the status is:

  • manylinux2010 is basically done on our end. We’re waiting for project maintainers to switch, and project maintainers are waiting for users to upgrade pip, but I don’t think there’s much we can do about that except wait.
  • the reasons to work on manylinux-next are:
    • CentOS 6 goes EOL in November 2020, so ~18 months from now. That means that ideally we’ll have manylinux-next support shipping in pip by, say, 9 months from now, so that it has time to percolate to users before we the EOL date. This isn’t super urgent, but it has to happen soon-ish-ish. And hey, right now everyone’s memories are fresh from manylinux2010.
    • Dustin has indicated that the corporate users he’s been talking to want manylinux-next more urgently than that, but I’m not yet clear on exactly why – hopefully their upcoming PEP will have some more details here.
  • I think the perennial manylinux proposal is pretty well understood at this point. I reviewed Thomas’s PEP draft yesterday, and there’s a bit of bikeshedding over names and phrasing still to work through, but nothing really substantive.

I think the big remaining question is: to me, the perennial manylinux approach seems obviously correct – it’s a strict improvement over how we did things before, has effectively no downsides, and doing a manylinux2014 instead would be shooting ourselves in the foot for no reason. My impression is that Dustin and the folks he’s talking to disagree, and think that the manylinux2014 approach is better. I don’t understand why they think what they think, and they might not fully understand why I think what I think. So the next big step is to sort out that confusion and at least get on the same page about who thinks what and why.

85

To summarise the naming discussion (/bikeshedding :wink: ) for the ‘perennial’ proposal:

  • The draft I wrote follows @dustin’s idea of using tags like manylinux_glibc_2_17_x86_64. This is lengthy, especially as part of an already long filename, but arguably makes it clear where the number is coming from.
  • @njs prefers tags like manylinux_17_x86_64. I’ll attempt to paraphrase his arguments:
    • Shorter filenames are easier to work with.
    • Putting glibc_ in there implies that it’s easy to extend this scheme to other libc implementations, which isn’t the case. If we do want to support Alpine Linux (which uses musl), a distro-specific alpine tag looks more practical than a new flavour of manylinux. And there’s still plenty of namespace for something like musllinux if we want to go that way in the future.
    • The 2_ is probably redundant because it seems highly unlikely that there will be a glibc 3.0.
    • Some people see glibc_2_17 and think it only supports one specific version of glibc, which is not the intent.

See Nathaniel’s full description of these points.

I don’t have a strong view one way or the other. If no-one speaks up to make the case for the longer tags, I’ll probably change the draft to Nathaniel’s preference, because I think he’s thought more about it and I respect his judgment.

86

Thanks. I agree - it sounds like we currently have two relatively well fleshed out proposals, and the main step will be to decide between them. I don’t have the background knowledge to have a preference from where we are now, so I’d strongly prefer that we get everyone to the point where they understand each other, at which point either we’ll all agree, or we’ll be able to state the trade-offs in a way that everyone is happy with, and picking a preferred set of trade-offs in that situation will be a lot easier.

87

Specifically, we need users to upgrade to pip 19.0 (2019-01-22) or newer, right? (And I see that version still runs on Python 2.7.)

(If there’s an existing discussion of what we’ve tried in the past to get people to upgrade pip faster, and how well it’s worked, and what we could do in the future (especially if we had budget for some marketing), could folks point me to it? Otherwise I’m happy to start a different discussion about that so as not to derail this one.)

88

I think that should be a separate discussion. It’s likely to be complex enough that it would be a distraction here.

1 Like
89

Yes. manylinux2010 wheels can only be installed with pip >= 19.0.

90

Here’s the draft for a manylinux2014 PEP. It’s largely a continuation of manylinux2010 and thus borrows heavily from PEP 571: https://github.com/python/peps/compare/master...di:manylinux2014?expand=1

I’m not really in a place to declare whether this is objectively better than another proposal, however I would like to raise two points:

  • This PEP may seem trivial, but it is the result of a lot of discussions with relevant stakeholders and heavy manylinux users who have historically not had a lot of input here. It represents what those users feel would be best for them, as package maintainers.
  • In addition, these folks are serious about helping implement this PEP, including providing relevant base images (which was probably the biggest hurdle for manylinux2010).

In order to organize this work, the Packaging-WG has approved @sumanah to help with the decision making process and do project management for whatever the next manylinux specification becomes, so if this PEP was accepted, I feel we would actually be able to effectively harness these folks and get things implemented.

I think this is important to note because we don’t exactly have a plethora of people on the PyPA side of things who have the time/energy/context/desire to implement the next specification. This isn’t to say that that these volunteers would refuse to work on a perennial manylinux specification, but it may be more challenging to get them on board with a specification they didn’t propose. (It may also be a strong argument for whichever specification requires less implementation).

To respond directly to @pf_moore’s questions from my perspective talking to this group:

Depends on who you mean by “people”, I’d say that the folks I’ve been working with are eager for the next specification to be determined. Considering that CentOS 6 is close to EOL, and that they’re hearing that there is the potential for their current distributions to be blocked from publication on PyPI (pypa/warehouse#5420), it’s hard to motivate the work necessary to make the distributions manylinux2010-compliant (although it is happening), they would prefer to just move towards being manylinux2014-complaint instead.

I think if we move forward with the manylinux2014 proposal, given our prior experience, the interest of third parties in assisting with implementation, and @sumanah’s oversight, it should take less time to get to the same point we are at today.

3 Likes
91

Thanks! I’ll quote the bit that compares manylinux2014 vs the equivalent perennial-manylinux-based manylinux_17, since that’s the part that’s probably most relevant here:

The manylinuxYYYY pattern has a number of advantages that motivate continuing with the current status quo:

  • Well-defined Docker images with clearly specified compatible libraries;
  • No need to survey for compatibility issues across multiple releases;
  • A single build image and auditwheel profile per architecture.

There are also some disadvantages:

  • Requires drafting a new PEP for every new standard;
  • Requires adding the new platform tag to installers (e.g., pip );
  • Installers are unable to install a platform tag which predates a given release.

I agree that these are disadvantages of manylinux2014 vs manylinux_17. But… the advantages actually apply exactly the same to both approaches: either way we’ll have well-defined docker images, either way we’ll have to do exactly the same amount of research, and in both cases there’s a single build image+auditwheel profile if you’re only looking at this particular era, and not if you’re looking more broadly (e.g. manylinux2010 will still exist either way).

92

Maybe this is what I’m missing about the perennial proposal. If our heuristic is roughly tied to the glibc version, doesn’t this mean that we have N ☓ M total possible images/profiles (where N is the number of architectures, and M is the number of possible versions of the heuristic)? Whereas manylinux2014 would just have N images/profiles.

93

There’s a universe of NxM total heuristics we might potentially choose to use; that follows from the basic idea of using glibc as our clock. The question is how to break up that universe into manageable pieces.

In the manylinuxXXXX framework, every few years we pick a subset of those NxM heuristics, and write a PEP about it + create an auditwheel profile + create a build image.

In the perennial manylinux framework, we write one PEP, and then every few years we pick a subset of those NxM heuristics and create an auditwheel profile + create a build image.

The auditwheel profiles and build images are exactly the same either way; it’s just an accounting trick where we stop writing so many PEPs. This does mean that the number of auditwheel profiles+build images per PEP will eventually be higher in the perennial manylinux framework, so if you’re focused in on a single PEP it might seem like perennial manylinux produces a relative proliferation of profiles+images. But that’s an illusion – in absolute terms, we’ll have the same number of profiles+images; the ratio changes because the number of PEPs changes.

94

Speaking from my newfound experience, the hard part here is definitely the “pick a subset” part, not the “write the PEP” part. It’s not very compelling to me that we avoid writing a handful of PEPs over the next ~decade. Writing the PEP is easy as long as we’re not reinventing the wheel (literally) every time we’re due for an update. :wink:

It is somewhat compelling to me that we could somehow avoid shipping updates to installers like pip, but I’m not sure that outweighs the advantage of being consistent here. Basically I agree with @takluyver here:

Also, it’s worth mentioning that accepting a manylinux2014 proposal now doesn’t prevent us from continuing to move towards something like perennial manylinux in the future, however the inverse is not true.

95

I agree, though I probably put more value on avoiding extra PEPs than you do, since I remember the year+ I spent failing to convince anyone to write the last PEP. It was just as simple, and I kept explaining that, but somehow it didn’t help :slight_smile:. Coordinating across multiple projects is deceptively difficult, esp. for volunteers who come and go. But yeah, getting ahead of the pip upgrade cycle is definitely the biggest practical advantage.

Can you elaborate some on the advantages you see to being consistent? The only concrete one I can think of is that it’s embarrassing to have to tell everyone we had a better idea, but to me that seems plainly outweighed by the technical advantages.

At a technical level, manylinux1 and manylinux2010 are exactly the same; the change in versioning is purely cosmetic. That’s why I feel confident enough that the core ideas are solid and we can streamline them.

96

From my point of view, it seems like a step backwards to move towards an implementation defined standard again. Are people likely to write a competitor to auditwheel? Well no – although I’ve defintely seen people who wanted to compile the code using their infrastructure and were OK with carefully crafting wheels in order to comply with the PEP-- but I think there is real value in making it so people could. We’ve spent a significant amount of effort, and still have a significant amount of effort left to go, of trying to get out from under the historical weight of implementation defined behavior.

As far as getting people to write the actual PEPs… I think that if Warehouse can start validating uploaded wheels, that is a huge driver for people (particularly companies) to put effort into this rather than just hold onto existing standards. Maybe a better way would be to basically write up a PEP template for new manylinux variations, hell there could even be a small tool that generates one from a auditwheel profile so the extra work is super minimal BUT it makes there be an obvious place to describe changes when something has to be done that is more than just revving some glibc version number.

3 Likes
97

Right now I’m pretty sure we actually have an implementation defined standard. We’ve tried to keep the PEPs in sync with auditwheel, but in practice we’ve probably missed stuff, and where they’re different it’s auditwheel that’s right, not the PEPs.

With the proposed perennial manylinux, the standard is “if your wheel works on all computers where pip installs it, then it’s compliant”. If someone wants to implement an auditwheel competitor that accomplishes that, then cool. If their wheels meet that standard can’t be uploaded to warehouse because they fail auditwheel’s validation, then that’s a bug in auditwheel’s implementation of the spec and we can fix it.

My extremely vague and unscientific impression is that profile changes are 90% simple and declarative, and 10% quirky stuff that involves code changes. So a tool to dump auditwheel profiles into readable text wouldn’t be 100% accurate, but it could be useful.

It seems like the simplest thing to do would be to have this tool dump text onto some pages on auditwheel.readthedocs.io, updated with every commit? That’s totally doable without bringing in the overhead of pushing PEPs through the review process and then getting pip and warehouse patches through their review processes.

98

Please be careful here - what if I wrote another installer (not an entirely theoretical question…)? The correct wording should be “if your wheel works on all computers where the manylinux standard says it’s compatible, it’s compliant”. And that sort of begs the question of what the standard says (which is basically “it has a certain glibc version”). The standard describes that as a “heuristic”, but in practice it’s our definition of what “compliant” means.

I get your point that all tags have a certain level of “practicality vs purity” about them - “win32” doesn’t mean “works on Windows 95”, for example - but I think we need to look at this from a user perspective, and while I have a good feeling for the type of user who’d be using “win32” and whether they would be right to expect a win32 wheel to work for them, I have much less clarity on what a typical “glibcX.Y user” might have available, and what extremes can reasonably be considered “out of scope”. Isn’t the whole point of manylinux that just basing the decision on glibc didn’t work? Has something changed? (Or am I wrong in my understanding of why manylinux 1 and 2010 specify a set of libraries, rather than just one?)

99

Thanks! What level of C++ is possible given the constraints on CXXABI_1.3.7 and GLIBCXX_3.4.19? C++14, C++17? Or is there no such limitation?

100

The manylinux2010 images already include a fully up to date compiler toolchain. There might be some corner cases that 2014 images will handle better, but there are no blanket C++ version limitations in any modern manylinux.