PEP 600: Future 'manylinux' Platform Tags for Portable Linux Built Distributions

Thanks Thomas, that did help clarify for me.

I’ve realised that this is the bit that I struggle with. Coming from Windows, that feels to me like a terrible user experience - I upgrade my OS in place, and wheels that worked no longer do? Or I build a new PC with a later version of my OS, and wheels that claim to work on that OS fail to do so? That to me seems like such an unacceptable situation that I can’t understand why Linux users accept it. But I have to accept that it’s the norm for Linux, and not let my perceptions based on different experiences mean that I hold this PEP to standards that the intended users view as too strict. (That’s not intended as any sort of “my OS is better than yours” comment - if I failed to avoid it sounding like it was, my apologies). I’ll keep that in mind from now on.

OK, with that aside, and having re-read the previous thread and the spec, I have the following comments. I hope most of these are minor, but I’ve learned not to try to guess what will be controversial in this topic :slight_smile:

Documentation

I still feel that the PEP should say that the rules that define the various manylinux profiles should be documented in human-readable form somewhere. Call them auditwheel specs, or guidelines, or whatever you want, but a number of people have expressed the point that they want to be able to read the definitions. The PEP explicitly says it doesn’t intend to make auditwheel normative, but without any rules, the PEP is essentially meaningless, so I think it’s reasonable to expect that some rules should be documented, if only as a baseline for others. And realistically, the auditwheel rules are the best we have in terms of “our current understanding”.

I’m perfectly OK with not being any more formal than “the definitions should be documented”. No need for processes around updating, claims that specs and implementations will never get out of sync, anything like that. Simply an acknowledgement that if people want to know, they should be able find out without deciphering code. I’m assuming that if the PEP makes this statement, the developers will make a good faith attempt to conform to it - I’m not going to get hung up over the possibility of someone rules-lawyering their way out of providing something acceptable. I find it hard to believe that something at that minimal a level would be controversial.

(This is notwithstanding @takluyver’s explanation - I still believe that even though he successfully argues that the spec is “just our current knowledge”, that doesn’t mean that we shouldn’t document what “our current knowledge” is).

Previous manylinux PEPs

It seems that PEP 600 supersedes and obsoletes the previous manylinux PEPs, as noted here. I think it would be worth explicitly stating that the existing PEPs will be marked as obsolete, and no longer maintained. It’s a key point of PEP 600 that the work involved in maintaining those old PEPs will stop, so making that explicit would be helpful.

This ties into the documentation point, as a natural question will be “so where do we go now to get the information that used to be in those PEPs?”

Compatibility checks

The example manylinux_tag_is_compatible_with_this_system function doesn’t seem to match up with the reality of how pip (and the explanation in PEP 425) works, which is to generate a list of tags supported by the current platform. As PEP 600 defines an unbounded set of tags, this is problematic, and I’d like to see more details on how this would be implemented in practice. A link to a reference implementation for pip would be a reasonable option here (and very much in line with how other PEPs handle implementation questions).

My main concern here isn’t with how the implementation would work in detail, it’s simply about whether the proposal is practical to implement - as there’s no point in accepting a PEP that can’t actually be implemented. Note: This is probably my biggest technical concern with the PEP as it stands.

Tensorflow crashes

I’m going to leave this to others to debate, specifically for the reasons I gave above - personally, I consider having a spec that lets this sort of crash happen to be unacceptable, but I’m not in a position to judge (much less dictate) what is acceptable to the Linux community.

Unless someone comes up with specific objections, and a plan for resolving them, I’m going to assume that the PEP’s statement, which is effectively that wheels causing such crashes are non-compliant in principle, even though we don’t have a check that allows us to detect them or prevent them being installed, is sufficient.

I would like this to not end up being a decision based on “who shouts the loudest”, so I’ll be looking for how many people express an opinion, and what concrete suggestions are made, much more than simply repeated assertions that it’s a significant issue.

Other user comments

Finally, a couple of comments that were made by others in the previous thread.

I’m not convinced that PEP 600 needs to cover this. None of the previous manylinux PEPs discussed migration strategies, and while PEP 600 does have a wider scope in terms of covering “all future manylinux tags”, that doesn’t (to me) immediately imply that migration processes come into scope.

Migration is an important process to consider, but I don’t actually think it’s something that belongs in a PEP. But even if it is, a separate process PEP “migration between manylinux versions” seems more appropriate here.

There were a number of other comments about going beyond glibc-based Linux (musl, for example) and about tying in additional constraints like the C++ toolchain. I’m going to declare these out of scope. Yes, they do mean that PEP 600 won’t be the last ever PEP on Linux compatibility tags, and it is certainly possible that future PEPs may even make PEP 600 completely obsolete (if we discover that we cannot ignore the C++ toolchain, for example) but I don’t think we should block progress now by trying to anticipate every possibility that the future can bring.

Summary

Overall, I think we’re nearly there. There’s a few things to address, but not much that seems controversial (I hope!) I’d like to see a few more responses here, just to ensure we get as wide a consensus as we can (given the specialist nature of the PEP) but at some point we do have to accept that anyone who wants a say has had their chance.

Thanks, @njs, for persisting with this proposal - I know it’s been frustrating at times, but I think it’s coming together.