Moving forward with the next manylinux specification

@pf_moore So it’s been a month without any motion on the PEP… any ideas on how to move forward?

I feel that this thread has got very muddled. It was initially about manylinux2014, then the perennial proposal got mixed in, and the tensorflow crash issues are also getting discussed here. I’ll re-read the whole lot over the next day or two, and come up with something more concrete, but I think we need the following, at a minimum.

  1. A new thread specifically to discuss the latest version of PEP 600. Basically reboot the perennial manylinux discussion on a dedicated thread.
  2. A more focused attempt to get confirmation that the points already raised on this thread (by @ncoghlan, @dstufft and @zwol) have been addressed to the satisfaction of the individuals who raised them. If we can’t get that confirmation (either because the individuals aren’t responding, or because it’s not possible to get agreement) I think we need the points to be explicitly noted and responded to in the PEP.
  3. Generally, more engagement from the community. At the moment it does feel rather like no-one other than yourself is interested in perennial manylinux. If you could clarify who needs to be engaged in order to deliver PEP 600, and actively solicit feedback from them, that would help give some assurance that we’re not going to end up with a standard that will just end up in limbo.

I don’t feel that PEP 600 is ready for pronouncement yet, to be honest. There just don’t seem to be enough people who understand and agree with what you’re proposing to make this a successful PEP. There have been a number of discussions, some relatively recent, where it still feels like people are talking past each other. I find it hard to believe we have a consensus if we don’t even fully understand what each other is saying :slightly_frowning_face:

To reiterate my role here, I’m looking to confirm that we have some level of consensus, and the proposal satisfies the community as a whole. I don’t have a strong technical view myself, but I do intend to summarise and restate concerns raised by others, if I feel we need a checkpoint to confirm where we are up to in the process of reaching consensus. But it’s not me you need to persuade or engage with, it’s the people who raised the concerns originally.

More to follow, once I’ve re-read the full thread.

1 Like

It is super muddled, yeah. It was initially about deciding between manylinux2014 or perennial, but 233 messages later it’s pretty impenetrable. And I saw @pradyunsg’s suggestion of using discourse’s moderation features to split off the C++ debugging into a new thread, and I think it’s a good idea and I have the permissions to do that… but I can’t actually figure out how to do it at this point.

Yeah, this is definitely a good idea. I’ll do that.

As far as I can tell, the PEP does explicitly address every point that’s been raised.

I’m sure there are things I could have done to explain my points more clearly. I’ve tried not to get frustrated I haven’t always succeeded, and I apologize to everyone for that. But if these are your criteria then I’m at a loss about how we can move this forward. Distributing portable binaries is a hugely complex topic and it involves a ton of super niche technical details and there’s a ton of misinformation floating around out there on the net. Even folks like you who are experts in packaging in general are mostly unfamiliar with the specific issues here. So it’s always going to be difficult to get a clear and unambiguous consensus. And that puts you in the really difficult position of having to somehow make a decision when you don’t have the time to really become an expert on the issues, and where most of the folks around you are also doubtful and uncertain.

But it feels like the end result is that we’re going to go around in circles forever, because the fact that we’ve been going around in circles is evidence that there’s lots of uncertainty, which makes people less confident, so it can’t possibly be accepted without more discussion, so we go around in circles some more…

The reason I’m pressing you here is because at feel like at this point we have a process issue, not a technical issue. The technical issues have been discussed to death, and led to some valuable refinements and a much clearer text. If someone wants to raise a new substantive issue or thinks I missed something in the PEP text then I’m still happy to discuss it. But at this point I literally have no idea what else there is to discuss; to me the PEP feels more completely baked than any of the accepted PEPs I’ve ever been involved with. And I think the status quo is hugely disrespectful of volunteer time, and I don’t think it’s OK for us to keep disrespecting volunteers just because the alternative got accidentally filibustered. (And also, I keep hearing complaints about how the pip upgrade problem is still blocking the manylinux2010 transition. But really it’s the part where we’re disrespecting people that gets me.)

1 Like


I see two main issues here with the process:

  1. People who expressed reservations on the previous version of the PEP are no longer commenting. This is bad because I have to decide whether to pre-emptively assume that silence means consent to the changes that have been made.
  2. IMO, discourse really sucks for long, complex threads like these. (Mailing lists aren’t much better, but personally I find them easier). I have no real sense of how many people have expressed opinions, who has reservations, how people’s opinions have changed over time, where the revised version of the PEP fits in the timeline of all this, etc. So I don’t have the fundamental information I need to make a decision, without doing a lot of digging (which RL issues mean I haven’t had any time to do for a month or so).

I will review this thread. And hopefully the new thread (thanks for starting it!) will generate some fresh comments.

Can you clarify here, please? We now have manylinux{1,2010,2014} agreed and either implemented or to be implemented. All of those are, for better or worse, done deals. There’s work to be done to implement them, and that is all being done by volunteers, but PEP 600 won’t change that.

I’ve already said that I don’t intend to accept another manylinux spec in the current style, so I’m not sure what part of the “current process” you’re concerned about here. And I’m honestly not clear where you see volunteer time being “disrespected”. I’m personally extremely conscious that the level of process we currently have is a burden - the only reason I accepted manylinux2014 is because we had a spec that was essentially ready to go, and promises of manpower who would work on delivering it (and helping with the remaining work on 2010). If that manpower hasn’t materialised, I’d like to know (even though I have no power to do anything about it, other than not accepting PEPs in future based on such promises :slightly_frowning_face:)

So I take the suggestion that we’re not respecting volunteer time here very seriously, but I need to understand what you think is wrong.

manylinux2014 seems to be on track IMO – I’ve seen PRs in pip, auditwheel and warehouse for it.

They aren’t done deals. Every manylinuxXXXX PEP requires an indefinite commitment of volunteer resources to keep it updated for as long as Linux distros keep making new releases. See PEP 600 for more discussion :wink:

I definitely appreciate that, but I hope you can understand why I’m going to stay nervous and feel like I have to keep pushing until it’s actually done… Especially since I can’t tell how to get there from here. You said you were accepting manylinux2014 b/c you thought perennial needed more work, but I said at the time that I thought perennial was ready and asked what more work needed to be done, and in the months since then no-one’s been able to answer that. I feel like the only reason it’s even still on the table at all is because I’m pathologically stubborn to the point of self-harm and keep nagging you to pay attention :-/. Not because you’re trying to kill it or anything, but just because inertia is the default state.

I bet if you threaten to accept it then folks will actually offer feedback :wink:

And it is kinda frustrating to watch all the one-off work going into manylinux2014 right now, since it’s all going to have to be repeated for the next PEP, when (it seems to me) we had a PEP ready to go that would have avoided that. All the promises of personpower that I’m aware of applied equally to manylinux2014 and perennial manylinux. The manylinux2014 proponents made clear pretty early on that they were really indifferent between the proposals as long as it let them start shipping CentOS 7 wheels on a reasonable time-frame. They just feared that we wouldn’t be able to get agreement on the better solution, and that became something of a self-fulfilling prophecy. (Edit: Not saying that’s the only reason that perennial stalled, just that it was a contributing factor.)

1 Like

You know, I also want to say explicitly: I do appreciate your promise to review the thread and trying to move things forward, and I know that takes a lot of work. I’m just trying to explain where I’m at and why I feel like I have to keep pressing like this.

1 Like

I may be missing something here. If PEP 600 is accepted, we won’t withdraw the existing manylinux PEPs, nor will we stop maintaining them (as I understand it). Or will we? Can you point me to where in PEP 600 it says that, because I missed it, and it seems pretty significant to me.

I guess what you’re referring to is the idea that manylinux{1,2010,2014} will simply become profiles under the perennial approach, and will be managed as such. But I’m still not clear how much effort that will save - the manylinux2014 stuff I’ve seen going on seems to be happening fine, there’s been a minor tweak to the PEP (around the arm spec, IIRC) that didn’t need any extensive work/debate to land. So I’m still to be convinced that PEP 600 will make a big impact on existing tags.

And yes, I accept that “you need to trust me, I’m the one who’s directly involved in the workload” is a fair response - I know next to nothing about what work is needed on the manylinux infrastructure. But if you want a concrete step to take to move the proposal forward, then explain it to me - and to everyone else, by clarifying the benefits in the PEP so that people can read them and say “yeah, that’s a lot of work being saved”.

Yes, it is. That’s true of any PEP, though - the author has to persuade the community that the proposal is better than the status quo. It’s always “status quo wins by default”.

In this case, I’ve made a commitment that status quo won’t win if that means another manylinuXXXX PEP without PEP 600 having been formally rejected. But equally community response to PEP 600 has been at best lukewarm, and some of the early feedback was honestly pretty negative.

It’s certainly an option :slightly_smiling_face: But I’m extremely reluctant to go down that sort of ultimatum-based approach, even if the current approach is failing.

My view is that the point where an ultimatum becomes real is when someone tries to lobby for a manylinux2019 (or whatever) PEP and I say “nope, you’ve had ages to agree PEP 600, get your act together and do it”). But that’s based on the principle that PEP 600 won’t alter the existing level of work involved in the current manylinux specs.

I know this is probably all obvious to you, but can you give some specifics of some “one-off work” that’s been done recently, that would not have needed to be done if PEP 600 had been accepted instead of manylinux2014? Note the important point here - “would not have needed to be done”, not “would have needed to be done this one last time”. Yes, maybe 2014 introduced an extra iteration of the old-style process, and maybe that means we’ve had to do stuff twice rather than once - but I see the promise of perennial manylinux to be “we do it once more, then never again”,not “we drop a load of unnecessary work that we never needed to do”. And 2014 went in because there was at the time a lot of pressure that it had to go in “right now” because of deadlines. And at the time (before the latest update!) PEP 600 still had a number of unanswered issues. So you can understand, I hope, why I’m averse to rushing a decision on PEP 600 when the only time pressure I can see is that you want to get it agreed “so we can get started on it”.

Yep, so now is the time to work on getting them on board. They should be completely aware that they won’t be able to get away with “we have an urgent deadline” as an argument again. So they need to engage with the PEP 600 discussion if they want a way forward for “next time”.

Actually, suppose I do the opposite? If I threaten to reject PEP 600 due to lack of community interest, what would you do then? :slightly_smiling_face: (And that is rhetorical, I’m no more intending to threaten you with an ultimatum than I am anyone else).

Here’s a suggestion. Get a list of the people you think should have informed feedback to offer on PEP 600 (include the various people who have commented on this thread). We’ll list them all on the new thread, and explicitly ask for comments from them, as well as any other interested parties. If comments are positive, then good. If people still have reservations, you can address them. If we get no feedback, then rather than outright rejecting it, I’d suggest we defer PEP 600 for a period (maybe 6 months?) with the intention to revisit the discussion then.

I don’t think I can in good conscience accept a PEP if no-one other than its author is willing to explicitly say they are in favour of it.

(OK, I’ll now stop responding to individual points and go re-read the PEP and the mega-thread).

Um, please can we not move posts around between threads? We now have three separate threads with parts of the PEP 600 discussion in them, and it’s nearly impossible to review :slightly_frowning_face:

We have a new thread for PEP 600, but if people post in the old thread then please let’s just leave their posts where they are. It’s easier to add a link to a comment in the wrong thread (or just copy the content wholesale) instead of trying to track things getting moved around :slightly_frowning_face:

@pf_moore Sorry about any confusion that the move may have caused. I’d requested to move everything after @njs requested a re-review yesterday, into a new thread. All the prior discussions are still over on the other thread (hopefully not affecting your re-review).

I’ve explained my reasoning for my request below, hopefully that makes it the situation less… “wait-what”?

Part of why that thread got so unweildy (a “mega-thread”), was because all-things-manylinux got discussed there. A linear discussion format doesn’t lend itself well to such discussions. :slightly_frowning_face:

Since we have a much clearer “breakpoint” – 1 month of no activity, a PEP accepted and being implemented – I figured the “next manylinux specification” thread can probably be called “done” and we’ll move our discussions to the other thread (PEP 600). However, there’s clearly a need to figure out how we’d go forward in terms of the process and the technical concerns… hence I requested to break off at that point into this new thread.

I’d imagined this topic could be the location for any process-related discussions around PEP 600 (hence splitting at the request-for-moving-forward). I imagine we’d continue the discussion, after you’ve reviewed the past discussion, in the dedicated thread for PEP 600 discussion that @njs made.

Hopefully that’ll make it clearer why the move happened, and sorry again for the confusion. :grimacing:

Not a problem, I understood your reasons, it just unfortunately clashed with my agreement with @njs that he start a “reboot” thread with the latest version of the PEP. I don’t like the fact that splitting a thread in Discourse needs an admin to do it, and I think that as a result there was a delay before the split happened making it even more confusing.

I suggest we abandon this thread here, and PEP 600 discussions happen on the “reboot” thread. Anyone wanting to continue discussing other manylinux related topics can continue on the original thread or start a new thread themselves (manually).

1 Like