These AMAs seem like a reasonable idea, so here’s one I’ll open myself 
You can see my nomination statement here.
Happy to answer any questions folks have, or just chat about packaging stuff as well
7 Likes
We need to make it easier to try something out and iterate on it over time, rather than having to define the complete and final solution up front, otherwise we’re going to be stuck with having to be very conservative in what changes we allow.
Hi Donald. I think that this is a very important view. My personal observation is that the current PEP process when it comes to packaging very quickly focuses on implementation issues before establishing need, who benefits, are some groups harmed, and what standard(s) need to be adapted. What are some ideas that you have on the PEP process and how it could be improved related to producers of packages, distributors of packages, and end users consuming packages?
P.S. For everyone, Donald co-authored the PEP 8016 that established the Steering Council for Python. I figured that I would highlight this since it wasn’t in his nomination statement.
5 Likes
What do you see as the responsibilities that the PPC should take over from the PyPA? What role do you see the PyPA playing once the PPC has been elected?
I think PEPs tend to get bogged down in the implementation details for a few reasons:
- For a lot of PEPs, I think people generally agree on there being a problem to solve and the question ends up becoming what is the cost of solving that problem? And for a lot of these PEPs the most obvious cost is the cost to implement it.
- PEPs themselves are designed to be a proposal for a specific technical change/design, and thus the PEP focuses on the technical and implementation specific aspects rather than higher level or more ambiguous questions of “why” (there is the Motivation section, but it’s wed to the technical design).
- Sometimes people don’t agree on there being a problem to solve (or one that’s worth solving), but people don’t like to straigt forwardly say that and instead focus on implementation details because that’s a more neutral “thing” to focus on.
When it comes to things like trying to figure out the actual implementation cost of things, a lot of times I see the discussion ending up having to talk about theoretical problems because we’re doing a very waterfall-esque design process where we can’t ship things until we’ve agreed on the final design, but it can be hard to predict the cost of that final design without actually getting some real world uses out of it.
Sometimes there are natural or easy ways to start testing these things out. When I was looking at PEP 440 for instance, one thing I did was compare the results of switching to PEP 440 across every version of packages that existed on PyPI at that time.That helped prove that it was a relatively safe change. Something like PEP 691, if we had ended up realizing that was a bad choice, the design made it so we could stop serving PEP 691 responses and things would just work.
That becomes a lot harder when we start talking about things like, wheel variants, lock files, or wheel 2.0 or any other change where end users are producing durable artifacts (in the general sense) or changes to their workflow rather than things that are more behind the scenes.
We have the “provisional” status for a PEP, but that has not generally been very useful for packaging, in part because it was designed for CPython, but also in part because we don’t really have any guidelines in how that should be handled, and so in practice we’ve treated “provisional” to mean “we’re pretty sure this is the answer, but we just want to give ourselves a chance to make changes”.
What I think would be more useful [1] is some sort of an experimental status, where we’re explicitly communicating to people that they should not be depending on a thing, coupled with guidelines on exactly when and how to use it. I don’t have a specific answer in mind, but I imagine that would look something like requiring explicit opt-in on the end users (and figuring out a way to signal that to PyPI as well). The hardest question out of them all is trying to figure out how we handle experimental artifacts that have been uploaded to PyPI. Do we remove them? Do we allow them to be graduated to non-experimental? What if the experiment has to change? Do we allow them to be deleted and reuploaded? etc.
On the other side of this question, we currently don’t really have a process for deciding that something is a problem we want to solve, and what the requirements are for a solution to that problem. A PEP currently has to play double duty and try to do both, but the design of PEPs push them more towards the implementation side, and even when they do talk about why this problem is somethign that should be solved, it’s typically done in service to a specific solution (the one in the PEP), so often times the motivation and rationale part of justifying this problem are done with a specific solution in mind.
So something I think could be useful is an optional [2] sort of a “pre-PEP” where it’s basically just defining a problem to be solved, figuring out what the requirements are for a solution, and getting rough consensus on that. For large problems, this would allow us to first figure out whether something is a problem we want to solve, and get some rough agreement on a rubric that we can use to judge a possible solution to that problem on in advance.
This would also mean that the “solution” PEPs can be smaller and more focused, because they don’t have to re-litigate whether something is actually a problem worth solving nor what the requirements for a a solution actually are.
Hopefully that answers your question?
Well, I think it depends on what exactly you mean by “the PyPA”.
There’s “the PyPA” which is basically the ecosystem of tools built up around wheels/PyPI/etc that have generally agreed to use the PEP process for defining how they interact with each other, but there’s also “the PyPA” which is a specific org on GitHub with a mailing list and a voting process and such. In an ideal world those the overlap between those two concepts might be a circle, but in reality it’s a venn diagram .
There’s somewhat of a fundamental difference the SC and the PPC (and the PyPA before it), the SC directly “owns” CPython and so has ultimate authority over it but the PPC and the PyPA do not own any individual project, each project is an island with it’s own governance.
So to extent, the PyPA doesn’t really have much in the way of power or responsibilities, and to my memory the only thing the PyPA voters have ever really voted on was adding more projects to the PyPA and PEP 772.
The original PEP 609 did outline some other responsibilities for the PyPA like “guiding decisions”, “enforcing a CoC”, “providing support for existing projects”.
I don’t know that the PyPA as an entity has ever really “guided decisions”-- certainly most of the members of the PyPA have offered their opinions and insights into various decisions over the years, but I’ve never really personally considered that a “PyPA” thing and more a “these people are experts and care about this problem space” thing? Certainly I expect the members of the PPC to also do this, but I expect most or all of the nominees to do this regardless of whether they are elected to the PPC or not .
The PyPA has gotten a few CoC complaints over the years and have had to act on them, and I suspect handling CoC issues within PyPA specific projects will remain a PyPA concern, as the PPC doesn’t (at least not in PEP 772) gain ownership over those projects other than the PPUG.
Likewise for the “providing support for existing projects”, I suspect that will remain a PyPA thing, but I also think that’s largely an aspirational responsibility to begin with. If suddenly nobody wanted to work on PyPI (to use an example), the PyPA doesn’t really have any mechanism by which to force people to work on it. The closest thing that we have to compel people to maintain a project is by paying them, which the PyPA itself never really had much in the way of ability to do that-- the packaging-wg did/does which I believe will be taken over by the PPC, so there is some ability to do that, but it’s also not a particularly long term solution.
So for the PyPA itself, I suspect it’s role will be roughly what it has been, a light governance over some shared resources of a bunch of member projects, who have all agreed to use the PEP process (and now the PPC) for making decisions on how they interact with each other and the larger ecosystem, while still being independent projects.
The responsibilities and powers that the PPC have actually taken over, are really the power that @pf_moore and myself have had, through the PEP delegations over the years) on deciding on PEPs. Those delegations are ending and being taken over by the PPC.
There is a version of this where things don’t change much except for who is making decisions on those PEPs, but I think that would be a pretty poor outcome overall (still probably better than the status quo though! some of these PEPs are legitimately hard for a single person to have to shoulder).
What I hope to see from the PPC (and I suspect this is a common hope) is that they’ll end up filling a long existing gap in the status quo-- we can make decisions on individual PEPs currently, but we can’t really make decisions on things that aren’t “PEP shaped” or make any real statements about ecosystem wide goals or plans or priorities.
Some of that will flow somewhat naturally I think out of the PPC having a mandate to “own” the PPUG/packaging.python.org, some of that they’ll have to carve out themselves by working with the individual projects to try and come to some sort of agreement on how to structure these things over time.
I feel like I’ve gone on a bit of a tangent here, but hopefully that’s useful
4 Likes
I completely agree with this assessment, and it does impede forward progress and iteration that may be useful for understanding.
Thank you for your thoughtful and thorough answer. I really appreciate this response, you, and the perspective that you bring to all things packaging.
3 Likes