It’s not hyperbole; it’s me struggling to understand what this PEP even means.
I understand PEPs that are proposals for what Python or PyPI should do: it’s code we collectively control, so if accepted, then we will do those things.
I understand interoperability PEPs, like most packaging PEPs: they describe the language different tools will use to communicate, so everyone can understand each other properly.
And I understand the pip devs making a decision about the UI for their tool: they control the code, they can change it however they decide.
This PEP isn’t any of those. It’s just making a pronouncement about what programs people are allowed to write: thou shalt not suffer a program with certain UI decisions to live. That’s pretty weird! Can we make a PEP that says that going forward, all GUI programs MUST default to dark mode? I mean, obviously, yes, in the sense that we have the physical capability of posting document saying that on the website peps.python.org. But would that mean anything?
That’s why I’m trying to figure out how, concretely, the world changes if this PEP is Accepted or Rejected/Withdrawn.
And yeah genuinely my best guess is that it being Accepted means that the general public will feel encouraged to harass anyone who creates Forbidden Programs, and if it’s Rejected/Withdrawn then it won’t, and… that’s it. Am I missing anything? I don’t know what pip specifically should do here; if the pip folks decide they want to make this change, then I’ll believe them that it’s a good change for pip users. But even if this PEP is supporting a change that’s good, I can’t really support a process that’s ultimately based on harassment as a way to get there.
You’re being a bit hyperbolic But I agree entirely.
As far as I can tell,[1] the intent is that acceptance would mean that PEP 582 gets rejected, and rejection implies that PEP 582 gets accepted. “This is intended to be an alternative to PEP 582.”
As such, it basically follows the same pattern. The main point of PEP 582 is to choose the name, and the main point here is to choose the name. However, this one doesn’t require any changes in CPython (unlike 582, which needs CPython to change its path calculation code), and so it’s solely about defining the interface between pip and… pip’s users, I guess? Which as Nathaniel points out (and Donald mentioned too), doesn’t require a PEP.
So it feels like it’s solely motivated to get PEP 582 rejected by using process. Which is not an uncommon thing for us to do, and it’s legitimate approach, but as 582 is clearly standards track and this is pretty obviously not (despite the metadata claiming so), I doubt it’ll work.
And I promise I’m being honest here, and not taking offense as the original author of 582. ↩︎
It’s not a wholly unprecedented thing, there are things like PEP 394 which exist primarily to serve as Python Core telling other people how they should implement something.
However, PEP 394 is pretty clear that these are just recommendations, and gives decently length rationale for the recommendations it gives. It may feel kind of bikeshedy to talk about recommendations vs requirements, but I think it largely drives to the point of what a number of people (myself included) have said.
With PEP 704, there’s nothing backing the requirement, there’s no actual requirement here. We’re just expressing what we’d like people to do, and by framing it as a requirement it gives more ammunition to people who will harass other people for the crime of not adhering to this PEP.
Maybe some people will do that even with it being couched more in terms of a recommendation, but at least a recommendation carries with it some sense or understanding that you are allowed to do something different.
Two other things about PEP 394 that I think are relevant:
The core motivation was to try to improve interoperability, ie let people write scripts that worked on multiple distros. The Unix cli is an unconventional API, but it’s still an API. OTOH PEP 704 is purely about human interfaces.
The whole thing was triggered by Arch pointing python at python 3, and people wanting to tell Arch to stop that. And… Arch’s response was “we have reasons and we stand by them”, and kept python as python 3. The PEP was still useful for helping other distros coordinate their migration plans, but it couldn’t magically make the disagreement go away. That’s hoping for too much from a PEP.
I’m trying to figure out how, concretely, the world changes if this PEP is Accepted or Rejected/Withdrawn.
IMO, it would set a common direction in Python packaging.
Without a PEP like this, it’s much too easy to invent a thirteenth alternative way of doing things. With the PEP, ignoring the recommendation should be a conscious, reasoned choice – much easier to work with if the goal is to standardize things & seek out all the various requirements and use cases.
Regarding PEP 394: Arch was an outlier/pioneer, and that was OK. Eventually they showed the way for everyone else. But in the mean time, the other distros didn’t need to design everything themselves and still behaved predictably.
That’s not the goal for me, at least. I view this as an alternative approach to solving the UX problem, wherein we lean further into the venv-based model for the ecosystem rather than adding an alternative place for putting installed packages.
I’ve explicitly stated that I see arguments both ways for PEP 582 and PEP 704 when I first mentioned that I’ve written it, and I also don’t think it’s as simple as we’d do one or the other (status quo is certainly an option).
Being accused of harrasing (or strong-arming through process) was not something I expected TBH, and y’all are reading this situation in ways that I do not understand.
I’m happy to withdraw the PEP 704, if people think using PEPs for discussing design choices that fundamentally affect the language’s packaging experience and serves as a direct alternative to another Packaging PEP is not something that is worthy of a Packaging PEP.
I really don’t know what is PEP-worthy at that point, and would like each of you to explicitly state both:
why you think this isn’t a PEP-worthy topic and
what format you would’ve prefered this proposal to take.
I was imputing motive, rather than stating procedural fact (and I probably shouldn’t have been doing that either, except two of the OP’s posts led with “this is an alternative to PEP 582” which implies a one-or-the-other situation was intended). Less obvious now this has all been pulled out of context, though.
I didn’t say it wasn’t a PEP-worthy topic, just that the proposal contained within this particular PEP doesn’t really seem to impact anyone but pip and pip’s users, and so doesn’t need to be any kind of standard (perhaps specifying “Standards track” in the metadata was a copy-paste error?).
But that seems to be on topic for the original thread, so I’ll leave it there. I don’t have anything to add on this particular question.
You’ve explicitly agreed that this isn’t something that requires a PEP:
I apologize for the candidness, but you’ve straight up accused me of trying to use process with an ulterior motive of achieving an outcome. I was trying to do neither, and now, I’m trying to understand what I could’ve done differently and where the communication gap is here.
What form would you have prefered this proposal then, given that you don’t think this is a Standards Track Packaging PEP?
I think claiming this is trying to cause people to harass tools is a bit strong unless you’re claiming any PEP is an invitation to harass anyone for not following it. If that’s the case then I’m the 4th most prolific harassment-causer via this mechanism the last time I counted.
Pradyun is trying to see if there’s consensus around providing a guideline to installer tools to require an environment by default. If this PEP gets accepted, we are saying the PyPA at least agrees with those guidelines as being good/best practices.
Now if you’re suggesting this should be an informational PEP instead of Standards track, fair enough and I can see the logic behind that. But once again, I think the term “harassment” is a bit strong for this context.
Don’t confuse topic with the specific content of the PEP at the time that I reacted to it
What I read was not an ecosystem-impacting specification, but a recommendation to display a recommended error message. That does not need a PEP, but solving the overall problem in a way other than documentation (contextual or otherwise) likely does.
I know you’re competent enough to know what an “alternative PEP” means - you’ve been in plenty of discussions with other ones. I even said it’s a legitimate approach, so I’m hardly suggesting you’re trying to sneak it past the SC (which is what “ulterior motive” would imply, as opposed to merely a “motive” ).
But because I don’t think this proposal needs a PEP, that means you could just go ahead and do it. And perhaps I was wrong, but I believed you recognised this as well, especially since all the pip maintainers have argued in the past that pip-things should be decided by pip and not by PEP (they were even doing it for this PEP).
So perhaps the breakdown happened because I assumed that you could also see the limited impact. Which may mean that you envisage greater impact, but didn’t describe it well enough (which tbh is what has hurt PEP 582 as well, though I did describe it to some people who then asked me to leave pip-things out of the PEP… so… )
Informational? We don’t have many other categories, but this is definitely just suggesting a UX for a certain subset of tools, which is in no way a standard.
I don’t think that’s the claim at all. Nathaniel can speak for himself (if he wants), but it’s not that the PEP is trying to cause harrassment, but that (in Nathaniel’s judgement, having seen how UX decrees have been implemented in the past) it is likely that nobody will choose to follow it until they are harrassed about it. I’m inclined to agree, though I think “harrassed” is a strong word for “user request”, but I’m very sympathetic to maintainers who do get harrassed over things like this.
I generally think PEPs should not dictate UX/UI, we’ve done it in the past and it’s pretty much always ended terribly. The recommendations on UX in the PEP only really made sense for one tool, or a specific subset of installers. That’s why my initial reaction was “this doesn’t feel like a PEP”.
Ultimately, I think our packaging PEPs are primarily used for two purposes:
To define standards for interoperability
To decide large changes on PyPI
The tone and focus of PEP 704 came across to me like it didn’t fit into either of those categories, it felt akin to writing a PEP saying that build-systems are required to support the src/ layout or to compute versions from VCS history or something.
Upon further reflection, I do think there is a part of it that does fit into the first of those, namely which is providing a standard location to put virtual environments so that various tools can all look to the same place for it. That part of the PEP feels like an implementation detail of the current PEP, which focuses on the UX of pip, which is why I didn’t really think much of it.
Oh whoa, I didn’t mean it like that at all. I know you’re not trying to harass anyone! Sorry that I phrased it badly.
It’s just… realistically, we know how people are. Just this week we had a widely-shared blog declaring “PyPA must be destroyed” because PEP 582 hasn’t been accepted. And even with the best of intentions, a statement saying that some software is Officially Naughty™ would encourage those sorts of responses, even if the software has good reasons for what they’re doing and all their users are happy about it. It’s a foreseeable consequence that I think we should try to avoid.
(This is also IMO one of the major flaws in semver: it only accomplishes its goals if everyone in the ecosystem gets on board, so the rational strategy is to apply social pressure via shame/harassment/etc. to projects that don’t. It’s not what the designers were intending, they just wanted better communication between upstreams and downstreams, but it’s how the social dynamics work out.)
I think PEPs are great for
Coordinating a design discussion within on of the “core Python” projects that we all share (Python, PyPI). These projects have to make decisions; PEPs help them do it.
Interoperability PEPs that define a common interface for different projects to use. These projects want to be able to produce and consume data from each other; PEPs help them do it. (Also packaging interoperability PEPs often involve PyPI changes.)
The point being that a PEP is fundamentally just a document; it doesn’t have any inherent moral weight. In the two cases above, the PEP is useful because having a shared document solves a pre-existing problem for everyone involved. But OTOH if a team or project concludes that the best way to serve their users is to violate a PEP, then they will – and they should! So as a corollary, we shouldn’t write PEPs that try to make pronouncements about how third-party projects should interact with their users; it won’t be effective, and frankly, it’s none of our business.
Also, I’m just trying to elaborate on what I was saying above, in response to the original draft :-). The latest draft is much better, both by softening things to SHOULD, and by proposing .venv as the venv to use, which is an interoperability question. I’m still not sure that making it a PEP is really better than just doing it in pip though. Realistically pip is the only installer most of us here have experience with, so it’s real hard to guess whether our intuitions would apply to new projects we aren’t imagining yet.
I guess what I’m hitting then is that large changes to PyPI end up being PEPs, but large changes to pip can’t be PEPs?
The PEP was significantly improved in the reviews from PEP editors and based on the feedback that things weren’t clear, and there’s still some work there - that’s a thing I’ve learnt now, that waiting for the PEP to be on peps.python.org before starting discussion on it is the way to go.
Yet we implement things on the package index side with little regard for how someone like Artifactory, Nexus, or whatnot would implement+expose that functionality in their mirrors?
I hear you that we haven’t done this before and perhaps it isn’t as obvious a connection as I thought it was… Nonetheless, I do think that major changes in a tool like pip/setuptools would benefit from going through a PEP-style discussion.
Traditionally, the view was that PyPI is special in a way that pip wasn’t. Now, the roles are actually reversed somewhat, in that PyPI changes are for all practical purposes interoperability PEPs now, as we expect alternatives like Artifactory and devpi to implement them.
Is pip special enough that large changes should be PEPs? Maybe. I dislike the idea of making pip any more special than it is, but I accept that maybe we shouldn’t fight the reality. But conversely, if we do decide that pip changes can be handled by PEPs, then:
We need to frame those PEPs on the understanding that we are constraining all hypothetical alternatives to pip that may arise (see the PyPI experience).
We need to accept that people can propose PEPs that will act as “force of law” on pip - we can’t reasonably go down this route and then decide not to implement an accepted PEP. As an example, what if someone proposes “Pip should expose a supported API to install packages” as a PEP, and it gets accepted (because we know a lot of people want it) in spite of the pip maintainers’ reservations, with a precise API defined? To be clear, I’m not trying to argue here that this is necessarily bad, just to make sure people have thought through what we’re agreeing to here.
There will be a class of issues that are currently raised on pip’s issue tracker that will need to be transferred to the PEP process - including, probably, a bunch of existing issues. Obvious examples are “pip should have a way to override dependencies” and “pip should handle indexes in order”. These issues are complex enough already - do we really want to subject them to “design by committee”? On the plus side, we’ll be able to shut down ideas that we don’t like by saying “please raise this as a PEP”
As an example, the recent pip installation report feature is, to me, very obviously something that, if pip was subject to the PEP process the way PyPI is, should have been raised as a PEP and agreed before implementation. I’m pretty sure that if that happened, it would still be in progress, and likely everyone would be too burned out to complete it.
Be careful what you wish for
(To be clear, I do know that you’re not arguing that we should go to this level. But it’s a natural progression from something simple like this PEP to something like the above).
I believe that’d be a governance crisis, at that point, either through pip’s maintainers being grumpy and not doing $thing or just… stepping away.
I’m optimistic that we’re not all out to get each other and won’t have the discussions/decision making procedures break down to the point that this happens.
I don’t think so, I think we’ll need to phrase it as design choices we’re making in pip given the current constraints. If someone comes in and designs a replacement to pip, they’re welcome to make different design choices.
… No? We don’t need a PEP for PyPI exposing vulnerability data in its non-PEP JSON API, or for how it handles Project-URLs or whether it validates certain types of data. So, likewise, I don’t think every detail about pip needs to go through the PEP process; only the bits that affect how it interoperates with other tools or significantly affect existing workflows.
For example, we wouldn’t have a PEP for implementing draft releases in PyPI, but we would for the bits that affect the upload API. In case of PEP 704, it’s about how pip interacts with core Python’s venv, Conda, end user’s workflows and more. It is genuinely a proposal for enhancing Python’s default packaging tool.
I also don’t think “design by committee” is mandatory for all packaging changes. It makes sense for our 8-author PEPs. We’ve done design by committee for things like PEP 621 but that doesn’t mean that we are locked into that for eternity or for everything. A proposal for a meaningful change that affects a lot of people can still be driven by an or two author championing the idea.
Hmm, I think you’re imagining something more extreme than I was. I refer you to PEP 658 - exposing metadata in the simple API and PEP 700 - additional fields for the simple API, or Metadata 2.2, which admittenly wasn’t a PyPI PEP, for examples of “things where a PEP was approved but Warehouse hasn’t implemented it”. I think it’s bad that we have PEPs that are indefinitely blocked like this, and I think it’s just as likely that this could happen with pip-focused PEPs (less so if they are raised by pip maintainers, see below).
That’s not how PyPI PEPs have traditionally worked. Artifactory & co are expected to implement the simple API, not “make different design choices”. Or are you thinking of other PyPI PEPs that I’m forgetting?
The “committee” I had in mind wasn’t the authors, it was the wider community arguing about details and bikeshedding.
Also, I think there’s a fundamental difference between PEPs raised by pip maintainers, to improve visibility of changes that we want to get a community view on, and where we’re ultimately trying to validate our own design choices, and PEPs raised by others, who maybe don’t have any intention of doing the implementation themselves, because they think something “should happen” and think that getting agreement via a PEP is somehow a more compelling approach than raising a pip tracker issue. I don’t have a great problem with the former, but I don’t see us being able to limit the process, so the latter will always be a possibility.
But I don’t want to make it sound like I think the world will end, or that this is some sort of disaster. I just want to make sure we’ve thought about the implications. And if we still think it’s OK, then fine.
PS I don’t think PEP 704 is an example of any of the “worst case scenarios” I’m presenting here. It’s very much a case of the “make it a PEP for visibility” situation that I think is fine in isolation, it’s only the implications of the process-related precedents it might set that make it controversial.
Hmm, I missed commenting on this. The installation report is designed to be about interoperating with other tools. Sure, we don’t have to make it a PEP if we decide to just implement it, but that just begs the question, what is the criterion? Do the pip maintainers decide what needs to be a pip PEP[1]? Can someone external propose one? Can someone external say “I want this to be a PEP” about a feature getting added to pip?
Again, just food for thought, not an argument that you’re wrong as such.
Another argument against having “pip PEPs” - it sounds silly saying it aloud ↩︎
But I agree entirely.
)
