I’m a bit late to the party, but I just wanted to say thanks to Brett for addressing all the late-in-process comments and nits.
7 Likes
In my opinion “MAY” makes more sense, especially considering that even in your expectation “dev” is in both groups.
1 Like
I don’t really care that much, although IMO “may” is little better than saying nothing at all, as it gives no real guidance on what the standard prefers.
However, I’ve now finished reviewing the discussion threads 
so I’ll be giving a decision in the next day or so. So change it now or it will be too late[1].
although it could still be done after the fact as a clarifying spec change, once the spec is on packaging.python.org ↩︎
4 Likes
One comment on the PEP, from the perspective of an installer maintainer, not as PEP delegate. This is not something that I think needs changing, just an implication that I would like to make explicit.
The installation process refers to evaluating the markers associated with each package. They do not, however, specify anything about what extras and/or dependency groups the evaluation should assume to be present. I’m fine with this, and I think it’s clear that it’s up to the installer how to determine that, and what UI to offer to allow the user to specify.
One place where this is particularly relevant is in the case where the user doesn’t specify anything, and/or the installer doesn’t offer a UI (the canonical pip install --lockfile pylock.toml case). In that situation, it seems to me that there are two obvious possibilities:
- Evaluate on the basis that no extras or dependency groups are set.
- Set just the dependency groups given in the
default-groupskey, and no extras.
IMO, (2) is the sensible one for an installer to set (and it’s clearly what the locker intended to be the default), but given that the PEP doesn’t make an explicit statement, (1) is a possible implementation choice[1].
I repeat - I’m not asking for this to be changed. If I did, the PEP delegate would quite reasonably be annoyed at me for leaving it this late to bring this up 
And I don’t think it needs to be changed, in all honesty. There are plenty of places where the PEP provides a mechanism that tools are expected to use responsibly - the most obvious example is that lockers can create a lockfile that omits dependencies, resulting in a broken environment. I just wanted to point it out, because the default-groups mechanism was added very recently, and people haven’t had much time to think about the implications.
and it’s potentially what someone who hasn’t read the spec in detail might end up doing ↩︎
4 Likes
I’ll make it “MAY” then.
I was assuming (2), but you’re right it does leave the door open for (1). I can update the PEP to clarify that installers need to decide what they may want to populate extras and dependency_groups with if the user performs no UX action to show a selection/preference.
FYI I will make the above clarifications today or tomorrow my time (Pacific time zone).
2 Likes
What I meant was that I prefer the existing wording (i.e., SHOULD NOT). Sorry for being unclear.
That’s fine, if you want to. As I say, I’m happy with it as it stands, so don’t bother just on my account.
Ah, then I will leave it alone.
Now worries as it’s a small update that I think makes sense.
I opened PEP 751: address some minor user feedback by brettcannon · Pull Request #4330 · python/peps · GitHub , but I’m going to let you decide if it’s an improvement or not, @pf_moore ; feel free to either merge it or simply close it.
2 Likes
I’m pleased to announce that PEP 751 is accepted. Congratulations to @brettcannon, and to everyone who has participated in something like 4 years worth of discussion and design. It’s been a long road, but I’m confident that the result will be a big step forward for the packaging ecosystem.
I’ll note that this is full, final acceptance, not provisional[1]. I’ll make a separate post going into more details, but I don’t think there’s any advantage in provisional acceptance here - it will only introduce further uncertainties and delay before people can confidently start implementing and using the new standard.
This standard wouldn’t have been possible without Brett’s hard work over many years. The community owes him an immense debt of gratitude for all the work he has put in on this proposal and its predecessors. But a standard like this is never the work of one person, and everyone who has participated in the discussions along the way should be proud of what we’ve achieved here - it’s a fantastic example of what community involvement can deliver.
We have now laid the groundwork, but there will still be a lot of work for tool maintainers to implement lockfile support throughout the ecosystem. The design is very flexible, and leaves a lot of room for tools to experiment and innovate, both in terms of the UI they will provide, and the capabilities they will use to deliver functionality - in particular for lockers, having this standard is merely the first step in a process of integrating it into their existing (custom) solutions.
One area in particular where the standard is known to be limited is in its support for monorepo-style development workflow. The “workspace” model of uv is an example of this. I think monorepo support is somewhere the ecosystem as a whole needs to be able to innovate and experiment on, and lockfile support is merely one aspect of that. In particular, I don’t think that we’re ready for standardisation in this area, and while that gives us some rough edges for the moment, an update to the lockfile standard at a later date when we have more experience with monorepo support will be a better outcome than trying to solve the problem in the abstract right now.
Thanks again Brett, and enjoy all the extra free time that you’ll have now
in spite of statements I’ve made previously in these discussions ↩︎
75 Likes
As promised, here are my thoughts on provisional status.
In the end, I decided not to provisionally accept PEP 751, but to go straight to full acceptance. I want to explain why I made that decision.
First of all, I don’t think provisional acceptance has helped much in previous packaging PEPs. Typically, it has been handled on the basis of delaying final acceptance until the standard is implemented in one or two key tools. But in practice, this has simply meant that the ecosystem as a whole has been unable to start working with the new standard until that happens, and the provisional period just acts as a delay and a period of uncertainty over the status of the new standard. By skipping the “provisional” phase, we won’t avoid the need for tools to implement the standard, but its status is clear, and people don’t have to worry that things might change.
One of the main arguments for provisional status is to iron out any difficulties that might arise as part of implementation. That’s a reasonable concern, but I don’t think it’s as significant as people imagine. Smaller issues can easily be handled using the standards “clarification” process, so it would take a fairly major problem to need anything more. But our standards are versioned, so even without provisional status, fixing a major issue should be nothing more than creating a version 1.1 (or in the worst case, version 2) of the standard - and assuming this happens within the sort of timescale that provisional status would cover, a version bump should be fairly painless. While we do have some problems introducing new versions of long-established standards[1], I don’t think those concerns apply to new standards.
Another disadvantage of provisional status is that it leaves the PEP author “on the hook” for any fixes, for an unclear period of time. I don’t think that’s a good thing - once accepted, a PEP is the property of the community, and tool authors who find issues while implementing a specification should be able to propose fixes to the spec themselves, without needing to work through the original PEP author.
Overall, therefore, I think we should be a bit more confident in our PEP process, and the community involvement that guides the design of our standards. PEP 751 is a great example of that, and we should acknowledge it.
wheel spec, I’m looking at you
↩︎
30 Likes
Thank you in particular for calling this out. It’s a very important part of our community culture, and the biggest problems I’ve seen over the years have largely occurred when this isn’t recognised.
Big thanks to Brett for getting lock files into shape. Now it’s all of our problems, and Brett can have a break! 
16 Likes
Thanks so much for handling the review, @pf_moore ! As you mentioned, this has been in the works for a long time and thus there wasn’t exactly a small amount of history to consider when making a decision.
And also as Paul said, a big thank you for everyone who participated in the discussion constructively! I call them out in the PEP, but a special thanks to the tool representatives for providing valuable feedback: @charliermarsh , @radoering , and @frostming .
Then is now not a good time to mention I have my next PEP already written, another one outlined, and yet one more planned if someone doesn’t get to it like they said they would? (I am serious, but their scopes are all small/tight and I won’t bring the first one up until May at the earliest.)
20 Likes
I opened Support PEPE 751 in `markers` · Issue #885 · pypa/packaging · GitHub to track updating ‘packaging’ for markers. I will also get a PR for packaging.python.org when I can (plan is this week or next depending on work stuff).
8 Likes
There really is no stopping you, is there?
5 Likes
These 3 are it from a packaging perspective. If this discussion had not gone for as long as it had it would seem like a “normal” Brett cadence on PEPs as I would have just done them as the ideas struck me.
6 Likes
Add PEP 751 by brettcannon · Pull Request #1848 · pypa/packaging.python.org · GitHub is the PR to add the spec to packaging.python.org .
8 Likes