PEP 751: one last time

Packaging Standards
82

Regarding groups and extras, consider this scenario:

  1. The PEP is finalised without extras and groups support
  2. Installers implement support for this PEP
  3. Some time later, the spec is updated to support extras and groups on some way, bumping the lock file version to 1.1
  4. A lock file using the new spec is created, with a package having different versions based on an extra
  5. An end-user tried to install the new lock file with an installer which hasn’t been updated to support extras
  6. The install fails because multiple versions for this package were to be installed (after warning about unknown key and unsupported version)

Is there a way we can have the installer in this scenario support this new lock file? I don’t think so, as the installer would have to somehow know which version to select when the extra is disabled, which it can only do of it knows how extras are specified.

I think that means that extras and groups support necessarily means the lock file version is bumped to 2.x in this scenario. This is the case for any new functionality which affects which version are selected.

I disagree. Likely the version used during development will be the version constrained by all scenarios, and therefore will be the most tested version. This is the version I want deployed.

In addition, I don’t think multiple lock files are necessary when you have extras and groups support, as you can switch between scenarios with eg marker = '"min-dep-versions" in extras' and marker = '"min-dep-versions" not in extras', right?

2 Likes
83

… when requesting the dependency group, not necessarily when it’s left out (i.e. the default install scenario). The whole point of these various ways of specifying what extras or dependency groups are used is precisely to support differing versions.

dependency-groups = ["group1"]

[[packages]]
name = "requests"
version = "2.23.3"
marker = "'group1' not in dependency_groups"

[[packages]]
name = "requests"
version = "2.2.0"
marker = "'group1' in dependency_groups"

The whole design of this has been to prevent restricting oneself to a specific version overall just because an extra or dependency group was chosen since @pf_moore made it clear early on that he didn’t want that to happen in his use-cases.

Multiple lock files, not just locks in a single file (hence the support for multiple lock files).

I don’t believe that’s true with what’s being proposed and I have explicitly designed it so that scenario is not true (see my response above as to why that is).

It just muddles things that direct being true means that you don’t know what’s an sdist or wheel when directly installed if known ahead of time. But I’m also fine with your proposal since it simplifies things in this situation and I don’t think it comes up that often to be a concern.

Yes, very likely.

FYI I won’t be writing that PEP; this is “now or never” for me on this topic.

4 Likes
84

This seems to contradict what @sirosen found, and stated in his post:

Or am I misunderstanding somehow? Is this simply how current lockers choose to work? But if so, then do we have any examples of the sort of multi-scenario lock that would be expressed with the lockfile you showed? (I’m sorry, I find it very hard to express my concern here - as I noted in my previous post, we don’t really have good terminology for these concepts yet, so the potential for misunderstanding and confusion is very high).

I wish we had better “user stories” showing how multi-scenario lockfiles would be used in real life situations. Ideally generically, so the same story can be used to show how uv, Poetry and PDM would handle the situation, and where their approaches differ. @sirosen’s example is easy for me to understand, but I can’t imagine a real situation where I’d actually do anything like that…

When it comes to pronouncing on this PEP, I’m concerned that I don’t understand what multi-scenario lockers like uv, Poetry and PDM actually do, and just as importantly how they differ. The pronouncement shouldn’t be about making a judgement on whose locking model is the best, but when it’s unclear how well the proposal matches the various existing tools, it’s hard to ensure that we don’t “accidentally” favour one tool over another because of some subtlety that hasn’t been captured. One advantage (if you can call it that :slightly_smiling_face:) of not supporting extras and dependency groups is that we’re definitely not playing favourites - all the multi-scenario lockers are equally unsupported.

Understood. Honestly, I’d hope that such a PEP would be written jointly by the uv, Poetry and PDM maintainers.

2 Likes
85

Apologies, I’m a little time limited, so this will be minimally edited. More thoughts later.

I think I wasn’t totally clear about my meaning. Yes, the spec supports these resolutions being independent.

What I’m getting at is the current, non-standardized locking behavior. Groups and extras will impact poetry and uv resolution of package dependencies. As users we have some, but not complete, freedom to tune the resolution behavior. But the goal for these tools right now is to produce a small lock, as fast as they can, with all of the various “install scenarios” supported or supported to the best of their ability.

That’s a reasonable, even noble, goal! Lots of users want that. But not everyone.

We should expect lockers to carry their resolution behavior over to the new format.

Here’s a simple scenario: uv has a flag to ask for it to resolve minimum dependency versions, but it applies globally. So if I want to lock my package dependencies twice, with and without that flag, great, I can do it! Each time I get a new uv.lock and export it to a different pylock file.
Unless uv develops new options to let me combine multiple distinct uv.lock files into a single pylock, I’ll need two files.

I hope that helps. I’m not talking about what the spec will support but rather the fact that because each resolver can only be customized “so much”, we should expect that even in the best case, there will be a drive for multiple locks for some projects.

Personally, I find this makes me less favorable towards trying to do the hard work to include that all now. But other folks could reach different conclusions even if they accept my basic reasoning here as correct.

86

Yeah, just to confirm what @sirosen said in the subsequent post: it is possible to come up with a resolution (and even represent it in uv’s lockfile today) that would pick the latest requests when group1 is not activated, and 2.2.0 when it is activated. In uv today, though, we wouldn’t do that, since we generally try to choose as few versions as possible. (I agree with @brettcannon that the proposed format would also allow us to express this via marker = "'group1' not in dependency_groups"; it’d be up to us as lockers to support a locking “strategy” that gave you that output.)

4 Likes
87

I think this is the main difference in our point of view: I’ve always been happy with the idea that in practice, some mutually conflicting scenarios may still require multiple lock files. I already have scripts that export multiple PDM scenarios to different requirements files, so I’m fine if I still need to do that.

What I’m looking for in the PEP is in principle support for merging those different scenarios into a single file, which will allow development of that capability in the lockers to be use case driven, rather than tooling devs having to say “We can’t do that because the standardised lock file format doesn’t support it”.

3 Likes
88

This example did help with my understanding of what a proposed marker syntax could look like. I think I probably lost it deep in the conversation but thanks. This helps to confirm my mental understanding of what possible support for markers and groups could look like.

From a bazel rules_python point of view, this would work well for our needs as an “installer” (not really what we do, but we do download and spread wheels etc).

Are you looking for a formal vote of confidence from uv, PDM, poetry that they would support it when producing or exporting lock files?

89

The key thing is they have a single lock file that allows you to pull in any extras and dependency groups as desired even if they only support a single version for a package. My proposal supports that in the most flexible way possible without restricting more advanced usage such as varying versions if some tool in the future chose to work that way.

I don’t think I ever said they couldn’t or shouldn’t with what I’m proposing.

I think people are focusing too much on the flexibility of what I’m proposing, as if tools must fully utilize that flexibility instead of allowing the usage. If you look at it as a spectrum, what we have are:

  1. Single-use lock files (e.g. pip-tools)
  2. Multi-use lock files that can support extras and dependency groups, but with single versions (i.e. current tools like PDM)
  3. Multi-use lock files that can have fully independent versions based on what extras and dependency groups are (not) chosen (no one is apparently doing this)

The PEP as-is currently supports option 1. The concern @sirosen is bringing up is with option 3 not being in use anywhere. But what I’m not understanding is why do we have to toss out support for option 2 just because option 3 is a possibility with that’s proposed? There’s no mandate in the PEP on how lockers choose what to lock on purpose; people should be able to choose whatever locking strategy they want. What I’m proposing gets us options 2 and 3 all at once without restricting the use of either.

For instance, to take the example that @sirosen proposed where a single version gets locked, it would look like:

dependency-groups = ["group1"]

[[packages]]
name = "requests"
version = "2.2.0"

Notice that no marker is necessary since ‘requests’ would always be installed for that version. My proposal doesn’t prevent this. And what if ‘requests’ was only to be used in the dependency group?

dependency-groups = ["group1"]

[[packages]]
name = "requests"
version = "2.2.0"
marker = "'group1' in dependency_groups"

So my point is that I don’t understand why there’s any concern about supporting a more powerful, potential use-case while it also supports current use-cases in other tools (obviously sans uv and their workspace stuff)? I think that’s why Poetry and uv preferred the marker expression approach over the other ones; no one is being “painted into a corner” with it.

7 Likes
90

They all said they would support the PEP as-is for exporting already.

As for using the PEP as their native format, PDM said they would if there was support for extras and dependency groups (what we’re currently discussing), Poetry would look into it if such support existed, and uv said probably not due to the PEP not supporting their non-standard workspace concept.

5 Likes
91

I think you may be right. I’ve been thinking some more about this, and in particular about what matters to me in terms of making a decision on the PEP. I’ve come to the conclusion that what matters most of all here is that we standardise something that supports single-environment lock files. The lack of a better alternative for “pinned requirement files” is a significant issue for the ecosystem, that’s been around for years now, and needs fixing.

Beyond that, everything to do with multi-use lock files is incidental. Yes, it’s nice to have a standard that supports what workflow tools like PDM, Poetry and uv are doing, but it’s not important for the ecosystem[1]. Again, looking at this from a PEP delegate POV, what matters here is that we don’t limit the ability of tools to innovate. So a more flexible solution is an advantage here. Ideally something that will support whatever multi-environment solution tools converge on, but we can’t predict the future, so that’s not guaranteed.

But I’ll be absolutely blunt here. If the PEP provides a good solution for single-environment lockfiles, I’ll be looking to accept it. If the multi-environment support is a problem, I’m more likely to ask for that to be ripped out than I am to reject the proposal outright (sorry Brett, I know that would be more work for you - but to be fair I don’t think it’s something that will happen). At this point, I really don’t want the multi-environment issues to sabotage the “pinned requirements file replacement” capability.

The fact that I, personally, don’t particularly like the behaviour of uv[2] when faced with extras/dependency groups is irrelevant. I’m making a decision on what’s good for the ecosystem, not what suits me personally.

  1. Yet. At some point, when the innovation has died down and we have a good idea of what the “best practice” is, things will be different. ↩︎

  2. And Poetry and PDM? I’m not sure how much this is specific to uv. ↩︎

5 Likes
92

I think the flexibility is important. eg Even with tools deemed to be nearer to option 1 on the spectrum above, there is a lot of overhead and duplication of files that can be removed if the lockfile format has the support for dependency groups. It’s not the goal to remove files, but where it is possible to reduce cognitive load of multiple files to keep in sync and time savings from not having to perform repeated resolutions I think they should be pursued. Similar to what Paul has been saying, I think this is good for the ecosystem.

In light of the example of markers shown above, if the maintainers of PDM and poetry can come forward and reaffirm if they’d prefer the PEP go ahead with the markers and that you’d look to support it as an export format and your native format (time and capacity willing), now would be a good time. :blush:

It does sound like there’s always scope for implementers not to support it too.

1 Like
93

There’s scope for lockers to not support (or rather, use) it. Installers will need to support the full format, otherwise it’s not much of a standard :slightly_smiling_face:

But it’s not a huge burden from the installer side, as far as I can see.

1 Like
94

I think this addresses my concern, assuming Brett finds it acceptable. My worry was that the downside risk of adding in this support is that it could derail the entire effort.

I wanted to be clear that, based on what I know of the tools involved, there will be a number of multiple-lockfile use cases. So the upside – to be understood as offsetting that risk – is that the spec supports these use cases in a single file, not that the tools will (at least, in the proximate future).

Maybe, being on the sidelines, I’m overestimating the risk that going after these additional features does harm. Or maybe I’m underestimating how hard it would be to develop a v1.1 or v2 specification.

1 Like
95

To be clear, the PEP even now supports multiple enviornments (e.g. you can have a lock file that works on multiple OSs). What it doesn’t support is multi-use lock files compared to single-use lock files (which is what I expect you mean, but I don’t want to have a misunderstanding).

I would much rather you have me rip something out of the PEP to accept it at this point than to reject it outright.

So let’s think about those risks. There are risks from it not working as needed and not being used/useless.

In terms of the risk of it not working as needed, the marker expression approach is maximally flexible for the problem it’s trying to solve. @radoering also suggested that he thought it was better than what Poetry was currently doing by overloading extra for their approach to supporting extras. That also suggests the approach solves the goal of making extras work since Poetry uses a similar approach already for this specific need. And if extras work then I don’t see why dependency groups wouldn’t work.

Adding marker expression support could be a feature that doesn’t get used at all or marginally used. That means installers added support for something that doesn’t really get used. It also potentially means slightly more complicated teaching because what’s possible is bigger (single-use vs multi-use), but then again workflow tools already support multi-use lock files and exporting to a requirements.txt file, so users seems to be handling it fine.

Finally, there’s the risk that it simply isn’t enough on its own for multi-use lock files for most cases. @frostming has suggested PDM would switch to relying on the standard if such support landed for extras and dependency groups. So I think it depends on whether you also want Poetry or some other tool to rely on it at the same time as a reason to leave it out (I’m leaving uv off the list due to their workspaces feature).

It’s also a question as to whether a feature that we know is necessary for multi-use lock files should go in now while we are already thinking about it, or only as part of a complete set of features that meet more needs for more tools to fully rely on it. I will say that if we leave out this extras/dependency group support it will be the one part of the packaging standards involving installation that this PEP knowingly doesn’t support.

Let’s think about what it would take for a v2 (since changing the marker syntax and keeping it in packages.marker is not a v1.1 thing to me since you can’t ignore it without installing incorrectly, telling me it’s not a minor update):

  • Someone has the motivation to write a PEP (the number of times people have told me they are shocked at my patience with these lock file discussions has been a little worrisome :sweat_smile: )
  • That person figures out the feature set(s) they want added
  • They drive the conversation for the PEP (for PEP 751 alone, the post count is at 1,011 posts starting a year ago in Feb, and that’s ignoring side topics and anything related to PEP 665 or discussions even before that about lock files; toss those in and it gets us to 1,714 posts starting 6 years ago, interestingly also in Feb)
  • They get enough consensus to get the PEP accepted
  • All the tools need to get updated to support v2

Now, hopefully I’ve done all the hard work so the next person tackling this can get it done in less than 6 years and 1.5K posts. :wink: But I don’t know if what it has taken to get this far will have scared anyone off short of the tools themselves coming forward to do the PEP (then again, I’ve been the one to do this and I’m not directly affiliated with any of the workflow tools).

10 Likes
96

This is a (shortened) real-world example of how Poetry does this at the moment [1]:

[[package]]
name = "torch"
version = "2.3.1+cpu"
# ...
groups = ["main"]
markers = "extra == \"cpu\" and extra != \"cuda\""
# ...

[[package]]
name = "torch"
version = "2.3.1+cu118"
# ...
groups = ["main"]
markers = "extra == \"cuda\" and extra != \"cpu\""
#...

I totally agree. Poetry will surely not utilize the full flexibility from the start, but the flexibility is important to not hamper potential future development.

I can only repeat:

  • without extra and group support: just an export format
  • with extra and group support: might replace our custom lock file

Both would be fine for us. Both may take a while, depending on how we set our priorities and how much time we are able to spent to implement it.

Honestly, I do not believe any of the current Poetry maintainers would write a PEP[2] since maintaining Poetry is just a hobby we are doing in our free time and nobody has expressed the slightest hint of any ambition so far.

  1. This is Poetry’s custom lock file format but I think it is easy enough to understand. Replace extra == "x" with "x" in extras and extra != "x" with "x" not in extras and you have Brett’s marker proposal. ↩︎

  2. at most help if someone else is doing the main work ↩︎

4 Likes
97

Hopefully it isn’t too much of a burden since I just updated the PEP to require it for installers! :grin: PEP 751 – A file format to record Python dependencies for installation reproducibility | peps.python.org

The diff is at PEP 751: drop `packages.direct` and add `extras` and `dependency-grou… · python/peps@5d27124 · GitHub , but the highlights are:

  • Add the marker expression syntax proposal we have all been talking about :crossed_fingers:
  • Added some stuff to “How to Teach This” as @pf_moore requested
  • Dropped packages.direct, made packages.archive support sdists and wheels, and clarified that packages.archive, packages.directory, and packages.vcs imply a direct URL install (as suggested by @sbidoul )

I still plan to update my PoC for the PEP, but at this point I’m only planning to remove features as requested unless some horrible oversight is found that would cause the PEP to be rejected.

17 Likes
98

I updated my proof-of-concept so it matched the PEP. You can see the results in the PEP (I’m not pasting the example here as Discourse will put it in an embedded window and someone will not realize they need to scroll).

One thing to call out with the example is the [[packages.attestation-identities]] table for attrs which I thought was neat!

And with that, I think the PEP is done! :face_exhaling: I can give people a little bit of time to see of there are any errors or clarifications to be made, but otherwise I will plan to submit this for consideration no later than Monday, March 17th when I’m back from vacation (or sooner if there aren’t any changes that come up).

26 Likes
99

The Additions to marker expression syntax section looks good to me. Just some minor remarks:

Fourth, a tool MUST raise an error if an extra or dependency group is specified in a marker expression that does not exist in packages.extras and packages.dependency-groups, respectively.

These changes, along with packages.extras/ packages.dependency-groups

extras and dependency-groups are top-level fields and not below packages, aren’t they?

Tools MUST default to no extras or dependency groups being requested by the user if no extras or dependency groups are requested.

Currently, Poetry defaults to all dependency groups (and no extras). Further, it has the concept of optional groups, which are not installed by default. There are means to select optional groups and to unselect groups that are installed by default. For backwards compatibility, Poetry would probably not comply with this rule - at least not immediately.

How users tell a tool what extras and/or dependency groups they want installed is up to the tool.

Does this also cover defining groups that should be installed by default in the tool section of the pyproject.toml? That might be a way for us to comply with this rule in the future.

2 Likes
100

Small aside: I was worried about the marker addition adding a lot of complexity. Now that I see it, I think I was overthinking how complicated it would be. It’s very nice!

I found that section a little hard to understand. For an installer like poetry with some config, it seems like it would be fine for the tool to have a preference (e.g., the dev group being included by default), and for use of that installer to be interpreted as “requesting those extras and groups”.

Maybe the spec doesn’t even need to get into this element of installer behavior and can just leave the door open here?

2 Likes
101

Correct, extras and dependency-groups are top-level to the file; that’s a typo on my part. :sweat_smile:

I can drop that part if people want. Installer actions have come up (and down) enough about consistency between installers that I may have over-corrected in this instance by being overly prescriptive.

I’ll take it out.

2 Likes