PEP 751: lock files (again)

[brettcannon]

@ncoghlan

For per-file locking, the [[common-packages]] array would be limited to those entries where there is only one [[packages.files]] entry for the package (or no file entries with a packages.vcs entry instead), and that file or VCS entry would list every defined file lock in its lock array.

I realized that PEP 735 and dependency groups may water down the usefulness of the separation. For instance, think of a dependency group to build your docs and another for your dependencies and testing needs. Those could be entirely disjoint, leading to no common packages.

@brettcannon

@charliermarsh

@brettcannon

What are you specifically after instead of packages.directory?

We have a distinction in the uv lockfile between those packages that should be installed as editable and those that should not. The schema is the same between those two kinds of entries, but the installer installs them as editable or not based on that key. I’m anticipating a response here along the lines of, “That shouldn’t be part of the lock, it should be an input from the user at install-time”?

If others feel the distinction is important I can add it (I assume it’s just a boolean).

I realized last night that if uv wants this but other people don’t speak up for it then it could be flagged in [packages.tool] so that uv as an installer can handle editable installs.

@DanCardin

@pf_moore

Personally I’d hate a tool where locking a project could potentially give a different result than installing it, but that’s not important as long as the standard allows tools that I do like to exist

Again, you’re already getting this with the different environmental markers. I dont see how this is any different. The point is that the resultant lockfile is more consistent and will give you the same versions across environment marker constaints and extras/groups

I think this could be viewed as a locker choice if we don’t want to codify it now. There’s nothing saying either approach is right or wrong, and so we could leave it open as an implementation detail for the lockers based on what the user asks for (if the PEP ends up supporting this scenario).

@ncoghlan

I explored how far adding a new install_filter marker to the set of environment markers defined in Dependency specifiers - Python Packaging User Guide would get us in handling the “markers vary by installation filter” scenario. As with the current definition of extra, install_filter would be defined by context, and treated as an unknown marker in any other situation (such as when defining package metadata rather than processing a lock file).

One thing I’m not sure about is whether a new marker is better than a some array like filters or dependency-sets or something that lists what group of dependencies the package version belongs to. What’s the specific win of updating marker support instead of using a key in the TOML? The marker approach has a nice consistency w/ how package versions are proposed to be filtered, but it does require updating the marker spec and tooling to handle that new key (i.e. packaging). Not the worst problem, but I think because of this I think readability and comprehension upon seeing a lock file for the first time should be the deciding factor if we go with something like this.

@ncoghlan

even it the PEP doesn’t standardise the translation of extras and dependency groups to installation filter names, the example below still works since the installers would be treating the filter names as opaque strings either way - any chosen prefixing conventions would be purely for human consumption, as even a combined locking+installation tool should be using the tool tables to communicate information between the two operations, rather than relying solely on specific filter naming conventions)

I have a concern here about how the user knows what names there are w/o introspecting the lock file ahead of time to know what they want? From a CLI perspective where tab completion doesn’t work, knowing how to tell the installer what you’re after w/o knowing the naming scheme of the locker might be a bit of a UX hindrance.

@ncoghlan

Not quite. “extra” refers specifically to extra names defined in the metadata of a published package that can be referenced in dependency declarations, while “install_filter” would refer to optional filters defined in a lock file, which may not correspond to published extra names. Initially, they’d most likely be the superset of published extras and unpublished dependency groups.

Right, which is why I was thinking your proposal was a generalization of letting you add a label to the marker expression to filter on, much like extra does. So violent agreement.

@ncoghlan

while I would expect to see extra-* and group-* as the two main sources of filter names, I don’t see a compelling reason to restrict them to covering just extras and dependency groups (instead prescribing only their behaviour, not their semantics).

You can also view extras as a subset of dependency groups; all the way down.

@ncoghlan

The benefit I see to file locks is being able to express details that environment markers don’t cover, and enforce a stricter “in the face of ambiguity refuse the temptation to guess” behaviour on installers

Playing Devil’s Advocate, you could still get that if you had restrictive markers on the package versions and minimized what files were listed (i.e. one or two). Since the PEP doesn’t say you can’t list a package and version twice but with varying markers, you could potentially get the same strictness.

---

What I’m hearing in the feedback is people want to minimize the scenarios that lead to multiple files if possible. That is manifesting itself as people wanting extras handled in the same file, and the same for PEP 735 if it gets accepted.

There also seems to be at least some push-back on separating per-file locking and per-package locking so explicitly, instead of potentially generalizing it and simply making sure lockers have the ability/expectation of only locking the files for a platform. I’m seeing this in both explicit asks and confusion on how to actually do per-file locking.

I haven’t made any decisions about my response to either of these things, but do know I’m thinking about them.