PEP 751: lock files (again)

[brettcannon]

@ncoghlan

It seems fine to me if lock files declaring defined environments are forced to say things like “dev server has the default & dev dependencies installed, CI has the default and testing dependencies, staging and production only have the default dependencies” rather than allowing the active groups to vary freely (installers would instead need to use the open package lock and resolve at installation time for free selection of groups).

Assuming that you’re tying in platform details for e.g., “dev server” and “production”, then that makes sense. At that point you’re just expanding what an “environment” is, and thus why you made your “resolved environments” terminology suggestion.

@dustin

Is it fair to say that “package locking” is just “file locking”, but inclusive of every possible environment, like wheel-tags = [*]? Would that be a better way to explain it to regular users?

That would be one way of doing it.

@dustin

If so, I think I agree with @frostming that the two can probably be unified.

That’s what I’m trying to propose with PEP 751: lock files (again) - #81 by brettcannon.

@dustin

I’m not sure what the solution would be: some equivalent of -r that allows an installer to combine multiple lock files? I’m hesitant to propose this.

I’m a little concerned of the impact on auditing as you now have to reference another file to get a complete view of what would be installed (and even assuming the installation would work).

@dustin

This PEP has no opinion as to the location of lock files (i.e. in the root or the subdirectory of a project).

I’m concerned that we’re losing the ability to reliably predict where a lockfile/requirements file is for a given source checkout.

I would not be opposed to a “SHOULD” line about where to put the files to help guide users towards common directory locations outside of the root of the project.

@dustin

I don’t see how file locking and specifically file-locks.wheel-tags would support installing anything that isn’t a wheel?

All file-locks.wheel-tags does is say, “you need the environment you’re installing into to have equivalent support of these tags”. It doesn’t require you use those tags in some way. But I assume it would be used to list the wheel tags used. But it could be the wheel tags used in your locked build requirements for your sdists, for instance.

@steve.dower

Would it be a helpful simplification for the package lock to be “the lock” and the file lock to be “the set of file (name/hashes) that are allowed to be installed for each package”?

Potentially. The trick is if you’re thinking of having a separate list of files/hashes to condense the details of the environment then you have to come up with a way to reference things like sdists w/ locked build requirements, VCS checkouts, etc. You can probably do it with an array of tables with critical keys specifying file, vcs, etc. and then the package version being referenced.

Or you say only wheel files are supported for file locking and if you want sdists you have to use package locking. (From a security standpoint that would the best approach, but considering what happened to me with PEP 665, I’m a little scared to propose that).

@DanCardin

i feel like solving for package locking concerns, and then also providing a way to predefine environments (like you did) solves a bunch of file-locking problems also.

That’s the miracle I’m hoping for.

@charliermarsh

I’d strongly prefer that the lockfile settle on a single format, rather than trying to support two with a single schema.

PEP 751: lock files (again) - #81 by brettcannon was trying to do that.

@charliermarsh

Since the latter has faded into the background in recent discussion

I actually have not gotten that impression.

@charliermarsh

the PEP was exclusively focused on File Locking, so it proposed a format that was optimized for that use-case

I’m starting to end up in that line of thinking if sticking with file locking is the way to go, then it either needs to be its own thing or stem from package locking in a nice way instead of the other way around (which is how these topics evolved).

@charliermarsh

Brett, can you walk through how the proposed idea solves for extras? Perhaps with an example? It’s not obvious to me from the description.

I can’t because I don’t have a solution yet. I have some thoughts, but they haven’t solidified enough to show yet, especially if the format might fluctuate more heavily into file locking or package locking specifically.

The tricky bit with the whole extras thing is when the resolve varies w/ or w/o a certain extra being present (and this is even considering it being different per platform). This instantly shows me the appeal of what you’ve suggested you’re doing with uv.lock and doing edges from package to package and there’s a consistent package version for all scenarios; not having to care about which package version to pull in is a very nice simplification. But Paul pointed out how he didn’t expect that behaviour and would want different package versions if called for if the extras influenced things that way. Toss in different versions for different platforms or varying requirements per file and it isn’t so simple to write down in a readable, compact form.

But I have an idea around package locking that takes what you seem to be asking for based on uv.lock where package locking is the overall format, and adds in stuff @steve.dower and @ncoghlan suggested that acts like extra bookkeeping on top of that instead of an entirely separate format in the same file (and thus would make @DanCardin happy ). But since I just came up with this idea while writing this reply I will have to mull it over on the weekend to see if it solidifies into something worth mentioning here.