PEP 751: lock files (again)

[brettcannon]

@ncoghlan

I’m curious whether the package lock formats in each of those ecosystems would correspond to package locks or file locks in PEP 751 terms

File locks. I haven’t really come across anyone who does package locking like we have.

@DanCardin

I know a couple of comments have been made about unifying file/package locks. So maybe dumb question, but can’t a locker just choose to produce a package-lock style file that’s described in a way that is a “file lock”?

Give me a minute.

@charliermarsh

I know I’m new to the PEP process and perhaps too naive, but I’ll just admit that it feels hard to commit to a format (and a standard) without more examples and real-world stress-testing.

The problem is unless I create a tool that people are willing to use that implements this PEP then I won’t get that “real-world stress testing”. And the other issue is the pre-existing lock files weren’t proposed as a PEP, along with their creators admitting there were design choices they would re-evaluate.

As for examples, the code at GitHub - brettcannon/mousebender at pep should work for per-file locking if you want to try out other examples. Admittedly, I had to write my own resolver to make that PoC work, so I can’t promise it to be bug-free.

@charliermarsh

I’m not at all suggesting that our format is perfect, but in building uv.lock, we’ve already iterated on it a ton based on filed issues and hard test-cases.

So are you hinting you want to propose it as a PEP once you have released it and gotten real-world experience with it?

@charliermarsh

The proposed schema is fairly different from anything that exists today

It’s based on PDM which is based on Poetry, so I don’t think it’s that radical. Really the biggest difference is the flattening of the package/file list instead of following a graph so reading diffs is more straight-forward. And it has been changed based on feedback from e.g., Randy, so while it might be different it isn’t without input from pre-existing implementers.

@charliermarsh

Where both bar1 and bar2 contain definitions for a package named bar, at the same version (let’s say 0.1.0). They’re not the same package, but they have the same name and version.

I was actually planning on updating the PEP to say entries need to differ by name, version, and marker requirement to be as flexible as possible about selecting the right scenario.

@charliermarsh

whereas the PEP allows directory, vcs, and files all at once.

(Relatedly: what use-case is that designed to support?)

The PEP says you have to choose one, but basically it’s there because previous feedback related to needing sdist support flowed into, “I need all possibilities supported as I don’t know what it will take for a user’s platform to get a working wheel”.

@charliermarsh

I know you’ll get critiques in the other direction

Change that to past tense and you’re right.

@charliermarsh

Second, I worry that attempting to support both File Locking and Package Locking in a single format adds a lot of complexity to the schema.

Give me a minute and I have an idea that may simplify things a bit.

@charliermarsh

it requires that we record the requested revision, not just the Git SHA.

I’m not sure what you mean by a “revision” compared to a “Git SHA”. What’s the difference?

And the PEP asks for the “commit ID” which “MUST be immutable”, which means Git SHA commit.

@charliermarsh

Can packages.multiple-entries be marked as optional?

Yes.

---

So, I’ve been thinking about how to simplify per-file and per-package locking and melding them together more as well as supporting multiple groups of dependencies, i.e. locking for all extra combinations for a pypproject.toml in a single file (although some are asking for more files; I’ll never win).

An idea – and I stress this is an idea and I won’t move forward with it unless people show appropriate interest – is the following:

• Drop [[file-locks]] and [[package-locks]] (and related other bits)
• Introduce [[supported-environments]] to represent what environments the lock file explicitly supports
  • It’s optional, so that if you leave it out it’s best-effort support for your environment
  • Finding the appropriate thing to install follows per-package guidance in the PEP
  • This is effectively assuming per-file locking is simple enough to not need to be as much of the focus of the PEP and is essentially a simplified scenario of per-package locking
• Drop dependencies and introduce [dependency-groups]
  • Much like PEP 735, table of named groups of dependencies
  • The default group is the empty string as it sorts to the top no matter what
  • project.dependencies plus project.optional-dependencies named with brackets, extra names are normalized, and extras as sorted, but otherwise list the complete list of dependency specifiers, effectively flattening the extras with the required runtime dependencies
  • If PEP 735 comes to fruition then they just slot right now based on the dependency group name
  • Can either follow Alyssa’s advice and normalize with markers by adding a group marker variable, or introduce a groups/part-of/applies-to/bikeshed-name-ater array in [[packages]] of dependency group names if the package version applies to only a select set of groups

And here’s the PEP example converted (due not you will need to scroll the text box to see the whole thing):

version = '1.0'
hash-algorithm = 'sha256'


[[supported-environments]]
name = 'CPython 3.12 on manylinux 2.17 x86-64'
marker = ""
wheel-tags = ['cp312-cp312-manylinux_2_17_x86_64', 'py3-none-any']

[[supported-environments]]
name = 'CPython 3.12 on Windows x64'
marker = ""
wheel-tags = ['cp312-cp312-win_amd64', 'py3-none-any']


[dependency-groups]
'' = ['cattrs', 'numpy']


[[packages]]
name = 'attrs'
version = '23.2.0'
multiple-entries = false
description = 'Classes Without Boilerplate'
requires-python = '>=3.7'
dependents = ['cattrs']
dependencies = []
direct = false
files = [
    {name = 'attrs-23.2.0-py3-none-any.whl', origin = 'https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl', hash = '99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1'}
]

[[packages]]
name = 'cattrs'
version = '23.2.3'
multiple-entries = false
description = 'Composable complex class support for attrs and dataclasses.'
requires-python = '>=3.8'
dependents = []
dependencies = ['attrs']
direct = false
files = [
    {name = 'cattrs-23.2.3-py3-none-any.whl', origin = 'https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl', hash = '0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108'}
]

[[packages]]
name = 'numpy'
version = '2.0.1'
multiple-entries = false
description = 'Fundamental package for array computing in Python'
requires-python = '>=3.9'
dependents = []
dependencies = []
direct = false
files = [
    {name = 'numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', origin = 'https://files.pythonhosted.org/packages/2c/f3/61eeef119beb37decb58e7cb29940f19a1464b8608f2cab8a8616aba75fd/numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a'},
    {name = 'numpy-2.0.1-cp312-cp312-win_amd64.whl', origin = 'https://files.pythonhosted.org/packages/b5/59/f6ad30785a6578ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl', hash = 'bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171'}
]

---

Tomorrow is BC Day so I won’t be reading any responses, and I have stuff going on Tuesday and Wednesday, so there’s a chance I might not reply for a few days.

I will wait to see if this idea gets more or less support than the PEP as it stands. At that point I will take stock and decide what next steps are to unblock progress towards a conclusion (potentially with a poll).