Lock files, again (but this time w/ sdists!)

[brettcannon]

@groodt

+1 for still being interested in the unambiguous environment locks if you do decide on a format for specifying constraints in the same PEP.

I won’t propose something that doesn’t support a lock format which you can get a flat list of files to install once you know the set of files applies to your environment.

@radoering

if we want to avoid re-resolving at install time (no plans so far) we have to make at least two (maybe more) changes/additions:

• add the resulting marker condition to each locked package / version

I believe this is what PDM does for their lock files.

@ncoghlan

I think having a fallback partial lock that specifies either a minimum or exact Python version would end up being pretty common if the option is available, even for projects that specify one or more comprehensive locks for CI and production deployments.

To me, I think “partial lock file” to mean “at worst you have to evaluate markers (and maybe wheel tags) to decide what to install”, and thus a resolver is still not necessary.

@pf_moore

@ncoghlan

a partial lock which will probably work on other POSIX-based platforms.

I’m not quite sure what you mean by “probably work” here. The install will either work or fail, there’s no “probably” involved (installers should never create a broken environment). If by “probably”, you mean “the install might fail with an error” then that’s fine, but it’s not what I thought you were after.

The failure case is how I’m choosing to interpret it.

@groodt

Should there be consideration for pyproject.lock?

This was brought up as part of PEP 665 and decided against (remember, I’ve already done this whole discussion once before ). Basically not using .toml as the file extension causes the usual tooling headaches of things not knowing how to interpret the results. I also don’t know if PDM or Poetry expect their .lock files to be manually inspected and thus care about the file extension empowering that use case.

@charliermarsh

I’m still unsure how lockers are supposed to generate the markers and tags for a given lock entry

I’ll talk about that below, but it depends on how much effort you want to put into making a lock be as broadly applicable while still being accurate.

@charliermarsh

nor how installers are supposed to choose the “right” entry

I’ll also talk about this below, but it depends on whether we would allow multiple locks to ever apply (i.e. require they be mutually exclusive).

@frostming

That also means we need to explicitly document how to decide the degree of speciality of a particular tag and marker.

If we allow overlap then I think you’re right.

@pf_moore

I like the idea of not needing a resolver at install time. It makes installing from a lockfile easier to reason about, easier to audit, and just fundamentally simpler. But it does imply that the locker needs to be a lot more intelligent about working out “what platforms does this solve apply to” if we’re to avoid overspecified targets. And as far as I know, no-one’s even looked at that problem yet.

I’ve been thinking about it and I’ll talk about it below.

@cemici

The reason that the current proposal doesn’t record the dependencies themselves is surely that it has absolutely no need of them.

Correct. Listing dependencies per distribution is just to help infer why some distribution was included.

@BrenBarn

there don’t seem to be any existing tools that try to do this kind of lock.

I would argue pip-tools does, but within the confines of requirement files and the info they are given by pip’s resolver.

@BrenBarn

whether or not a resolving installer is needed for lockfiles, it will still be needed in general to do “regular” (i.e., not locked) installs. So it seems like not requiring a resolver for lockfile installation means everyone will still be using a resolving installer, but they will have the additional option of keeping an extra, weaker (i.e., non-resolving) installer around just to install lockfiles. That doesn’t seem like a real huge benefit to me.

Just because no one has put in the effort doesn’t mean people wouldn’t use something. I very much expect my work to use such an installer (otherwise I wouldn’t be doing this). I also suspect some people see a file w/ a .lock extension and assume e.g. Poetry has a stricter lock than it does.

@ncoghlan

A resolving installer would behave differently (this is the Poetry/PDM use case):

• identify all defined target environments that match the current environment

I don’t think we would need to go that far, but I’ll go into more detail below.

@EpicWink

While not technically non-trivial, I think it would be pretty straightforward to compute the most restrictive tags and the union of all markers for all dependency specifiers of all files in a lock entry. This would allow all environments to those specific constraints to install from that lock entry. I think Brett alluded to this before.

I agree that I don’t think propagating e.g. markers through the resulting dependency graph to know what is important in a lock is insurmountable (the PoC I did for Charlie already does this).

@EpicWink

When I’m auditing, I don’t want to open the file to scroll up 2000+ lines to see if a file is under [[common]] or [[conditional]]. Perhaps instead, you have a [[files]] array and put target-names in each file, and either leave targets unspecified or set it to a sentinel (eg "*") for the formerly common files.

I already had that in my head in that way.

@steve.dower

I wonder how much of this current topic is about being able to reverse engineer the original input to the resolver? I don’t see any problem with a lock file representing “the specific files implied by these versions at this time for this platform,” in which case none of the markers or dependency information is really needed.

At least my proposal isn’t. Originally I just recorded all of the inputs. I suspect if I come up w/ an alternative it would still capture what’s pertinent.

---

Lock file situations

@pf_moore

Here’s a thought - if we had a corpus of “typical lock specifications”, that might help us understand better whether the relatively complex cases we’re worrying about here actually happen in practice.

Over the weekend I thought about this a bit and the sort of scenarios one could find themselves in when trying to create a lock file.

Flat list of files

This is the “boring” case where there are no environment markers for a dependency, everything has a wheel, and you only care about one OS, CPU, and Python version. That’s just a list of files where you need to make sure the wheel files apply to your environment, maybe requires-python all-up if you happen to only have pure Python wheels.

Marker on a dependency

An easy way to look at this example is Rich because it has a conditional dependency on typing-extensions; python_version<3.9.

The way Poetry handles this is it captures the requirements of Rich directly so they can resolve the requirement.

Poetry's lock on Rich

[[package]]
name = "rich"
version = "13.7.1"
description = "Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal"
optional = false
python-versions = ">=3.7.0"
files = [
    {file = "rich-13.7.1-py3-none-any.whl", hash = "sha256:4edbae314f59eb482f54e9e30bf00d33350aaa94f4bfcd4e9e3110e64d0d7222"},
    {file = "rich-13.7.1.tar.gz", hash = "sha256:9be308cb1fe2f1f57d67ce99e95af38a1e2bc71ad9813b0e247cf7ffbcc3a432"},
]

[package.dependencies]
markdown-it-py = ">=2.2.0"
pygments = ">=2.13.0,<3.0.0"
typing-extensions = {version = ">=4.0.0,<5.0", markers = "python_version < \"3.9\""}

[package.extras]
jupyter = ["ipywidgets (>=7.5.1,<9)"]

PDM, though, records the marker requirement via both the requirement from Rich and typing-extensions itself.

PDM for Rich and typing-extensions

[[package]]
name = "rich"
version = "13.7.1"
requires_python = ">=3.7.0"
summary = "Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal"
groups = ["default"]
dependencies = [
    "markdown-it-py>=2.2.0",
    "pygments<3.0.0,>=2.13.0",
    "typing-extensions<5.0,>=4.0.0; python_version < \"3.9\"",
]
files = [
    {file = "rich-13.7.1-py3-none-any.whl", hash = "sha256:4edbae314f59eb482f54e9e30bf00d33350aaa94f4bfcd4e9e3110e64d0d7222"},
    {file = "rich-13.7.1.tar.gz", hash = "sha256:9be308cb1fe2f1f57d67ce99e95af38a1e2bc71ad9813b0e247cf7ffbcc3a432"},
]

[[package]]
name = "typing-extensions"
version = "4.10.0"
requires_python = ">=3.8"
summary = "Backported and Experimental Type Hints for Python 3.8+"
groups = ["default"]
marker = "python_version < \"3.9\""
files = [
    {file = "typing_extensions-4.10.0-py3-none-any.whl", hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"},
    {file = "typing_extensions-4.10.0.tar.gz", hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"},
]

In this situation there’s an optional distribution to install, but it’s a yes/no thing. So if you locked this to requires-python as >=3.8 (due to the minimum Python versions of some dependencies), you could get a “partial” lock where you scan all the files and decide whether to include or distro based on the environment marker(s) that influenced the inclusion of that distro (i.e. if a distro was conditional then its dependencies should be conditional as well unless some unconditional requirement pulled it in). And since markers support and and or you should be able to compose a marker that accurately captured the environment requirements which led to a distro being needed. And if you propagate those markers through the dependency graph I think you can still end up w/ a flat list of files to consider.

Multiple options work

This comes up w/ any distribution which has binary wheels and a pure Python fallback. An example of this is debugpy. If this was your sole dependency, what wheel do you choose (I think this is @charliermarsh 's question about what is an installer to do in the face of ambiguity?). Installers currently have a hierarchy to wheel tags for a platform, so this single case is easy to resolve: choose the “best” wheel.

But what happens when you scale this up to multiple distributions which might not cleanly all have binary wheels for your platform? My PoC I did for @charliermarsh calculated an overall weight/preference average for all of the wheel files and chose the one w/ the “best” weight. Unfortunately there isn’t a clean way via markers to say what e.g. glibc version you are on and so this pure Python lock shouldn’t apply it you can match this wheel tag, although we could introduce such wheel tag block list to prevent ambiguity.

Conflicts across environments.

Think:

packaging>=23.0; os_name=="nt"
packaging<=21.0; os_name=="posix"

This is what Poetry can handle and requires a resolver unless the possibilities are small enough it isn’t a bit deal to just generate all lock file possibilities eagerly (i.e. the concern over a combinatorial explosion of environment locks). PDM didn’t handle this and just locked to packaging 21.0 on my machine even when I asked for a cross-platform lock file.

The range of specificity

Here is the range of environment locking specificity from most to least specific:

1. If you match the wheel tags, Python version requirements, and environment markers, just install the list of files.
2. If you match the wheel tags, Python version requirements, and environment markers, go through the list of files and evaluate the markers per-file/distribution to decide whether to install it (the Rich case); this is still a linear install (i.e. no resolver)
3. If you match the wheel tags, Python version requirements, and environment markers, go through the list of distributions and choose the best file to install (the debugpy case)
4. Do a full resolve based on what’s in the file

Now in scenarios 1-3 all depend on whether lockers can generate appropriate checks and how specific we are okay w/ them being. This could be extra specific (e.g. <marker> == <value> for all environment markers) to putting some effort to try and minimize the requirements (e.g. only list environment markers used in the resolution). @radoering has a gut feeling that just environment markers

I think it really comes down to how much flexibility do we want to allow for so that lockers even have a chance to generate a lock that is as broad as possible while still being accurate? Going back to the Rich example, how would you expect it to handle the change in requirements that happens at Python 3.9 and newer?

1. Have two environment lock specifications/headers that differ by Python version, marking the similar files as applying to both environment locks
2. Have one environment lock specification overall and have the files/distributions be evaluated to decide whether they apply for Python<=3.8 case

I’m not sure which opens more possibilities, which helps w/ auditing more, or which is demonstrably easier to implement.

[alicederyn (Alice)]

@brettcannon

If you match the wheel tags, Python version requirements, and environment markers, go through the list of files and evaluate the markers per-file/distribution to decide whether to install it

My concern with this is, what is there are two lock files whose top level tags, requirements and environment markers match your machine, and which only differ in the markers on the files themselves? You could easily end up with a different installation on the same machine just based on which order you evaluated the options in. (This can apply to any solution, I think, if the tool generates overlapping constraints, but this option seems at increased risk compared with the first.)

Possible solutions that I can think of:

• Require the tool put negated markers into one of the files to make it unambiguous when to use it
• Define in the standard an ordering in which locks will be evaluated

[tmk (Thomas Kehrenberg)]

Sorry, I can’t really see the difference between the PDM and poetry example. Poetry will also put typing_extensions into the lock file:

Full poetry example

[[package]]
name = "rich"
version = "13.7.1"
description = "Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal"
optional = false
python-versions = ">=3.7.0"
files = [
    {file = "rich-13.7.1-py3-none-any.whl", hash = "sha256:4edbae314f59eb482f54e9e30bf00d33350aaa94f4bfcd4e9e3110e64d0d7222"},
    {file = "rich-13.7.1.tar.gz", hash = "sha256:9be308cb1fe2f1f57d67ce99e95af38a1e2bc71ad9813b0e247cf7ffbcc3a432"},
]

[package.dependencies]
markdown-it-py = ">=2.2.0"
pygments = ">=2.13.0,<3.0.0"
typing-extensions = {version = ">=4.0.0,<5.0", markers = "python_version < \"3.9\""}

[package.extras]
jupyter = ["ipywidgets (>=7.5.1,<9)"]

[[package]]
name = "typing-extensions"
version = "4.10.0"
description = "Backported and Experimental Type Hints for Python 3.8+"
optional = false
python-versions = ">=3.8"
files = [
    {file = "typing_extensions-4.10.0-py3-none-any.whl", hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"},
    {file = "typing_extensions-4.10.0.tar.gz", hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"},
]

Am I missing something?

[groodt (Greg Roodt)]

Brett didn’t suggest that Poetry did not collect typing-extensions. His comment was explaining how PDM propagates the markers through the graph so that the marker requirement is expressed not only by the rich entry, but also by the typing-extensions entry.

[brettcannon]

@alicederyn

My concern with this is, what is there are two lock files whose top level tags, requirements and environment markers match your machine, and which only differ in the markers on the files themselves? You could easily end up with a different installation on the same machine just based on which order you evaluated the options in.

This is already what happens w/ installers when there are pure Python wheels and binary wheels, so this isn’t new to this situation in terms of potential ambiguity. Otherwise I think you’re imagining the markers at a different level than I am (you’re saying files and I’m thinking versions of distributions like PDM and Poetry).

[alicederyn (Alice)]

Ok so let’s say version 1.1 of foo-bar supported manywheels2010 and version 1.2 supported manywheels2014. Your installer creates two sets of locks as a result (because you want to support RHEL7 still). The lock sets differ significantly due to a different requirement in foo-bar version 1.2. If you’re on a recent Linux distribution, either set could be installed. It could vary based on anything: user installer, maybe a randomised hash algorithm from the filenames being put into a set.

This is more volatile than, say, Poetry creating a similar lockfile, because there at least you know it will try to use the most recent version, using a consistent algorithm.

[groodt (Greg Roodt)]

@alicederyn

This is more volatile than, say, Poetry creating a similar lockfile, because there at least you know it will try to use the most recent version, using a consistent algorithm.

I think that the installation tool will still be deterministic. i.e. the same lockfile run on the same machine by the same tool should always result in the same distributions being installed. So in that sense repeated runs of $newInstaller tool will behave the same as repeated runs of PDM and Poetry and pip when targetting the same environment. When run against different targets however, the 3 tools will vary in their installation approach.

I think you are asking the same question as Charlie?

@charliermarsh

nor how installers are supposed to choose the “right” entry

An installer tool can have different mechanisms for selection of which distributions to install if multiple locks are compatible. I think this is the “specificity” concept mentioned by Brett.

Installation strategies

Explicit target

The most specific would be where the user running the installer provides some sort of name / alias / identifier for the target environment and requests to install it:
installer --target-environment=manylinux_2_28_x86_64 --lock-file=./pylock.yaml --destination=./venv
and the tool could bail-out if that target environment isn’t compatible with the tags and markers it’s being run under.

I think this is the only option that can guarantee a fully enumerated list of dependencies because the installer makes no decisions and evaluates no markers.

Implicit target

It’s also possible that the installer tool can provide a mode of operation where it “sniffs” or guesses the most appropriate lock in the face of ambiguity. Brett has mentioned a number of different challenges in how best to guess. I think that as long as the tool follows a deterministic algorithm, it should result in a valid environment that matches the requirements.

We aren’t standardising dependency resolution algorithms as far as I’m aware. It’s already true that pip, poetry, pdm, uv will resolve and install certain dependencies differently. I think we want tools to be able to make their own decisions and innovate on “best” or provide user choice. Brett had this in his “weighted best” selection in mousebender. uv let’s the user provide a resolution strategy and I think poetry or PDM may also enable similar choice. I think the same could be said for environment selection.

I think this is a good compromise because for scenarios where the target environment is known ahead of time, the best mechanism to perform an installation would be to target it explicitly so that you know exactly what you are installating.

[groodt (Greg Roodt)]

@pf_moore

PEPs defining what PyPI does are one thing, but PEPs defining something that all indexes must do are a very different matter. I’m not saying we couldn’t, but it would involve getting buy-in from commercial vendors like Artifactory, the Azure artifacts people, etc. That’s going to be hard to do.

Yes, but we did all seem to agree that the biggest users of the boundary/constraints/partial lockfiles are the open-source communities where they are using PyPI for dependencies in the majority of cases.

@steve.dower

It would also involve coming up with a way to make it work consistently with a directory containing .whl files. Hash-based and even filename based approaches handle this quite naturally.

Yes, but the wheelhouse scenario is largely used inside organizations and the private sector. It’s not that common for open-source projects to be shipping a wheelhouse committed to a repo or to a filesystem somewhere. It’s still possible to collect and store all the hashes to warn / fail if non-determinism is detected.

So perhaps there are “reproducible / deterministic” package sources and “non-reproducible / non-deterministic” package sources.

reproducible / deterministic package sources

This could be PyPI if releases were not open-ended and installers used a mechanism to filter based on timestamps.

uv has a “reproducible resolution” setting: GitHub - astral-sh/uv: An extremely fast Python package installer and resolver, written in Rust. that appears very similar to my idea (or maybe identical).

It’s still possible to warn / fail if non-determinism or tampering is detected here by collecting hashes.

non-reproducible / non-deterministic package sources

This would largely be indexes and packages sources that are “open ended” for mutation and don’t really have a mechanism to filter on time. So this would be wheelhouses on filesystems, or indexes that are open-ended or vcs packages etc. I think here it’s still possible to warn / fail if non-determinism is detected by collecting hashes.

This is largely the distinction between “reproducible / deterministic resolution” and “reproducible deterministic installation”

reproducible / deterministic resolution

This is the partial locks / constraints / boundary style set of candidates produced from a set of input dependencies and their version specifiers. Approximates a breadth first search to collect all valid candidates, driven largely by exploring all the transitive dependencies of the inputs. Note: Here the solution is not yet guided or heavily-pruned for a singular target environment, it is driven by exploring the metadata of the dependencies as it explores and builds out a graph.

A deterministic resolution algorithm, run against a deterministic package source (my hypothetical PyPI with timestamp pruning and no open-ended releases) would not need the graph serialized or scraped to verify a reproducible resolution. It could collect hashes to verify integrity that nothing had changed and be assured that no new possible solutions would appear on subsequent runs.

The result is not a single installation plan, but a graph of possible valid installations. A tool producing this could warn / fail if it encounters non-reproducible package sources. It can also create a serialized snapshot of the graph that essentially converts all packages sources into a reproducible / deterministic package source.

reproducible / deterministic installation

This is where this proposal started. This is where an installation plan has been produced for a specific
target environment. This is a precise installation plan for a specified target environment, serialized for later installation.

These installation plans can be produced directly from package sources. This would be very fast because they can prune and optimize towards the target environment(s) as their goal. More like a depth-first search of the solution space.

They can also be produced from the output of an earlier resolution step that captured the full graph of possible installation plans and prunes it down into a specific target environment.

[steve.dower]

@groodt

Yes, but the wheelhouse scenario is largely used inside organizations and the private sector. It’s not that common for open-source projects to be shipping a wheelhouse committed to a repo or to a filesystem somewhere.

Prevalence of a workflow is a really weak argument. If you can argue that using a wheelhouse isn’t a good idea, then go for it, and maybe we’ll decide to exclude it from this scenario.

“It’s not used in my community” is a very dismissive way to participate in a discussion that impacts multiple stakeholders. Perhaps considering approaching it as “what are the merits of this approach, and why aren’t we using it? Should my community adopt this?” You’ll find people more willing to listen to and engage with you, and may even learn something, rather than merely speaking.

[mikeshardmind (Michael H)]

I think the wheelhouses should not be excluded, but that we don’t actually need to consider them directly to determine an issue.

@groodt

A deterministic resolution algorithm, run against a deterministic package source (my hypothetical PyPI with timestamp pruning and no open-ended releases) would not need the graph serialized or scraped to verify a reproducible resolution. It could collect hashes to verify integrity that nothing had changed and be assured that no new possible solutions would appear on subsequent runs.

This approach causes the resolver to necessarily trust the remote timestamps provided and trust that any index is closed-ended on releases (while we can guarantee that for pypi outside of bugs or compromise of pypi, that’s a security concern still when discussing lockfiles) to ensure determinism, providing a full set of allowed artifacts and expected hashes of them does not.

[pf_moore (Paul Moore)]

@steve.dower

Prevalence of a workflow is a really weak argument. If you can argue that using a wheelhouse isn’t a good idea, then go for it, and maybe we’ll decide to exclude it from this scenario.

Wheelhouses have significant issues for deterministic installation, a file path is not guaranteed to be meaningful, or to point to the same resource, in all situations where a lockfile might want to be used (file URLs have the same issue, of course). I think it’s reasonable to restrict lockfiles to artefact references that are expected to be available without needing a specific context.

Having said that, I’ve no direct experience of situations where locking using wheelhouses are a real consideration, so I’m happy to be persuaded there’s a value to including it in the design. But just like prevalence of a workflow is a weak argument, existence of a workflow is just as weak. The argument needs to be based on the value of using wheelhouses, not just the fact that (some) people do.

[steve.dower]

@pf_moore

Wheelhouses have significant issues for deterministic installation, a file path is not guaranteed to be meaningful, or to point to the same resource, in all situations where a lockfile might want to be used

I don’t think they impact determinism of the installation, though they do dramatically increase the likelihood that the user gets an installation failure when the file they need isn’t there. That’s entirely on the user though - we don’t have to protect them from themselves.

My experience of wheelhouses has been that they make installations deterministic, since it’s currently so difficult to do that when referencing a remote index. The additional use cases are for air-gap installs (i.e. download the required packages first and then install them on another machine) and caching (essentially the same thing).

The relevance to the design here is the proposal to restrict the maximum timestamp to some arbitrary time, and the fact that timestamps for local files may have changed even though the contents of the file has not. (You need to trace back up through a few replies to get the context, but the up arrow button makes that fairly easy.)

Ignoring the hypothetical/suggestion, the relevance is still that an installer should be able to match a requirement to a specific file when all it has is the file. We can’t rely on that file coming from a web server, and we can’t rely on the metadata that typically accompanies files from typical web indexes. If it can’t match the file and the user has asked it to only use a set of local files, it will just have to fail.

[pf_moore (Paul Moore)]

@steve.dower

I don’t think they impact determinism of the installation, though they do dramatically increase the likelihood that the user gets an installation failure when the file they need isn’t there.

I was thinking in terms of C:\Wheels\something.whl meaning something different when on a different machine. But hashes mean that’s “just” a failure rather than non-determinism. To an extent, though, hashes mean that non-determinism is entirely restricted to cases where a resolver is involved (which is something that still isn’t confirmed as in scope of this proposal).

@steve.dower

Ignoring the hypothetical/suggestion, the relevance is still that an installer should be able to match a requirement to a specific file when all it has is the file

I’m mostly looking at the discussions about resolving installers and lockfiles as restricted indexes as speculative design explorations at the moment, and I don’t have a firm opinion in that area.

[brettcannon]

@groodt

uv has a “reproducible resolution” setting: GitHub - astral-sh/uv: An extremely fast Python package installer and resolver, written in Rust. that appears very similar to my idea (or maybe identical).

This was one of the motivations for the creation timestamp, but people very much didn’t like having such a value in the file.

---

OK, here’s where my mind is at.

Other lock files

I want to outline what Poetry and PDM have in their lock files to point out the differences and similarities (I’m skipping pip-tools since the list is short: name, file hashes, and dependents). That way we can perhaps agree on something that either meets multiple scenarios upfront or at least is flexible enough to grow in the future.

Formats

Everything in bold is unique to a format.

Poetry

/cc @radoering to check my work.

• Metadata
  • Version
  • Python version support
  • Content hash
• Package
  • Name
  • Version
  • Description
  • Optional
  • Python version support
  • Files
    • File name
    • hash
  • Dependencies
  • Extras

PDM

/cc @frostming to check my work.

• Metadata
  • Groups
  • Strategy
  • Version
  • Content hash
• Package
  • Name
  • Version
  • Python version support
  • Description
  • Groups
  • Dependencies
  • Marker
  • Files
    • File name
    • hash

Commonality

• Metadata
  • Version
  • Content hash
• Package
  • Name
  • Version
  • Python version support
  • Description
  • Dependencies
  • Files
    • File name
    • hash

Questions/Observations

People didn’t like having a hash for the file, but both file formats do.

There is only a single hash value per file. Indexes can actually provide an arbitrary set of hashes. Are people okay w/ just one hash value, or would they rather capture all possible hash values (i.e. do lockers get to dictate the used hash, or do installers choose from the options presented to them)?

Poetry gathers the maximal Python version support in a single place in its metadata.

PDM specifies the marker expression which must be true for a package to be installed on the package itself as well as the raw dependency specifier on the package(s). I’m assuming the former is to allow for linear reading of the lock file for installation while the latter is for documentation purposes.

PDM seems to only specify a single version of a package when locking, but that isn’t inherent to the format.

Why the inclusion of a projects’ description? To help remember what something is for?

Both use the name "package"and not “distribution” or something else more generic.

Both seem to only extract dependencies from a single file and apply it for the overall package version. That’s technically incorrect since every file can have its own unique metadata (both from an sdist/wheel perspective, but between wheels as well). In practice I don’t think most projects tweak per-file and simply rely on markers in order to have a single set of dependencies. I personally think that’s fine, but following this practice would enshrine the expectation in a PEP.

Potential scenarios to support

Environment lock

This is listing the exact files to install if an environment met a specific set of conditions. The conditions would probably a marker expression (support for and and or along w/ a way to invert any operator makes composition possible) and a set of wheel tags.

This is what I initially proposed and covers the case where your environments are finite and known ahead of time (i.e. dev and prod where you’re using a specific Python version).

Package (version) constraints

You list a version(s) of a package and an installer decides to install that package version independently. Whether the package version is considered an optional install is whether a marker expression on the package is met, else it’s skipped. The absence of a marker means it’s required. You would list all files available for that package version and the installer chooses which file to install.

If you restrict this to a single version of a package I believe this is what PDM does (@frostming is that right?). This would help cover the case where the concern over combinatorial explosion comes in, along w/ the “we don’t restrict OS or Python version” case that @charliermarsh brought up from a previous job. I would also hope this covers the open source case of “we want to all agree on what to install to build our docs” case that @rgommers brought up.

Shrunken worldview

All package details are included in the file and acts like a restricted view of the world. This would require a resolver to figure out what to install.

This is what Poetry does.

What I’m currently thinking

I obviously want to support the environment lock scenario. I think the package constraints scenario could also be supported as long as we come up w/ a nice way to signal the file is to be interpreted in such a fashion (probably via a “strategy” or “scope” key at the top-level of the file or by the existence of either a [[env-lock]] or [constraints] table, but not both).

I think expanding out to the shrunken worldview is a future PEP thing (if people ultimately want it), but making sure no decision is made that would prevent it would be good. And w/ @radoering having suggested that marker-only inclusion/exclusion decisions might be enough for Poetry someday, I don’t think leaving out this scenario automatically excludes Poetry.

If this makes sense to people then I think we would be after something like what PDM provides plus some stuff for the environment lock scenario. This is what I’m currently thinking we should aim for (unless you all come forward and say I’m misreading what you all are after ).

My free time is about to evaporate

In case people have missed my public statements on this, my wife is pregnant and due later this month, which means my free time will suddenly disappear anytime between now and the end of the month. As such, I can’t make any promises on getting a PEP written before June or participate in conversations going forward. But assuming someone doesn’t write a PEP in my absence, hopefully we can agree on the scope of a future PEP now and I can plan to write one when I’m at least back at work in June.

[frostming]

@brettcannon

Both seem to only extract dependencies from a single file and apply it for the overall package version. That’s technically incorrect since every file can have its own unique metadata (both from an sdist/wheel perspective, but between wheels as well).

That’s true, but it’s a trade-off we need to make to prevent an excessive number of file entries in the lockfile. In most cases conditional dependencies should be specified with environment markers.

@brettcannon

Why the inclusion of a projects’ description? To help remember what something is for?

Not much useful, PDM lockfile is adapted from poetry at the beginning.

@brettcannon

If you restrict this to a single version of a package I believe this is what PDM does (@frostming is that right?).

Yes

[pf_moore (Paul Moore)]

@brettcannon

In practice I don’t think most projects tweak per-file and simply rely on markers in order to have a single set of dependencies. I personally think that’s fine, but following this practice would enshrine the expectation in a PEP.

This was a matter of some debate in the sdist metadata PEP, if I remember correctly. It would have been much more useful to require that dependency data must be static, but people weren’t willing to accept that. I’m reluctant to say it’s acceptable here without a better explanation of what’s different this time.

And even if we do allow it, the spec would need to say what lockers do when they encounter projects that violate this expectation. Refuse to produce a lockfile? Not check and silently produce an incorrect lockfile?

[groodt (Greg Roodt)]

Ah, yes. My favorite topic. There seems to be a lot of reluctance, even though the number of packages like this is small. 275 at the last count. See the conversation here Insights into how poetry.lock works cross platform - #24 by jvolkman

Poetry and PDM already seem to work this way. So does uv if I’m not mistaken.

I’m not sure where that really leaves us to be honest. If major tools are behaving as if the feature doesn’t exist, maybe it’s not a great feature to keep supporting?

I do think that it probably should be it’s own PEP? Or an edit to an existing PEP? Feels like major scope creep to throw it into this beast of a PEP for Brett. Although I’m no expert on what’s easier here.

[pf_moore (Paul Moore)]

@groodt

Ah, yes. My favorite topic. There seems to be a lot of reluctance, even though the number of packages like this is small. 275 at the last count.

I’m currently running some queries to get figures. I’m not using the BigQuery data, because the documentation isn’t clear that dependency data does actually come from the wheel, so I’m using the newly available standalone metadata files from the simple API.

So far, I’m seeing only 17,507 projects out of 518,575 which have any versions that publish multiple wheels. Specifically, there are 161,301 project/version combinations (out of a total of 5,417,159) which have the potential of varying metadata. My scan of those cases is currently only about 1/16th of the way through, and has found 800+ problem cases so far. So I’m afraid I’m not convinced by the value of 275 that you quoted.

The problem is, there’s a lot of ways of viewing this data, and the scale makes “brute force” counting hard. So we end up with a lot of gut feeling. My personal gut feel is that the number of cases where this happens is very small, but not sufficiently small that we can dismiss the possibility.

@groodt

I’m not sure where that really leaves us to be honest. If major tools are behaving as if the feature doesn’t exist, maybe it’s not a great feature to keep supporting?

Tools are able to make assumptions like this, and if anyone complains, treat that as a bug. Standards like the lockfile proposal don’t have a simple “bugfix” mechanism, so we have to be a lot more careful to consider such edge cases. There’s things we can do to avoid solving the issue (“undefined behaviour”, “up to the implementation”, “out of scope”…) but they all come with trade-offs in terms of how useful the standard will be.

@groodt

I do think that it probably should be it’s own PEP? Or an edit to an existing PEP? Feels like major scope creep to throw it into this beast of a PEP for Brett. Although I’m no expert on what’s easier here.

If we want to make it official that projects are not allowed to publish multiple wheels for the same version of the project unless those wheels all have the same dependency metadata^[1], then that’s a major new standard. I’d personally be all in favour of it, but whenever the topic has been brought up before, there have been some fairly strong objections to the idea. And I don’t personally have the inclination to fight that fight.

IMO, what’s easier here is to accept that dependency metadata is defined per wheel, and write the lockfile standard on that understanding. It will mean the lockfile standard will be a bit more complicated, and might have to compromise on either capabilities or scope, but that feels like the lesser of two evils. Worst case, we don’t try to standardise poetry-style locking right now, and stick with Brett’s original proposal.

---

1. Which effectively requires that the sdist metadata for that version must mark the dependency metadata as “static”. ↩︎

[groodt (Greg Roodt)]

@pf_moore

So far, I’m seeing only 17,507 projects out of 518,575 which have any versions that publish multiple wheels. Specifically, there are 161,301 project/version combinations (out of a total of 5,417,159) which have the potential of varying metadata. My scan of those cases is currently only about 1/16th of the way through, and has found 800+ problem cases so far. So I’m afraid I’m not convinced by the value of 275 that you quoted.

Thanks for doing that! Please record the results somewhere when it’s done if you can. It would be interesting data to look at.

@pf_moore

IMO, what’s easier here is to accept that dependency metadata is defined per wheel, and write the lockfile standard on that understanding. It will mean the lockfile standard will be a bit more complicated, and might have to compromise on either capabilities or scope, but that feels like the lesser of two evils. Worst case, we don’t try to standardise poetry-style locking right now, and stick with Brett’s original proposal.

You’re right. I think so too.

[jvolkman (Jeremy Volkman)]

I ran the query in the linked post again today and there are now 722 results. It will be interesting to see how that compares to processing the actual metadata files.

@pf_moore

I’m not using the BigQuery data, because the documentation isn’t clear that dependency data does actually come from the wheel, so I’m using the newly available standalone metadata files from the simple API.

My reading of the code suggests that the metadata comes from the uploader of the file. The uploader (e.g., twine) should be reading the metadata from the wheel itself, but I don’t think this dataset is checking for liars.

@pf_moore

Specifically, there are 161,301 project/version combinations (out of a total of 5,417,159) which have the potential of varying metadata. My scan of those cases is currently only about 1/16th of the way through, and has found 800+ problem cases so far.

Note that I was only looking at the latest version for each project. If I remove that filter, there are ~11k matches.