PEP 751: lock files (again)

[brettcannon]

@dustin

Got it, so assuming a locker is listing the wheel tags used to generate the locks for that environment, [[packages]].files could contain source distributions as well in this example if there wasn’t an adequate wheel for that set of tags, correct?

Not in the case of per-file locking, but for per-package then yes.

@konstin

This is is not a file URI as per RFC 8089, see RFC 8089 - The “file” URI Scheme which only allows absolute paths, and File Uri Scheme and Relative Files - Stack Overflow for details on the RFC.

I’m going to be a bit blunt here: what are you after? Separate keys for file paths or URLs? I didn’t do that as I don’t expect most lock files to specify file paths in packages.files.origin. Or do you not want relative file paths supported after asking about them?

@BrenBarn

What I mean is that, if I were worried about making a secure environment that were restricted to specific files, what I would want would be to simply distribute a “greenlist” of files that I know are safe.

You got a good answer from @flyinghyrax , but the one thing I would add is simply listing files doesn’t guarantee they don’t interact in a way that’s a problem; being “safe” in isolation isn’t enough.

@dimaqq

Selecting an extra in this package may affect extras in direct dependencies, which in turn may affect extras in transient dependencies… turtles all the way down.

I didn’t immediately see how that would be expressed in the spec.

Because it’s not. My initial expectation was separate lock files would be used for any extras you wanted to lock for.

@frostming

I now believe that creating such an “allow-any” cross-platform lock is either impossible or extremely difficult to do correctly.

Unfortunately you, Poetry, and now uv have done a good job convincing folks it’s possible.

@frostming

We need to change file-locks.markers-values to allowing a marker expression instead of marker value.

If I’m reading what you’re suggesting correctly then is you’re advocating for always having some way to specify what requirements a lock file was generated under, whether they are overly strict or very broad?

@frostming

In this way, file locking is just another package locking with more restrict environment constraints, I don’t think it’s necessary to keep two lock formats anymore.

From a pure, “get me this stuff installed”, you’re right. But as I said in PEP 751: lock files (again) - #99 by brettcannon there’s an aspect to the explicit lock file scenario that makes it fit the security aspect a bit better than asking folks to do a fake install in their head or run some tool to know what would be installed.

@steve.dower

I’m not sure if my support on this point counts for enough to calm the crowds, but you’ve got my support to go this way

I would definitely make it a MUST that lockers have to make it an opt-in to use source-based installs.

---

Over the weekend I thought about this from the perspective of a lock file of package versions and their files and having some way to express explicit/tight constraints (i.e. take something like what PDM, Poetry, and uv.lock offer and build an optional, security-focused aspect on top of it; per-package to per-file instead of the other way around). To me this feels more unified than my previous proposals.

Now I want to remind people this needs to work for both library developers (i.e. pyproject.toml) and app developers (i.e. requirements.in, maybe PEP 735 in the future).

Also remember that I don’t want to have to run a resolver to figure out what to install.

I also have not tweaked this based on anything since PEP 751: lock files (again) - #107 by brettcannon .

With all of that said …

Outline

version = "1.0"
hash-algorithm = "sha256"
dependency-groups = {
  '' = ["..."],  # Or name "default" and have it manually sorted to the top? Or have a top-level `dependencies` that is ignored if a dependency group is specified?
  # Extras are to have their names surrounded by `"[...]"` to differentiate from PEP 735 dependency groups. All names must be normalized (including all extra names). Extra names within a single dependency group should lexicographically sorted.
}

[[resolved-environments]]  # Optional
dependency-group = ""
requires-python = "..."  # Optional
marker = "..."  # Optional
wheel-tags = ["..."]  # Optional
packages = [
  {package = "...", version = "...", wheel = "..."},
  {package = "...", version = "...", source = "sdist|VCS|directory"},
]

unresolved-environments-allowed = false

[[packages]]
name = "..."
version = "..."
dependencies = ["..."]  # Optional
extras = [  # Optional
  "...": ["..."],
]
dependents = ["..."]  # Optional
package-index-project-detail-url = "..."  # Optional
requires-python = "..."  # Optional
direct = false
files = [  # Optional
  {
    name = "...",
    dependencies = ["..."],  # Optional
    # Optionally has `extras`, but its use disallows inline table usage so not shown in this example.
    origin = "...",
    package-index-project-detail-url = "...",  # ?
    hash = "..."
  },
]  # Optional

[[packages.build-requires]]  # Optional
# ...

[packages.vcs]  # Optional
type = "..."
origin = "..."
commit = "..."

[packages.directory]  #Optional
path = "..."
editable = false

[packages.tool]  # Optional
# ...

[tool]  # Optional
# ...

The way to think of this is [[packages]] is a simple graph where the top entry into the graph is listed in [dependency-groups] and then you determine what to install by following where the graph takes you (I will cover this in detail shortly). The security side of things is [[resolved-environments]] which basically is a graph traversal that’s flattened and written down ahead of time.

I also put in support for extras. Probably the most controversial thing is how extras at the top-level of the lock file are normalized into dependency groups.

Lockers

[[resolved-environments]] gets written out in priority order. This resolves the question of what to do if you lock to a pure Python solution on top of OS-specific solutions.

Installers

Here’s an outline of how to decide what to install along w/ some open issues …

• Iterate through each entry in [[resolved-environments]] based on the dependency group selected
  • [[resolved-environments]] is listed in priority order
  • If an entry passes requires-python, marker, and wheel-tags, install the listed files/packages as specified; DONE
• Else if unresolved-environments-allowed is false, raise an error
• Get the list of package dependencies based on the dependency group selected (defaults to '')
• For each dependency, filter based on markers
• Find the matching [[packages]] entry for remaining dependencies
  • Filter based on requires-python
    • Too “resolve-y”?
      • This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
  • Choose the newest package version if a version specifier that supports multiple entries is used
    • If entries are sorted by packages.version from newest to oldest then it should be the first entry that passes muster
    • Too “resolve-y”?
      • This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
    • Could require that version specifiers can only use == to avoid this while allowing varying versions
    • What if a compatible version lacks the specified extra(s)?
      • Error out?
      • Continue search?
      • Too “resolve-y”?
        • This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
  • Look for a supported wheel
  • If no match is found in package version:
    • Raise an error?
    • Try next version?
      • Could require all package version specifiers only use == for simplicity and clarity
      • Use source if the user said using source was okay, go back and look again for source:
        • Install from source in order of most reliability and simplicity that the source code has not changed
        • Install from sdist if allowed
        • Install from VCS if allowed
        • Install from directory if allowed
        • Check source first before looking at wheels of all other versions?
          • Expensive in the face of source building
          • Also less secure
      • Too “resolve-y”?
        • Notice there is NO backtracking in any of these scenarios
  • Find the dependencies for the package (and extras)
    • Specified at the packages.files level
    • [[packages]] level
    • Whatever is specified from the source build
  • Repeat installation of dependencies based on algorithm specified for top-level dependencies

One thing I would be interested in is if [[resolved-environments]] meets the needs for security folks, so if the “security” label applies to you and your job/expertise, please leave a comment whether [[resolved-environments]] meets your needs, isn’t enough, or is unnecessary.