How should a lockfile PEP (665 successor) look like?

Packaging
82

To use a few more words, my take on this is that…

3 is not tractable because we allow for dynamic build dependencies for building via an sdist and for dynamic runtime dependencies in wheels that come out of sdist (read dynamic as “not knowable without doing the build”) and 2 results in a watered down variant where a reproducible build cannot be achieved using the lockfile; unless we exclude sdists which is 1 and… we established against 1 being good enough through the rejection of PEP 655. :slight_smile:

So… I think our best bet is to do 2 at this point which throws away the benefit of being fully reproducible. At that point, we might want to loosen other guarantees as well.

However, if I had the bandwidth to author a follow up to PEP 655, it would involve writing down my argument above against 3 in more detail with an example, adding support for source distributions while treating them as if they are gonna build the exact same thing as it did when we locked it. And, to include a strong recommendation to encourage lockers to support generating lock files that don’t contain sdists at all.

I do think cross platform lock files are still valuable and, as established in PEP 655’s discussions as well as here, we’d need to include more “examples of workflows” to make things clearer from that perspective as well.

2 Likes
83

I would be fine if the build fails if the build and wheel requirements can’t be satisfied by the pins in the lockfile (where the build dependencies are pinned). It’s not often there’s a scenario where you can’t reproduce the install environment (which, in theory, should have the same requirements).

1 Like
84

I think the key here is to precisely explain the implications of the scenario @pradyunsg described where we can’t determine the dependencies without building, and then document what that means in terms of what a lock file consumer would do in that situation (which might just be “undefined behaviour”).

Then we need to get consensus that the resulting limitations are acceptable. One of which is that having a lockfile isn’t enough by itself to guarantee reproducibility.

1 Like
85

This is the theoretical sequence of locking and installing I have in my head.

  • The locker determines the dependency tree
    • For wheels, this is straight forward
    • For sdist, a build is done in an isolated environment.
      • The dependencies that are installed into the build environment are recorded to be included in the lock file. This allows for static and dynamic build time dependencies.
      • This might be optimization instead of a full build once the build time & runtime dependencies are known.
  • The locker writes the lock file with all the wheels & sdists (with associated build dependencies).
  • The installer is passed the lock file
  • For each listed sdist
    • The installer creates an isolated environment to build an install-able wheel.
    • The installer installs the listed build dependencies.
    • The installer performs the build but DOES NOT install build dependencies requested by the sdist. (I suspect this is the part that would be “hard” as nothing expects it to be possible and might break current standards.)
  • The installer installs all the resultant wheels (either built locally or pulled from the index).

So that would be in line with @EpicWink’s comment about allowing things to fail if an sdist can’t be built a second time with build dependencies that were successful previously.

86

This is indeed the hard part. Not actually because tools aren’t set up to work like this at the moment (that’s a relatively simple problem to solve, as you’d just need to create a virtual environment and then use pip’s --no-build-isolation flag to let you manage it yourself). But mainly because there’s no assurance that if you use the “locked” build dependencies that the sdist will build, or the resulting wheel will work, in the target environment.

For a simple example, consider a setup.py that imports pywin32 for some reason. And the build backend injects pywin32 as a build dependency on Windows. Then, suppose you lock on Linux, giving a set of build dependencies that does not include pywin32, and try to install from the lockfile on Windows. The setup.py file will simply fail with an import error.

If you think “well, that’s just a matter of installing on the same platform as you built on”, then how about if I make the dependency on pywin32 conditional on the machine name? Or on the current date?

This is what @pradyunsg was referring to when he said locking build dependencies was “not tractable”. There will be cases that fail. And they can fail in a way that invalidates the whole deployment via the lockfile. What we need to know is what those cases are, and how much people care about them. And that depends a lot on how people will use the lockfile feature, something that we frankly don’t know yet - typically, only people with relatively straightforward use cases tend to get involved in the discussions, and everything seems relatively clear cut, but then when the feature gets released we get all sorts of weird cases that people expect to work.

All of which is why it may be better to stick with a simpler model initially, one that we can describe precisely (people will still try weird stuff, but we can point to the spec and say “that’s not supposed to work” :wink:). The evidence from PEP 665 is that a wheels-only proposal, while easy to describe, is too limited for people. So we need to know what the middle ground is - and because sdists involve arbitrary code execution and dynamic configuration, that’s really hard…

We can’t even say “as long as the sdists build, we have a copy of what was locked” - what if the sdist installs malware 5% of the time, randomly? How do we describe what you can do with a lockfile in the face of that? “As long as building the sdists gives the same wheels as were created on the original system, and as long as the sdist builds don’t fail, and as long as the generated wheels don’t have behaviour that depends on the contents of the build environment, the installed environment should be similar to the environment that got locked… Unless it isn’t, of course…”

(Yes, I’m exaggerating. But assuming everything will be fine is just as much of an exaggeration, just in the other direction.)

87

I agree with what everyone has mentioned about sdist and arbitrary code execution, but I think we are at risk of giving too much weight to the theoretical impossible vs the practical reality.

As many have mentioned, including me How should a lockfile PEP (665 successor) look like? - #46 by groodt

As long as we are clear upfront about the trade-offs, I don’t believe users will be surprised.

Essentially, if an sdist is encountered anywhere (even if installing on an installation platform that is the same as the locker platform) a warning should be printed that can point to a README about the intractable nature of sdist and arbitrary code execution.

If we assume that most popular sdist packages are well-behaved (which I believe is true), then installation using a lockfile will still be successful in the majority of real-world situations, which is what most people are asking for.

1 Like
88

I agree, and this is precisely what I’m trying to argue for. The crux of the issue, though, is describing those trade-offs in a way that lets users determine up front whether their usage is safe (to put it another way, whether a lockfile is suitable for them). Right now, I’m seeing a lot of people use phrases like “if sdists are well-behaved”, but very little explanation of what precisely makes a sdist “well-behaved”…

89

Alternatively, download the sdist and build dependencies into a folder, and use -f --no-index.

Does that assurance need to be made? If the answer is, “without it, volunteers will have to deal with many bug reports,” are you saying lockfile functionality is not worth the sacrifice?

I would say those sdists are buggy, and require fixing before using in a locked set. There isn’t an easy general solution for publicly-used lockfiles where the dependency is deep in the tree, but for most cases I expect this to be a non-issue or simple to fix.

I would like to see an optional feature which warns if the contents of the wheels to be installed don’t match expectations. This is pretty close to “guaranteeing reproducibility”, but I don’t expect all installers to implement it, nor do I think it needs to be precisely fleshed out in the spec.

I don’t think that’s all exaggeration at all, if anything you’ve likely missed some conditions. I’m saying that I’m personally happy with dealing with or ignoring those for my level of reproducibility. No environment can possibly be perfectly reproducible, so at some point distributors have to make a judgement call.

I think an sdist is well-behaved (in the context of dependency pinning) when it can be built with the same dependency set for every build on a platform (OS, architecture, build environment), generating a wheel with the same API, functionality and side-effects.

This does not necessarily mean the same build requirements, nor a byte-for-byte match of the wheel’s contents (but these are easy ways to guarantee a well-behaved sdist).

In conclusion, I’m arguing for option 3 (pin wheels, sdists and sdist build-dependencies), but with leniency on the output reproducibly and install success.

2 Likes
90

This feels like a good starting point that could be fleshed out more explicitly for the next PEP.

It is true that currently it is possible to do anything with arbitrary code execution. I think most people here would agree that things would be both simpler and “better” if some of that was restricted. Environmental Markers provides a base example for how Python was able to statically define dynamic runtime dependencies. I just tested them in build-system.requires with build and it already works.

I can see someone doing something dynamic based on the machine name, but I would argue that there is a better way to do that. That machine name is more of a short hand for an expected environment (OS, architecture, etc). However, that machine might have the OS upgraded or the package now needs to be able to build on two different identical machines. Using a short hand makes these things harder.

While I’m sure there is a package out there that does install malware 5% of the time, I think that we can agree that that is bad and is low on, if not off, the list of use cases we should support.

91

Here’s a legitimate workflow that depends on the dynamic nature of sdists to work around a tooling limitation: a package that has different dependencies (say… for mxnet) based on what GPU/CUDA/pick-your-custom-accelerator is available on the machine.

That’s not malware or random.choice or machine name. That’s a legitimate use case, that installs different dependencies based on the build environment. You can’t represent this with environment markers. If the build environment doesn’t match the install environment with an sdist, there’s a problem here and you might have subtly incorrect behaviours — and, more importantly, we’re locking out those workflows which use this escape hatch and making them incompatible with lock files. This does two things: it’ll push people to pick one or the other — the escape hatch for limitations in existing Python packaging tooling or working within lock files, and lead to subtle failure modes that require take a lot of context/understanding to diagnose (think: new contributor to an open source application who doesn’t use the exact same environment/hardware as the developer who generated the lockfile, but is on the same OS) and could have non-trivial workarounds.

Another example would be a package that depends on a different version of a runtime dependency, based on what’s available in the build environment. In UX for constraining build-time and runtime dependencies · Issue #29 · FFY00/meson-python · GitHub, there’s extensive discussion about how this is a relevant problem for working with multiple foundational data science packages. I’m unsure if this fits @EpicWink definition of well-behaved. It’s a lot more subtle than “sdists won’t work” or “sdists included but all bets are off”.

That said, it might be a worthwhile definition to work with and… well, instead of opining further, I’ll defer to whomever writes down the design document on this to investigate and elaborate on the specific trade offs. :slight_smile:

2 Likes
92

It seems like the main problem is what to do about dynamic sdist installs. An option used by some tools now is to observe what is installed and lock that. Those tools are making a tradeoff that they aren’t guaranteed to be reproducible with other platform markers. Another tool could make the opposite call and error in that case.

We could add a way to manually record “this package is an sdist that might install any of this set of packages”. Instead of recording the exact set of dependencies (with platform markers) for a wheel, the tool generating the lock file records the possible dependencies for the sdist as specified manually by the user. Then the tool installing from the lock file sees that the package is an sdist, lets it run to figure out what packages it actually wants to install, then installs those from the lock file.

# pyproject.toml

[project]
dependencies = ["regular-wheel", "complex-sdist", "basic-sdist"]

# a new section used by lock tools
[[lock.dynamic-sdist]]
name = "complex-sdist"
possible-dependencies = [
    "library>=5",
    "library<5",
    "windows-only",
]

If lock.dynamic-sdists isn’t defined for a given sdist, a tool can decide to do what tools do now and lock what ends up installed, or show an error. The error message could be helpful, or could just be a warning that assumptions were made:

Can’t lock “complex-sdist” because it doesn’t have wheels available. See the docs for how to specify its possible dependencies for locking. On this platform, I observed that it installed “library==5.0.3” and “windows-only==1.1”.

Can’t guarantee that all possible dependencies for “complex-sdist” were locked because it doesn’t have wheels available. See the docs for how to specify its possible dependencies if needed.

Yes, it would require some extra work on the user at that point to determine what to actually write in as the dependencies. That could be an incentive to report that to projects and either have them release wheels, or otherwise document what their possible dependencies are if they have to be an sdist.

The lock file might end up looking like this. I’m just going off what pdm.lock looks like.

[[package]]
name = "my-app"
version  = "1.0"
type = "wheel"
dependencies = ["regular-wheel==2.0.1", "complex-sdist==1.4", "basic-sdist==1.3"]

[[package]]
name = "regular-wheel"
version = "2.0.1"
type = "wheel"
dependencies = ["numpy==1.23.3"]

[[package]]
name = "complex-sdist"
version = "1.4"
type = "sdist"
possible-dependencies = ["library==5.0.3", "library==4.8.1", "windows-only=1.1"]

[[package]]
name = "basic-sdist"
version = "1.3"
type = "sdist"
possible-dependencies = []

# and then the pins for the dependencies/possibles as well
# and the file names and hashes

The tool has recorded what type of thing it is locking, which is not done today. When installing, if a tool sees wheel it can just follow each dependency and install it as usual. If it sees sdist it can run the sdist but only install from packages that match what’s in the lock file.

2 Likes
93

I think we can’t call them “reproducible” as soon as sdists are involved. At that point you’re locking down some of the inputs to the installation and nothing more.

:+1:

I have always assumed sdists would be an opt-in thing so people don’t unintentionally end up in a bad situation. This is especially true in terms of how it might impact generating cross-platform lock files.

One way to solve this is to have the concept of trusted indexes for sdists. For instance, you could allow sdists from your company’s internal index as you know they won’t be malicious (intentionally), but public sdists from PyPI are out of the question (wheels can be viewed as different as you can audit that code more easily and then lock to the hash of that wheel file to avoid tampering). Now that might require your internal sdists to pin their build dependencies to facilitate controlling the outcome, but if you’re using an sdist from a trusted index then you are already implicitly trusting the source.

This sounds like a constraints file. The other question is whether people will bother to figure out that list or copying it into their lock file if the project happens to give it to them (assuming the project even knows as the entire build dependency tree would have to participate)?

94

Given that the lock file design seems intractable otherwise, we’re just constantly spinning on “but what about this sdist”, it seems better to provide a manual approach than nothing. Tools still have the choice on whether lock the observed dependencies or require manual input. I’m sure users would notice in the latter case, and probably would have chosen such a tool for that more “precise” sdist handling to begin with.

I’m not suggesting that the whole tree is specified manually, only anything that sdist would end up using in install_requires, just like a project only specifies its direct dependencies. The lock tool would treat those manual items as just more items to traverse and add to the lock file.

2 Likes
95

I guess another way to put it is:

We’re looking for a file format for specifying lists of packages at given versions and what those packages in turn depend on. How a tool decides to generate that file, and how accurately it generates it (for satisfying multiple platforms, reproducible builds, etc.) is up to the tool, not up to the lock file spec. The lock file spec is only concerned with representing the tree, not generating it to begin with.

3 Likes
96

If that is what we’re looking for, how does it differ from a fully pinned requirements file like the ones pip-tools generates? I thought someone had already said those weren’t sufficient…

97

This boils down again to whether we want to have a reproducible installation, or a fully reproducible build. The first can only be achieved with wheels when using the Python tooling alone. The second cannot be achieved by Python tooling alone (because of dynamic dependencies but also because of native dependencies).

However, third party tooling can achieve fully reproducible builds, but to reasonably get there a best effort needs to be done to include sdists. Manual locking as also suggested by @davidism can be done by third-party tooling (in my case, with Nix).

Stepping back to the 3 cases that were listed before. If we go for 2) instead of 3) it means third-party tooling that wants reproducible builds needs to manually lock all build-time dependencies. It means maintaining a file with build dependencies such as

Note however that this specific solution (the JSON file) is problematic, because here the assumption is made the build system never changes. We should use conditionals instead. Regardless, by locking/recording as much as possible the burden of maintaining such a list of build systems significantly decreases.

98

requirements.txt is flat, one line per installed package, and all lines will be installed. (Possibly some will be skipped if they include environment markers, but their dependencies won’t have those markers.)

The lock file I’m describing (which already exists, I took the format from PDM) is toml, it contains a section of metadata per package. Each package lists its name and version, like requirements.txt, but also its list of dependencies and hashes for every file that PyPI has for that version. So you can build a tree from it, rather than only having the flat list. The change I’m suggesting is to also record the type, wheel or sdist.

The install tool can traverse the tree from the root (the project being installed). As it encounters each node, it adds that packages dependencies to the traversal queue. If a given dependency’s environment markers don’t match, it can stop traversing down that branch. If a package is an sdist, it lets it run to generate install_requires, makes sure that it is a subset of the listed possible dependencies, then continues traversal with that subset of branches.

2 Likes
99

Here’s some pseudo-code for the lock file generator and installer:

root_deps = read_deps_from_pyproject()
q = deque(root_deps)
lock_infos =[]

while q:
    name = q.popleft()
    info = get_pypi_info(name)

    if info.type == "sdist":
        possible = pyproject["lock"]["dynamic-sdist"].get(name)
        
        if possible is None:
            # either observe egg_info, or raise an error
            possible = observe_egg_info(name)

        info.deps = possible

    lock_infos.append(info)
    q.extend(info.deps)

write_lock_file(lock_infos)

tree = read_lock_file()

q = deque()
q.append(tree.root)

while q:
    info = q.popleft()

    if info.type == "wheel":
        deps = info.deps
    else:
        deps = observe_egg_info(info.name)
        assert issubset(deps, info.deps)

    # only add the lock info for dep that match env markers
    q.extend(tree[d] d for d in deps if env_markers_match(d))
    # install the package using the lock info
    install_package(info)
100

I wrote the following before @davidism proposed the possible-dependencies idea, but unfortunately forgot to post it. The content is therefore somewhat out of sync of the latest discussion. I decided to post it verbatim, however, since it approaches the situation from a different angle (the possible dependencies should be declared by the build backend).

I wonder if this could be solved by flipping the perspective. Wheels are easy because their dependencies are static, so perhaps we could add a field somewhere for a backend to declare all its build dependencies statically. At build-time the statically declared requirements would act like constrainsts i.e. only matter if they are actually needed. There are still issues to be figured out, such as what a tool should do when a build dependency does not declare this static value (maybe the practical approach is to guess by calling get_requires_for_build_wheel and assume the return value is static), and how can a backend/sdist explicitly declare it having non-lockable build dependencies and the lock tool should abort. But having the build backend declaring (possible) build dependencies statically seems to be a reasonable start toward a solution of some form.

1 Like
101

This may be a silly question, but would it be easier to simply standardise an existing format like this? Presumably there would still be people whose use cases weren’t covered, but at least we’d know there was a reasonable user base whose needs were catered for.