Recognising that there is a spectrum of possible things that might be called lockfiles, perhaps what is missing is a clear statement of the case for defining a format that supports only one of them.
(I can think of potential reasons, I do not mean to imply that this is an impossible case to make).
It would surely be feasible to extend the proposal to support a broader selection of the possible semantics. Temporarily adopting a familiar format - for clarity, not for advocacy - we could have
# version pins as per typical requirements.txt today
# installers of a lockfile like this must choose files based on tags
foo==1.2.3 \
--hash=111 \
--hash=222
bar==2.3.4 \
--hash=aaa \
--hash=bbb
# environmental variation without explosion
# installers of a lockfile like this must be able to evaluate markers
# this is probably hard to produce
foo==1.2.3 ; python_version < "3.10" \
--hash=111
foo==1.4.5 ; python_version >= "3.10" \
--hash=333
bar==2.3.4 \
--hash=aaa
NB none of these is the poetry-style mini-repository. That would be considerably harder to express in this kind of format because poetry records not only the available packages but also each of their dependencies - so that the install-time solving can happen entirely from the lockfile.
It seems plausible too to define a format that initially supports exactly one of those types of locking, but with an understanding that it could be extended to others in future work.
Huh? All of the locking proposals under discussion include files, plus all of their dependencies, in the lockfile. I don’t think anyone has ever proposed an approach that only locks the top level packages under the name of a “lockfile”…
We are at cross-purposes. For a top-level requirement foo, poetry lockfiles record not only that transitive dependencies foo, bar and baz are all in play: but also the individual requirements of each of those packages, alongside any relevant markers.
The current proposal doesn’t record the dependencies themselves, because they can be obtained by reading the file to be installed. But it does record the files that those dependencies resolve to.
I don’t think that copying the dependency data into the lockfile gains anything beyond a small performance gain. And if a tool cares about that, the dependency data could be put into the [tool] section of the proposed format. But regardless, I think this is a digression from the main question.
There’s an exception to this as well. If a lockfile that has the first listed behavior of “Constrain what the installed is allowed to consider” only has 1 option for each dependency, it has the behavior of the latter.
So in essence, if the format provides for both, then the lack of resolution at install time could still be possible for tools that constrain to only 1 solution.
I agree with you up to “because”. The reason that the current proposal doesn’t record the dependencies themselves is surely that it has absolutely no need of them.
However I agree that this is a digression, the thrust of my post was intended to be: let’s talk about the positive case for a format that is deliberately limited in capability.
From my understanding that is also the position of Poetry, and PDM expressed support as well.
The environment locking of specific targets with specific URL artifacts is necessary today but nothing like this exists in the open nor is standardized. I think we should forget the partial lock/version pinning/snapshot-of-package-indices concept for now and exclusively focus on the proposal as-is, especially since there is buy-in from tools.
We are having a good discussion here, certainly not bike shedding, however I see this as having the potential to drag on indefinitely like in the past. Let’s be focused
This is probably my main concern as well. There are several situations I think we want to avoid:
the locking operation isn’t precise enough, so it says it locked for Platform X, but Platform X actually includes several more granular distinctions, so trying to install the resulting lockfile doesn’t work
the locking operation tries to lock precisely, but in doing so winds up overspecifying the necessary criteria, so trying to install the resulting lockfile results in “this lockfile doesn’t work on your platform” even though there is a combination of files mentioned in the lockfile that would actually work
the locking operation works and locks precisely, but to do so it goes down rabbit-holes of locking based on very specific criteria, so that it takes so long as to be impractical (and maybe also generates a huge lockfile)
These are basically in order of badness: saying it installed but not working is worst, saying it can’t install even though it could is bad, and taking a long time is annoying but least bad.
The main thing that worries me here is that what matters is whether these things happen in practice, and I personally cannot gauge how likely that is based just on this spec. This is partly because, as others have mentioned, there don’t seem to be any existing tools that try to do this kind of lock. The spec seems fine conceptually (if we set aside the debate about whether it has the right scope), but it won’t catch on unless tools are built on it and do what users want.
I agree this is a conceptual difference, but I’m curious why (or whether) it’s of practical importance. As you say, we don’t currently have any installers like that. Plus, whether or not a resolving installer is needed for lockfiles, it will still be needed in general to do “regular” (i.e., not locked) installs. So it seems like not requiring a resolver for lockfile installation means everyone will still be using a resolving installer, but they will have the additional option of keeping an extra, weaker (i.e., non-resolving) installer around just to install lockfiles. That doesn’t seem like a real huge benefit to me.
In particular I think that for many use cases, whether the installer requires a resolver is less important than how “annoying” the installation process is in practical terms. By this I mean things like “can I copy around some kind of big file bundle that will allow installing this locked environment without internet access” or “will creating/installing this lockfile take an obscene amount of time”.
I think that’s a great idea (although I don’t have any bright ideas about how to gather such data). I would add that in addition to typical cases it would be good to have some “acid test cases” that involve lots of complicated packages, in particular packages which are known to have cross-platform differences[1].
like ones that have different dependencies depending on the OS, Python version, or other factors ↩︎
The practical benefit is that the algorithm for installing is trivial - take this list of files and install them. That has significant implications when it comes to being able to reason about the content of a lockfile. I can confirm from practical experience with working on pip issues that people routinely get confused about what precisely a resolving installer will do when faced with a set of requirements.
The first case is simply a bug in the locker. It’s the locker’s job to record under what conditions the lock applies, and if it isn’t doing that correctly, no standard in the world can help. The second case is (IMO) not critical - you can always add a lock entry for the specific target you want to allow. And the third is simply a consequence what we’ve already covered, that the spec currently doesn’t state how to define applicable conditions (the tags and markers fields are intended for this purpose, but the spec needs to say how they will actually work). Until we know how targets are defined, we can’t reasonably have a view on how hard implementing the necessary algorithms will be. But having said that, implementation complexity is something for the tool developers to deal with when making decisions about how much to generalise the targets for a lock entry, so we don’t need to know now.
Ultimately, I agree that the PEP will need to discuss these points. But I don’t think they are showstoppers - we need to remember that we’ve had plenty of other successful standards with equally open implementation questions. And while I’m asking for a reference implementation, I’m not expecting that implementation to be the last word on the subject. Approaches can be developed and refined as we gain experience.
(Deleted posts are just from the Discourse editor on Firefox for Android freaking out and posting early a couple of times, before I gave up and switched to a full desktop client)
OK, this may be the point of confusion as my proposal that happens to allow partial locks covers this part of the problem, and the way it does so is also the place where partial lock support shows up as a natural consequence of the design rather than partial locks being specifically introduced as a design requirement (i.e. it isn’t that I think we have to have partial locks for the lock file format to be useful, it’s that they’re a natural consequence of my suggested lock file format, and I don’t think we should go out of our way to disallow them).
Suppose a project wanted to support the following targets:
Python 3.12 on 64-bit Linux
Python 3.12 on 64-bit Windows
Any other Python 3.12 installation
To express this, the lock file would include a targets array defining 3 target environments. The environment names are technically arbitrary, but in this example reasonable names would be py312linux64, py312win64, and py312
[[targets]]
name = "py312linux64"
markers = [ "python_version == '3.12'", "platform_machine == 'x86_64'", "platform_system == 'Linux'" ]
# partial = false is the default
# This means that having more than one artifact for a distribution tagged with this target name is an error
[[targets]]
name = "py312win64"
markers = [ "python_version == '3.12'", "platform_machine == 'x86_64'", "platform_system == 'Windows'" ]
# partial = false
[[targets]]
name = "py312"
markers = [ "python_version == '3.12'" ]
partial = true # Allows more than one artifact per distribution for this broader target
The targets array replaces the markers, tags, and requires-python fields from Brett’s proposal at the top of the thread.
There are then two different ways of using the targets table at installation time, depending on the installer.
A non-resolving artifact installer would do the following (this is @brettcannon’s original use case):
run through the targets array in order, ignoring any partial locks as irrelevant
for the first matching target environment found that defined a comprehensive lock:
run through the artifact lists in the rest of the lockfile, installing any artifacts found with that target listed
if no matching target defining a comprehensive lock for the current environment is found, emit an error
if the nominally comprehensive lock has two or more artifacts tagged for the same distribution, emit an error
A resolving installer would behave differently (this is the Poetry/PDM use case):
identify all defined target environments that match the current environment
collect all artifacts associated with those targets in the rest of the file
use the resolver to decide which exact artifact to install when there are multiple options
if no matching targets for the current environment are found, emit an error
In the example given, a non-resolving artifact installer will error out on platforms other than Windows 64-bit and Linux 64-bit, since there are no applicable comprehensive locks, while a resolving installer would use the py312 partial lock to define the acceptable artifact set to install, using the more specific environment markers of the actual target environment to choose a specific artifact or version when there is more than one option in the partial lock.
Even a resolving installer would error out on versions other than Python 3.12, since there’s no lock targets defined for other Python versions.
After the targets array, the rest of the lock file would specify the artifacts associated with each of those targets (artifacts not associated with any target should not appear in the lock file). To enable easier answers to auditing questions like “Are there any differences in the versions of dependencies installed across different platforms?” this could use a nested-array structure like the following:
# Array of distributions with no version variation across platforms
[[common]]
name = "..."
version = "..."
[[common.artifacts]] # Even for common distributions, there may still be different artifacts (due to wheels)
targets = [ ... ] # Which defined targets include this exact artifact
filename = "..."
origin = "..."
hashes = { ... }
direct = true/false # Optional, defaulting to false if not present
sdist = { ... } # Optional, extra info for sdists (e.g. build-requires, or reference to a build env lock file)
vcs = { ... } # Optional, extra info for VCS references
# Open question here on whether there should be one multi-type artifact list including everything
# (as shown), or separate lists for the different defined artifact types (wheel vs sdist vs VCS repo).
# My own inclination is towards one list with optional type-specific subfields, but I think either is viable
# Array of distributions which differ across targets (by version or by presence/absence)
[[conditional]]
targets = [ ... ] # Which targets include some version of this distribution
name = "..."
[[conditional.versions]]
targets = [ ... ] # Which targets include this version of this distribution
version = "..."
artifacts = { ... } # Same internal structure as [[common.artifacts]] entries
The nested array structure is more to enable human auditing than it is to help out the automated tools (this was the key point @alicederyn raised that put me off the multi-file idea, and made me a fan of @steve.dower’s “conceptually multi-file, but de-duplicated for improved readability” idea).
If the [[conditional]] array is empty, then you know at a glance that there are no distribution version (or presence/absence) variations across the defined targets.
If the [[conditional]] array is not empty, then at least it only lists the distributions that do vary across target environments, so there’s less info to look through when determining if the differences are expected and acceptable.
So “version pinning” is officially a Bad Name. Perhaps those files are “full constraints” files, which “fully constrain” an environment (ie every installed package is constrained, but not necessarily pinned), usually including integrity checking.
While not technically non-trivial, I think it would be pretty straightforward to compute the most restrictive tags and the union of all markers for all dependency specifiers of all files in a lock entry. This would allow all environments to those specific constraints to install from that lock entry. I think Brett alluded to this before.
The initial proposal seemed to leave that for the lockers to figure out, but it seems many here want this included in the specification. I’m indifferent on this. I am assuming that markers which aren’t specified means there’s not constraint on those markers, and I think the spec should explicitly state that. I’m also not too familiar with edge-cases with markers.
Basically, this.
I would be very happy to replace pip with a dumb parse-pylock-and-download-and-install-files app (using eg installer, and standard-library TOML parsing and HTTPS requesting libraries). I’d probably even publish this app to PyPI, for others specifically building apps in locked environments. It’d be easier to read the source than pip, for another thing.
It’s not just a comparison against bundling; for that, you use Docker images (etc) or zipped virtual-environments. Time is also a non-issue, as it happens once for many installs (likely, a locked environment will save more time as each install won’t need to be resolved, and things can happen in parallel).
Rather, it’s a question of having near-identical environments on supported platforms, and also being able to reproduce the environment if you lose the bundle.
In addition, in my eyes having hashes removes the need for not needing internet; I don’t work on air-gapped systems though.
When I’m auditing, I don’t want to open the file to scroll up 2000+ lines to see if a file is under [[common]] or [[conditional]]. Perhaps instead, you have a [[files]] array and put target-names in each file, and either leave targets unspecified or set it to a sentinel (eg "*") for the formerly common files.
In any case, I think you’ve fleshed out your proposal to a point where I think you should stop discussing it in this thread, as it’s becoming separate. Instead, I suggest creating a new Discourse post to move the discussion there. It will also help people be acquainted with the format.
It’s been raised before but I’m concerned that the examples we’re seeing are still using this as an example target. This is not enough information to select wheels. 64-bit what, intel? Arm? What version/s of manywheels is it targeting? Will it support RHEL7, for example? Is manylinux2014 the goal? What if you get a wheel built only for a more recent target?
It’s hard to judge how practical an option is when it’s oversimplified.
This feels like a major issue to me - two different installers can process the same file differently when targeting the same environment (and therefore could give different results[1]). That ruins any possibility of auditing a lockfile.
Yes, I know the results could, and realistically should, be the same. But proving 2 algorithms give the same result isn’t a reasonable thing to require - especially when the exact algorithms are implementation details. ↩︎
It occurred to me that one reason (the only reason?) why the constraints / boundary locking is necessary at all is because:
PyPI releases are open-ended
I don’t believe it’s possible to supply a “highwater mark” timestamp to the candidate listings and resolver
If at least item 2 was possible (ideally both), deterministic resolution algorithms could work directly from the direct dependencies (project.dependencies) and a timestamp field. It wouldn’t be necessary to build a tags and marker “scraper” to provide a serialized projection/view over the candidates for deterministic resolution.
The invariant of blocking open-ended releases means that new hashes won’t appear later inside a version.
The ability to limit the max timestamp of releases during resolution means that new versions won’t become candidates that were released after the timestamp.
Taken together, this would result in deterministic resolution I think. If implemented on PyPI, it could potentially solve problems at the ecosystem level and might be a reason other ecosystems don’t require these constraints/boundary/partial style locks.
We should be careful not to make assumptions about the index. We can’t guarantee people will be using PyPI, after all, and a different index might have no restrictions at all (open-ended releases, ability to delete or replace existing files, anything you like). There’s no standards on what an index must do beyond expose the simple API.
Where it’s often from a security point of view. There is also a comment about how poetry ignores old releases for efficiency reasons.
I don’t know if there’s ever been discussions about a “max_time” anywhere though.
Might neatly avoid the debates about storing or standardising these projections in the short term and the existing tools can continue to store their existing scraped data in the “tool” section.
Then the rest of the proposal to standardise the “lock”/installation plan (no SAT solver) could remain a lot simpler.
PEPs defining what PyPI does are one thing, but PEPs defining something that all indexes must do are a very different matter. I’m not saying we couldn’t, but it would involve getting buy-in from commercial vendors like Artifactory, the Azure artifacts people, etc. That’s going to be hard to do.
It would also involve coming up with a way to make it work consistently with a directory containing .whl files. Hash-based and even filename based approaches handle this quite naturally.
I wonder how much of this current topic is about being able to reverse engineer the original input to the resolver? I don’t see any problem with a lock file representing “the specific files implied by these versions at this time for this platform,” in which case none of the markers or dependency information is really needed.
There isn’t really a need to do this, but why not embed the original specification (i.e. the unpinned/less-pinned list of packages) in the lock file as a plain old string? That way at least if you somehow end up with the lock file but not the original file (because the dev didn’t give it to you? didn’t check it in? bit weird, imho) you still have the list of packages and can refresh the lock file with the original resolver.
But really, I’d expect to see both a requirements.txt and a pyproject.lock or whatever the names are together in a repo, and I’d expect the former to need some kind of resolver while the latter only needs a for loop, urlretrieve and ZipFile.extract_all. If the indexes have changed in the meantime, this is how you get the current “best” set of packages.
