Lock files, again (but this time w/ sdists!)

OK, so the dependency tree information is available from resolvelib (admittedly via a result attribute that’s technically undocumented) but pip throws that information away long before the installation report gets a chance to see it. So we’d need to add that to the installation report if we wanted to make it available in a lockfile. I’ll omit the data for now with the intention of worrying about adding it to the install report later.

2 Likes

I think we should absolutely be thinking about how a lockfile would be produced as well, and that’s what I’m mostly looking at right now. The two are closely tied together, though - should lockers write lock entries that are precisely targeted at very specific environments (marker + tag combinations), or ones that are as widely applicable as possible? Broad lock entries won’t be installable anywhere by an installer that insists on a strict/exact match[1], but tight ones could easily not be installable in environments where they would work perfectly well.

From a producer’s point of view, the situation is:

  • For markers, it’s easy to write a precise specification, you just dump out the environment you used when evaluating specifiers for the resolve. That’s very precise, and quite probably includes data that’s strictly not needed (if the specifiers have no markers, the whole environment is irrelevant), but it’s easy to produce. To write just the relevant environment values requires the locker to parse the markers, which is complex and currently not available as library code. There would have to be a strong need to justify doing that.
  • For tags, dumping out the valid tag set for the resolve environment is also easy[2]. But again, it’s over-specified, and worse, if you don’t have access to an interpreter running in the target environment, it’s impossible to calculate in a deterministic way (there’s no standard for what’s in a system tag set, so even if you do have access to the target interpreter, the best you can do is produce “what the packaging library returns”). In this case, though, it’s relatively easy to build a “minimal” tag set - just parse the supported tag sets for all the wheels in the solve, and the minimal set is the union of those. Although now that I think about it, because tag sets are ordered, set union isn’t actually sufficient - but I can’t think of a place where ordering would be needed by a lockfile installer (which shouldn’t be doing any resolving) so it’s probably “good enough”.

I don’t want to derail this attempt at getting lockfiles by digging too deeply into the weeds of “what it means to define a compatible environment” but I think we do need to make sure that whatever’s in the spec is something that can be implemented in a practical way by both producers and consumers.

From the feedback from the poetry folks here, it sounds like their approach is different, so I don’t know if their insights (beyond “don’t do it that way” :wink:) will be applicable. But even if the proposal here is not what they do, practical experience is very valuable in any form. It would also be useful to hear from PDM as to how they addressed this issue. And doesn’t pipenv do locking - what’s their view here? I don’t know who to ping for pipenv, unfortunately…


  1. Because a broad spec is by definition not an exact match ↩︎

  2. Typically - I have a problem doing this, but that’s an oddity of the source data I’m using ↩︎

2 Likes

@matteius maybe?

1 Like

I don’t know how other ecosystems without a hash in the lock file
solve this issue or if they just accept to get invalid lock files
after merging. :person_shrugging:

In OpenStack we use a central pip constraints file (hashless) as our
makeshift lockfile, with a pip requirements file containing loose
version specifiers as the input to a constraints generator run in
CI. Contributors propose exclusions or other adjustments to the
requirements file, and the constraints file is essentially a pip
freeze of the result of installing the requirements into a venv fora
representative platform (there’s a bit more magic than just that,
and some rough edge corner cases I’d solve with adjustments to the
model if I had time, but it’s working out pretty reliably for a
combined set of around a thousand requirements).

I guess the point is, I agree that proper lockfile updates need to
be the output of a generator that solves across your proposed input,
naively proposing direct edits to lockfiles is definitely
problematic as you scale up.

1 Like

My proof-of-concept can emit locks for Linux, Windows, and pure Python for Python 3.8 - 3.12, so I don’t consider this insurmountable.

Why can’t we have both? :grin: For instance, I have a --maximize argument in my proof-of-concept that takes speed or compatibility as arguments. The differences is the latter reverses the tag preference order for the resolver so you get pure Python wheels when available over more specific wheels.

If you look at mousebender/pylock.compatibility-example.toml at resolve · brettcannon/mousebender · GitHub you will notice that I locked for CPython 3.10 for manylinux 2.17, CPython 3.12 for 64-bit Winndows, and then Python >=3.8 as an example of this (and all on calculated on the same machine). So that covers both the “specific” and “widely applicable” scenarios.

I agree that the ordering is only important in selection of a wheel file during locking/resolving, not in determining compatibility. In thinking about this topic, as long as requires-python and the wheel files are all compatible with the environment then that’s all that’s truly necessary to determine if a lock entry would work (sdists obviously mess this up like they always do).

Now determining which of multiple lock entries is “best” is a bit more work. But a naive way to determine this would be to calculate an average weight of “perf” based on how the wheel tag triple ranks for my environment’s preferred tag triple order and then go with the one with the “best” weight. Thanks to lock entries within a lock file meant to represent the same outcome in different environments you can’t really have a “wrong” choice as long as the compatibility needs for the environment are met.

1 Like

FYI, I did try your proof of concept, trying to guage real world requirements to lock files. But I found many typing constructs that were only available in newer versions of Python, such as @typing.override.

Even after fixing this I was often just given the error, even for very simple requirements, that the requirements were unresolvable. I didn’t spend much time investigating this, does your resolver rely on PEP 658 metadata files being available? If so it seems I will need to wait a few months for PyPI to full backfill.

Otherwise should I report or try and investigate why it’s unable to resolve?

1 Like

I think that might be the important point to document in the PEP - installers (and other consumers) must reject (because they won’t work) lock entries where either the requires-python doesn’t match, or one or more of the wheels in the lock entry aren’t compatible with the environment (based on tags).

Are they allowed to flat-out reject additional entries that the above rule doesn’t require them to reject? I guess we can’t stop them, but we run the risk of ending up with installers and lockers being tightly coupled because they have a shared, but idiosyncratic, idea of what “compatibility” means. That doesn’t feel like a good outcome. Maybe I’ve been wearing my “interoperability” hat for too long, but I feel like a lockfile standard that doesn’t ensure that users can pick which locker and which installer to use independently, hasn’t done its job (after all, that’s what we have now - lock files tied to specific tool implementations).

Determining the best valid lock entry to use is up to the consumer. I think that’s fine. The producer’s job is just to provide a list of lock entries that work, not to judge the trade-offs for the target environment.

But with the above context, I’m not actually seeing the benefit of having the “marker” and “tags” values in an entry. They are (possibly) useful information, but beyond that I’m not sure. Put it this way. Suppose I’m writing an installer, and all I know is that I’m going to be handed a file that conforms to the lockfile standard, that was produced by some tool which is supposed to have included at least one lock entry that will work on this environment[1]. How can I use the marker and tag data as part of my decision process?

This isn’t hypothetical. Pip will need to install lockfiles, I imagine, and we’ll need to support a command line like pip install --target staging_dir --platform manylinux_2_17 --lockfile xxx. We won’t know who produced the lockfile, or what platform they were on when the locker ran, all we’ll know is that somewhere in that lockfile there’s a lock entry that the user believes is valid for the target platform. We won’t be able to build a markers mapping for a foreign environment, and our supported tag array is built using our own implementation, which could be subtly different than the one the locker used. I’m struggling at this point to see the benefit for an installer in looking at the tags and markers entries in a lockfile…

Sorry if I’m missing the point badly, here. I don’t have any experience with use cases for cross-platform or multi-platform lockfiles (or just cross-platform installs, to be honest!). That lack of knowledge may be what’s making all of this seem so hard to me…


  1. Basically, I’m assuming the user won’t try to install a lock file that they know isn’t going to work ↩︎

1 Like

Yes, hence why it’s a proof-of-concept. :wink: I don’t think replicating pip or uv is the best use of my time, so I did the simpler solution

It should be less than 3 weeks.

Thanks for the offer, but no. With uv now available I will very likely not move my proof-of-concept any farther forward beyond what it does now.

:+1:

I agree and have tried to take that into consideration by recording all of the key inputs a locker’s resolver would have needed to take into consideration (that we have standards around, e.g. I’m not recording the fact someone requested the locker calculate for the oldest version).

I put the markers and tags used to do the resolution for cheap validation of a lock entry when you’re using the lock file to cache the listing of what you installed (hence the “strict” check you saw in my proof-of-concept). I also use lock.tags in the compatibility check as I assumed the tag set would be a subset cleanly. But I like your point that it’s overly conservative to assume those tag inputs define compatibility when in actuality it’s what is eventually resolved.

I’m going to update the proposal to mark lock.markers and lock.tags as :fire: and update the PoC to not rely on them for installation to think through if there’s any reason to keep those keys around.

2 Likes

Update:

  • Marked lock.markers as :fire:
  • Marked lock.tags as :fire:

I may be misunderstanding, but how does this requirement interact with source distributions, where you may be required to build the distribution in order to determine its requirements (and where the distribution itself could be impossible to build on the platform you’re running on)?

3 Likes

I also don’t know the answer to this but agree that it’s important. My machine has 582 tags via packaging.tags, and on Linux that will also discriminate on glibc version, etc.

In this case, though, it’s relatively easy to build a “minimal” tag set - just parse the supported tag sets for all the wheels in the solve, and the minimal set is the union of those.

Is the thinking here that you’d generate the lockfile by starting with the markers, perform the resolution, then determine the appropriate set of tags?

Edit: at least in mousebender, it seems to be structured as: given a Python version, generate a representative set of markers for (e.g.) ARM Windows, then generate a set of tags using packaging.tags. (Somewhat similar: in uv, we support resolving for a different Python version by patching the markers.)

Two questions here:

  • Can you say a bit more about why including the indexes is helpful? Is this meant to be used by resolvers, rather than installers themselves?
  • Is --find-links supported at all here? (Sorry, I don’t know if there’s a more formal term for that behavior.)

Yeah, our intent is to support resolving across multiple platforms and Python versions. But what’s undecided is whether we’ll design and implement this as…

  1. Generate a “universal resolution” (like Poetry), i.e., attempt to generate a “single” resolution that is then re-resolved and narrowed-down at install time based on the current platform.
  2. Take the set of supported platforms and Python versions as input from the user, and generate a separate resolution for each combination. This approach is much simpler (in my opinion) but brings with it a bunch of theoretical challenges, since markers are extremely granular.

While we’ve talked about it a bunch internally, we just haven’t gone through the process of actually working through the problem yet. (Design #2 would be more amenable to this proposal; Design #1 would not be amenable at all, I don’t think.)

2 Likes

Regarding extras: is it that every combination of extras would require a new entry, or an a new lockfile entirely?

I did it and it worked (as expected)!

If we go the route of dropping lock.markers and lock.tags then we could change lock.wheel to have version, build-number and tags keys if people didn’t want to bother parsing filename to get that information.

If build-system.requires is specified and PKG-INFO has a metadata version of 2.2 or greater and Requires-Dist (and friends) are not dynamic then I believe all the information you could want is specified statically. But I guess if get_requires_for_build_wheel() decides to be fancy there’s only so much you can do short of throwing your hands up and say, “don’t use sdists” (although the last time I said that I got a PEP rejected :wink:).

Yep, that’s one possibility I had in mind.

Yes. If you want to know how the lock file was generated that’s a key piece of information. It also comes into play if you want to append other lock entries later. If people want to throw out the idea of appending lock entries then that key would be superfluous.

No because that is a pip-specific thing and not standardized.

Yep, although I am willing to bet the “extremely granular” markers are rarely used (famous last words, I know, but who is relying on e.g. platform_version which is the most granular environment marker to the point that you know you wouldn’t work with anything but the exact environment you’re targetting?).

Correct, this proposal does not support a Poetry-style lock file at all, and that’s on purpose.

Because people keep bringing up Poetry’s approach, I do want to say this proposal doesn’t support that style of lock file on purpose. As @ofek alluded to earlier, from a secure supply chain perspective this proposal is a better fit in my opinion (else I would have proposed something else :wink:). You can be very clear and know upfront what will be installed without any work with this proposal; you’re locked into what would be installed. Poetry’s lock files, though, are more of a subset of the world at the time of locking where you still need to figure out what files apply to you at install time. To me that isn’t locking files as much as locking a subset of files that may apply to you, but you won’t know until you run an installer. That makes auditing harder, etc. which are things secure supply chains care about (and thus I care about; I want to be able to look at a lock file with a single lock entry and go, “yep, that’s what will be installed” w/o having to think about it).

The same can be said about reproducibility of environments; I believe the potential variance of what gets installed is simpler to figure out with this proposal than with Poetry’s approach.

But as has been pointed out, the drawback of my proposal is the lock file is not inherently universally cross-platform without a locker choosing to make that happen, and even then it’s going to potentially be for a subset of platforms (it’s all a case-by-case basis based on what Python versions you care about, whether pure Python wheels are available, sdists, etc.).

Pros and cons, but for me I care about secure supply chain and installation reproducibility that easy to reason about. If people disagree then so be it and I will step aside and let someone else propose a standard for a different type of lock file.

I don’t understand the question. When you lock for a set of requirements they are recorded in dependencies, so whatever extras you have in there are what you are locking for. Each lock entry represents a different set of distribution files that work together for some environment, so the only way I can think of extras in any way influencing the outcome is if some e.g. wheel file has another extra added in due to some marker making to only show up under a certain environment.

3 Likes

Why does it feel from the conversation so far like it must be one or the other type as a standard? It looks to me like both types of lock files are important, just in very different scenarios. For example,

  1. If I am working on some application inside an org and deploying it in production on a single well-defined platform, a narrowly targeted lock file with very explicit and predictive behavior about what will be installed (this proposal) sounds great,
  2. If I am doing something like building and deploying docs with Spinx in a CI job for an open source package, I want to lock versions of sphinx, docutils & other dependencies with == but I also want contributors on other platforms, python versions etc. to be able to build & debug locally, too much platform-specificity is going to be actively unhelpful. So there the Poetry behavior seems more useful.

The danger here with blessing either one choice as “the standard Python lock file” seems to be to prefer one type of use case over the other. Standardization in a PEP also usually means pressure on tool authors to adopt that way. I can imagine Poetry & co adopting this lock file type in addition to what they already provide, but I don’t see them breaking their current users by dropping what they already have.

Since, as you say, there are pros and cons to both types of designs, would it make sense to explicitly start your future PEP with “there are two ways of doing this” and leaving some room for standardizing the other type of lock file in the future?

11 Likes

This came up in the previous lockfile discussion, where an important idea was that installing a lockfile should be possible without needing a resolver. And honestly, to me it doesn’t actually feel like things are “locked” if you have to perform a resolution.

@brettcannon is “you must be able to determine what will be installed from a lockfile without needing a resolver” a design constraint for this proposal?

7 Likes

A few thoughts:

With static metadata for wheels and sdist, we can probably lock for a lot more environments than we think?

Is it possible that with containers similar to cibuildwheel, I guess it’s possible for a locking tool to also lock for many environment? From macos, you can lock for macos, Linux, Windows. From Linux and Windows, you can lock for Linux and Windows…

It also seems like this file format supports appending? So is it an idea that to support additional environments they could be appended afterwards? For example, when a new contributor joins the project?

3 Likes

This is a genuine question, but why does this use case need a “lock file”, as opposed to just a set of version constraints that can already be supplied to installers via a constraint file? It seems to me that the key point about a lockfile is that it specifies the exact file to download, whereas what a use case like this wants is just to specify the version. If that’s the case, you can do this already using constraint files (which, unlike requirement files, are simple enough that they don’t need standardising, they are just a UI feature for installers).

I’m not trying to dismiss the use case, just to understand whether we need to look at it differently and not assume “lock files” are a solution for every problem in this area.

… or maybe we need both and what we need to do is come up with some terminology so that the name “lock file” isn’t overloaded with two different meanings.

Also, like local file names, it’s inherently tied to a single machine (ignoring shared filesystems for the moment), which again doesn’t seem to fit with the idea of a lockfile that can be given to another user.

5 Likes

A note from an end user with a long history with pip-tools and Poetry.

One thing as a user that concerns me with this proposal (if I’m understanding it correctly) is that it could allow different versions of the same package to be locked for different platforms (e.g., httpx 0.10.0 on mac and 0.11.0 on Linux depending on the available wheels). This means that you could be unit testing on Mac with different versions of a package than are used in production on Linux or on CI.

Obviously, this kind of behavior is expected for different packages when you truly have markers that say something like “install tensorflow-macos on mac, but tensorflow on Linux”, but having the identical packages possibly lock to other versions seems worrisome.

For example, we use Poetry and while it can sometimes be annoying to track down an issue where you lock on x86 and then someone tries to install on ARM and a wheel is missing, at least you get a nice error and then you know that you can manually pin to an older version. There aren’t “silent” differences that could sprout up unexpected differences at runtime due to differing versions of the same package.

We’ve found in practice that Poetry’s “lock everything” and “pick what to install at runtime” approach to be a lot less brittle than the pip-tools method of making independent, platform-specific lockfiles. But I do understand the nuance that there are tradeoffs between both approachs.

10 Likes