Thanks for the clarification, it’s getting hard to follow some of the threads here (and I think I’m taking it on myself to respond to too much, I should probably take a break, for my own sanity if nothing else )
I’m honestly not 100% sure. But I think it’s mostly down to what we are willing to consider pathological. If the locker does a resolution under a given set of marker/tag conditions, then IMO it’s reasonable to expect that any sdist involved will behave the same if re-built under those same conditions. The locker will then decide precisely what conditions to put into the lockfile - which might be extremely precise (i.e., the exact set of markers/tags used by the build machine) or might be broader (with the locker having some logic to decide what to consider “compatible”).
I’d say:
It’s the locker’s responsibility to not loosen the constraints more than is “reasonable”, in the sense that it’s fair to expect sdists to behave consistently in all included environments.
We haven’t thrashed out how environments are specified yet, and how lockers and installers agree on what matches a specification. I’m assuming we come up with something “reasonable” here.
Lockers that don’t like this level of uncertainty can refuse to include sdists, and installers that don’t like it can refuse to install lockfiles that need sdists. So there’s an opt-out.
I’m studiously not looking too closely at that question, because there’s no way to say which isn’t subjective. The previous lockfile proposal (which you weren’t involved in, I think) demanded reproducibility, which meant sdists were unacceptable in any form, and that was too restrictive to be acceptable for a large group of users. This proposal tries to loosen those constraints to allow sdists, but not too much, so we can still reasonably consider sdists that build inconsistently across the specified targets as being “pathological”.
It’s going to be tricky to get the balance right between specifying environments tightly enough that it’s acceptable to assume build consistency isn’t an issue, but loosely enough that the lockfile is of practical use. That’s the discussion I really think we should be focusing on now, rather than “Poetry-style” locking.
That’s one approach, but we typically want to spell out how tools verify things to provide better error messages. There are a couple of ways of tackling that specific problem, and so the question is whether any PEP I write should be the one to solve it, and regardless of that which approach to take?
Correct, as well as ask why lock files have this specific requirement and the rest of the Python ecosystem doesn’t?
There’s a reason I left them out 3 years ago. I think most people who need sdist support want a best effort solution, not one that’s perfect.
That’s a non-starter based on past history of this discussion.
generate the metadata from the sdist when generating (required anyway)
hash that metadata (the important bits anyway)
store the hash in the lockfile
verify that hash when installing
And if a package fails to match its hash, error explicitly saying “Package X v123 has changed its metadata since this lockfile was generated; sdists with non-reproducible metadata cannot be locked.”
I would think same machine near the same time (so discovery of external libraries and tools is consistent), and same environment variables (as typically they’re used to configure builds) would produce the same metadata is a sensible assumption.
I think though there are two separate sdist cases:
One is old (probably pure-python) projects that never produced wheels, but are either consistent, or have metadata that can be represented via markers (e.g. depending on additional packages on windows, or on old python versions); a wheel only lockfile cannot handle these, and the ideal solution would be a minor update to fix up the package metadata
The other is where projects depend on other python projects, but where the requirements change depending on how the package is built, such as different configuration options, or if a different version of a package is used when building (i.e. Depending on packages for which an ABI matters - pypackaging-native). The latter case seems like it can be handled by Brett’s current proposal, and the former could be handled by tracking what the environment/configuration was on building the lock file (this may not make the lock files cross platform, as you may need a different set of values on different platforms, but would at least reduce the variation down somewhat).
@brettcannon Would you be open to adding to the lock.sdist (and I think also the lock.git) an additional field environment (or configuration) which would hold the “relevant” key/value pairs (how to get this is a separate issue—maybe there should be an array somewhere in the pyproject.toml where projects can set what environment variables configure their project). Then builders using the lack file can configure the build environment to match the environment variables?
If you have that level of specific need I think you should either build the sdist separately or have the locker leave out the sdist build requirements and not lock them. Otherwise I feel like this just heads down a path where every single build requirement is going to get shoved into the lock file format and that feels never-ending.
By the way, this sounds reasonable to me: lockers can make assumptions about how sdists will behave (or can refuse to do so), and installers can choose to accept those assumptions or not.
There exists a current “lock” file with sdist support. pip-compile lock files are fine to include hashes with sdist and my experience is it generally works out. I have had installs fail on inconsistent hashes due to non-reproducible builds, but I consider that necessary tradeoff given nature of sdists. My main reason for wanting sdists is to deal with dependencies published to pypi/similar that only have sdists today and no wheels. Most of my dependencies have wheels and I am happy for that. But when a library has 1 transitive dependency that happens to be an sdist, if there’s no support at all for that, it quickly makes lock files not feasible for me to use.
Environment question in background here is especially hard when some notable packages (pytorch) have dependency metadata change in ways that are currently outside all of python environment markers. In pytorch case one very significant question is what does GPU/cuda library look like. And there is no python packaging metadata today for GPU. Nor do I think we should block lockfiles on that when space of Cuda/similar non-python dependency relations is very large.
So I’d lean to be lenient on lockers regarding sdists and have installers only be responsible for checking hashes. Same requirement pip install -r requirements.txt on a file with hashes present today has.
I personally prebuild wheels and upload them to my devpi instance (and use multiple indices as needed), so I’d be happy with a wheel-only lock file. My concern is projects not providing accurate metadata (which was done in the early days of wheels, so I wrote scripts to add in missing requirements such as numpy), because they’ve been forced to in order to ensure “consistent” metadata because lockers have made assumptions (as seems to be the issue with poetry and pytorch).
It feels to me like consensus is forming and that largely Brett’s original proposal solves the pure strict lockfile scenarios for specific environments and through all the long discussions can probably be expanded to support a pragmatic “good enough, but not 100% guaranteed” solution for the rest.
I’m largely left with some questions / topics which I’ve not seen discussed explicitly in this thread. This is largely the crux of it.
My impression is that a PEP and capital “S” Standard is for interoperability. If I’m wrong, please let me know. So this implies a contract / interface between some Locker and Installer. Some tools may combine these, but they should also be able to be purely separate in my opinion.
So, then what are we agreeing between Lockers and Installers here? Are we saying that an Installer MUST be able to install a lockfile from ANY Locker? I think we’ve determined that’s not possible without other trade-offs. I think the trade-offs can be acknowledged, but even when they are, I still think there are scenarios where behaviour isn’t defined. I think it would be helpful to clarify these.
I’ve tried to capture some scenarios in a table to describe what I mean. I started with a Truth Table, but in the end I couldn’t simplify things enough so it’s just a table of scenarios that I think is complete enough to illustrate my points.
Definitions
LT1 = A Locker Tool that only produces environment lockfiles for one or more target environments
LT2 = A Locker Tool that only produces package version constraint lockfiles (meta: is this relevant for the scope of the PEP?)
LT3 = A Locker Tool that is able to produce both environment lockfiles for one or more target environments and package version constraint lockfiles
IT1 = An Installer Tool that can only install from environment lockfiles for one or more target environments
IT2 = An Installer Tool that can only install from package version constraint lockfiles (includes some style of SAT solver or similar dynamic decision making) (meta: is this relevant for the scope of the PEP?)
IT3 = An Installer Tool that can install from both environment lockfiles for one or more target environments and package version constraint lockfiles (includes some style of SAT solver or similar dynamic decision making)
Scenario
Description
Locker Tool
Installer Tool
Target Environment
Result
1
LT1 produced an environment lockfile for a single target environment E1. IT1 is given the lockfile and executed on E1.
LT1
IT1
E1
IT1 MUST succeed and install the exact packages that LT1 generated for that E1 (* sdist callouts apply). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch. Fails fast at locker time if a valid solution does not exist for E1.
2
LT1 produced an environment lockfile for a single target environment E1. IT1 is given the lockfile and executed on E2.
LT1
IT1
E2
IT1 MUST refuse to install. E2 != E1. Don’t panic yet, see rest of table before complaining!
3
LT1 produced an environment lockfile for multiple target environments {E1, E2}. IT1 is given the lockfile and executed on E2.
LT1
IT1
E2
IT1 MUST succeed and install the exact packages that LT1 generated for that E2 (* sdist callouts apply). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch. Fails fast at locker time if a valid solution does not exist for both {E1, E2}.
4
LT1 produced an environment lockfile for multiple target environments {E1, E2}. IT1 is given the lockfile and executed on E3.
LT1
IT1
E3
IT1 MUST refuse to install. E3 not in {E1, E2}. Don’t panic yet, see rest of table before complaining!
5
LT2 produced a package version constraint lockfile for a specific python version. IT1 is given the lockfile and executed on E1.
LT2
IT1
E1
IT1 MUST refuse to install. IT1 ONLY supports environment lockfiles. Don’t panic yet, see rest of table before complaining!
6
LT2 produced a package version constraint lockfile for a specific python version. IT2 is given the lockfile and executed on E1.
LT2
IT2
E1
IT2 MAY succeed, and in this scenario it does (* sdist callouts apply). It MUST only install distributions, including build distributions captured in the lockfile (via hashes). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch.
7
LT2 produced a package version constraint lockfile for a specific python version. IT2 is given the lockfile and executed on E2.
LT2
IT2
E2
IT2 MAY succeed, but in this scenario it fails. IT2 is unable to satisfy the installation constraints for E2 using the distributions (incl. build) captured in the lockfile. This late failure MAY occur because a package version constraint lockfile can’t guarantee that it has captured valid solutions for all possible environments.
8
LT2 produced a package version constraint lockfile for a specific python version. IT3 is given the lockfile and executed on E1.
LT2
IT3
E1
IT3 MAY succeed, but in this scenario it fails. This scenario exists to contrast against scenario 6. IT3 is unable to satisfy the constraints for E1 because it uses a different algorithm to IT2 to select the best candidates for installation and in this scenario, fails to find a solution.
9
LT3 produced an environment lockfile for a single target environment E1 AND a package version constraint lockfile for a specific python version. IT1 is given the lockfile and executed on E1.
LT3
IT1
E1
IT1 MUST succeed and install the exact packages that LT1 generated for that E1 (* sdist callouts apply). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch. Fails fast at locker time if a valid solution does not exist for E1.
10
LT3 produced an environment lockfile for multiple target environments {E1, E2} AND a package version constraint lockfile for a specific python version. IT1 is given the lockfile and executed on E2.
LT3
IT1
E2
IT1 MUST succeed and install the exact packages that LT1 generated for that E2 (* sdist callouts apply). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch. Fails fast at locker time if a valid solution does not exist for both {E1, E2}.
11
LT3 produced an environment lockfile for multiple target environments {E1, E2} AND a package version constraint lockfile for a specific python version. IT1 is given the lockfile and executed on E3.
LT3
IT1
E3
IT1 MUST refuse to install. IT1 ONLY supports environment lockfiles. Don’t panic yet, see rest of table before complaining!
12
LT3 produced an environment lockfile for multiple target environments {E1, E2} AND a package version constraint lockfile for a specific python version. IT2 is given the lockfile and executed on E3.
LT3
IT2
E3
IT2 MAY succeed, and in this scenario it does (* sdist callouts apply). It MUST only install distributions, including build distributions captured in the lockfile (via hashes). A failure here is either a corrupt lockfile or a sdist and Target Environment mismatch.
13
LT3 produced an environment lockfile for multiple target environments {E1, E2} AND a package version constraint lockfile for a specific python version. IT2 is given the lockfile and executed on E4.
LT3
IT2
E4
IT2 MAY succeed, but in this scenario it fails. IT2 is unable to satisfy the installation constraints for E4 using the distributions (incl. build) captured in the lockfile. This late failure MAY occur because a package version constraint lockfile can’t guarantee that it has captured valid solutions for all possible environments.
14
LT3 produced an environment lockfile for multiple target environments {E1, E2} AND a package version constraint lockfile for a specific python version. IT3 is given the lockfile and executed on E3.
LT3
IT3
E3
IT3 MAY succeed, but in this scenario it fails. This scenario exists to contrast against scenario 12. IT3 is unable to satisfy the constraints for E3 because it uses a different algorithm to IT2 to select the best candidates for installation and in this scenario, fails to find a solution.
Essentially are we going to require specific capabilities from Lockers and Installers, or can they be free to support only parts of it? What are the guarantees that all Lockers and all Installers must be able to interoperate? Is there freedom in their implementations (support for sdist, support cross platform, support for different resolution algorithms, etc) to give different results? I hope so, otherwise this isn’t a Standard, we’re just specifying that all tools need to behave identically everywhere and we then just are requiring a single tool to be used for the entire ecosystem which is an entirely different discussion.
Are we saying that:
ALL Locker tools MUST be able to produce environment lockfiles for specific environment(s). Where valid solutions exist and within their ability to support sdist.
ALL Installer tools MUST be able to install environment lockfiles for specific environment(s). Within their ability to support sdist.
The environment lockfiles for specific environment(s) produced by different Locker tools MAY be different. Some may choose to exclude sdist, some may prioritise lower versions, etc.
The same environment lockfiles supplied to different Installer tools MUST have the same result when run on the same environment within their ability to support sdist.
Some Locker tools MAY be able to produce package version constraint files for a specified python version. Where valid solutions exist and within their ability to support sdist.
Some Installer tools MAY be able to install package version constraint files for a specified python version. Within their ability to support sdist.
The package version constraint files produced by different Locker tools MAY be different. Some may choose to exclude sdist, some may have different resolution algorithms, etc.
I have to mention that the most popular packages (no rigorous stats, but I’m under the impression) are not using Metadata Version 2.2, even they have static dependencies.
Maybe someone could make a query on that.
(apologies if this has already been mentioned, I’ve just started catching up with the discussion here)
For non-static metadata from sdists, IMO it is sufficient to encode the dependencies that were used for the lockfile generation within the lock file and require installers to detect deviations and error out.
This would mean we’d shift the goal post: instead of needing METADATA declared with the right flags set by a new enough backend, we’d only care whether the package generates consistent dependency information (i.e. observable behaviours).
It removes the need to be blocked on wider METADATA 2.2 adoption (which, is gonna take quite a lot of time) and enables actually being able to generate such a lock file today for a typical environment where there’s no weird dependency (as well as with older packages, which is very valuable).
These questions capture the key capability I’m proposing by always including a [[targets]] list in the lock file rather than only putting the full environment markers directly on the individual files: it narrows the install-time problem down to “find the first matching entry in the [[targets]] list, then install all the nominated artifacts for the chosen target”.
It’s always unambiguous at install-time (order in the target list resolves any otherwise ambiguous cases, one of the options Alice noted later in the post I quoted), and it allows the specification to evolve over time by allowing for more target types:
comprehensive (narrow definition): exactly one artifact tagged for every distribution to be installed on the target. If every tagged artifact is a wheel, installer just needs to be able to retrieve and unpack zip archives to the appropriate location. If the installer isn’t explicitly told which target to install for, it needs to support environment marker evaluation in order to find the first matching entry in the target list.
partial at the artifact level (or a more pragmatic definition for a comprehensive lock): exactly one version tagged for every distribution to be installed on the target, but potentially multiple artifacts (e.g. progressively older manylinux wheels, or allowing an sdist/vcs build from source fallback). Only additional installer requirement is to check artifact platform tags against the current environment and fail if the version is tagged for the target, but none of the tagged artifacts are suitable for the installation environment (or the only matching artifact requires a source build, and the requested installation mode is binary-only).
partial at the version level: multiple versions may be tagged for some distributions to be installed on the target. Installer needs a full resolution capability to ensure a consistent set of dependencies is installed from the available candidates.
This mostly corresponds with Brett’s notions of “environment locks” (first case), “version constraints”, and “shrunken world view”, but splits the latter two a bit differently by moving any scheme that allows multiple versions for a single distribution to the final category, leaving only the single-version-multiple-artifacts case in the second category. No matter how the flexibility is expressed in the file format, having to choose between multiple versions at installation time is the trigger for needing a resolver, as you otherwise risk getting an installation where one or more declared dependencies isn’t being respected properly (i.e. a broken installation instead of a failed installation, which is the one thing we want to ensure is categorically ruled out).
Only artifact selection can reasonably be performed without affecting the consistency of the dependency set, and even that relies on the locker enforcing the single version metadata consistency rules that Brett described earlier in the thread.
The discussion so far has convinced me that the initial version of the format at least needs to support the first two cases, since it’s hard to usefully lock for Linux or macOS without accepting multiple candidate wheels (due to either manylinux evolution over time, or the variety of macOS CPU architectures). It’s necessary enough that I’m inclined to consider “partial at the artifact level” to still be a comprehensive lock (it’s the kind of lock that pip-compile generates) rather than trying to treat these two categories as separate things. I suspect this is also the reason why that “lock to exactly one artifact hash” issue in pip-compile has failed to gain any real traction over the years: attempting to lock to exactly one artifact rather than a set of them turns out to disable installer behaviour that’s genuinely beneficial.
The last “consider multiple versions” case is more about ensuring that the chosen format doesn’t preclude adding that capability later rather than being essential to defining a useful first iteration of a lockfile spec. I don’t like the idea of having to designate an entire lock file as “this is a comprehensive lock file” vs “this is a partial lock file”, since that’s something that varies by target, not by the set of input requirements (hence the recurring example in the thread of including comprehensive locks for specified targets, together with a partial lock or locks for particular Python versions as a fallback for other platforms)
(Tangent: I had been conflating Poetry & PDM as both falling into the third category, but Brett’s format review made me realise that the way PDM works is closer to the way pip-compile works, just with extra metadata in the lock file that indicates why there are multiple artifacts recorded against a given distribution package)
I also think “deduplicate information to improve readability” is well worth considering, which would mean storing a structured distributions->versions->artifacts tree (similar to the way package repositories are set up) rather than simply storing a flat list of artifacts.
Beyond the above points, I don’t think the exact spelling of the detailed file list actually matters that much, as long as installers get the information they need and likely auditing rules (like “all defined targets environments will use the same version of each distribution that they install”) can be reasonably implemented via an automated check. Being able to easily check things by eye is a nice-to-have, but far from essential. Again, including a [[targets]] list is helpful here, since any auditing checks won’t have to care about how installers choose the nominal target for a given environment, they’re just enforcing rules about the mapping between target names and distributions. Without a [[targets]] list (or an equivalent), auditing checks have to somehow infer why the different entries in the detailed file list exist before they can start enforcing restrictions on them.
For the “efficient relocking” and “relocking without the original input file” information, I’m wondering if those should be left in the [tool] table for now, as the important information to capture in the standard format is the info that installers need to install a consistent set of artifacts for a given lock. Do we really care if pip-tools can’t easily regenerate a lockfile originally produced by PDM, or vice-versa? Metadata hashes to check lockfile freshness would fall into the same category. “Different installers install the same packages for a given lock file” is an important requirement. “Different lockers generate the same lockfile for a given dependency set” doesn’t fall into the same category, since there are all sorts of reasons why they might give different answers (e.g. even the same locker may produce a different result based on which package indexes it is told to query).
The one question I would have on that front, is whether the entries for individual items should have [tool] tables, so tools can make per-distribution, per-version, and per-artifact notes without having to replicate the entire artifact tree. (Allowing this space for tools to make per-distribution and per-version notes is one of the reasons I’d like to see a structured heirarchy over a flat list of artifacts)
Something else that I think has emerged from the discussion, but hasn’t necessarily been made clear to folks following along more casually: when multiple lock files exist, they should reflect different input requirement sets, rather than different target environments. So having parallel locks like pylock.deploy.toml, pylock.dev.toml, pylock.docs.toml would be expected, but having pylock.linux.toml, pylock.windows.toml, pylock.macos.toml would be weird.
I did that this past week, and other folks made most of the points I would have made. Thoroughly recommended
Finishing with a smaller side topic:
Single-hash-per-file makes it painful to evolve the hash scheme if it needs to change to something new as a result of attacks being found against older hashes. With the ability to record multiple hashes per file, it’s straightforward to include an older hash for backwards compatibility with older installers, and a more future-proof hash for tools that support it. Similarly, installers can have a preference list of hashes, and use the first one in their preference list that the locker provided.
When the locker and installer are the same tool, that’s less of a concern (since it’s more reasonable to say “locking and deployment should use the same version of the tool” in that case).
For the benefit of locker implementers, it might be worth offering the following two suggestions:
locker picks an algorithm (e.g. SHA256), indepependently verifies the published artifact hash by downloading the artifact and hashing it, includes only that hash in the lockfile
locker trusts the hash metadata published by the index server, filters out known-flawed hashes (e.g. MD5, SHA1), then includes the remaining hashes in the lockfile
Lockers that are downloading aritfacts anyway could readily use the first option, while those relying on the repository metadata API would need to use the second option.
Quoting this twice, since my first response was more abstract, but I also wanted to work through exactly how this would look if using a [[targets]] list together with a hierarchical tree of distributions.
Rich publishes only a source tarball and a single pure Python py3-none-any wheel.
To simplify things, I’m going to consider a lockfile installing an example package that has a single example_dependency; python_version <= "3.9" dependency rather than considering Rich itself.
Note that this version of the format sketch includes a separate optional source subtable that can hold either a VCS or sdist description, together with an optional list of wheel references, rather than the single combined artifacts list that I used for each version in my original sketch. I also realised that we can use the more widely known “packages” term in this context, since we’re talking about both distribution packages and installation packages, so the disambiguating nature of the “distributions” term isn’t needed.
Case 1: only lock for recent Python versions
[[targets]]
name = "recent_python"
markers = [ "python_version >= 3.9" ]
[[common_packages]]
name = "example"
# ... maybe other distribution level info? Perhaps a `tools` subtable?
[[common_packages.versions]]
version = "0.1"
# ... maybe other version level info? Perhaps a `tools` subtable?
[common_packages.versions.sdist]
origin = "https://<path to host download location>/example-0.1.tar.gz"
# Details of the sdist reference for this version (and potentially a `tools` subtable)
# Lockers would omit this field to ensure only binary-only installs were allowed
# This field may be replaced by a VCS reference if no sdist is published anywhere
[[common_packages.versions.wheels]]
origin = "https://<path to host download location>/example-0.1-py3-none-any.whl"
targets = [ "recent_python" ]
# Details of this wheel artifact (and potentially a `tools` subtable)
# example_dependency doesn't appear in the file at all, since no defined target needs it
Case 2: lock for all Python versions
[[targets]]
name = "recent_python"
markers = [ "python_version >= 3.9" ]
[[targets]]
name = "older_python"
markers = [ "python_version < 3.9" ]
[[common_packages]]
name = "example"
# ...other info as discussed above
[[common_packages.versions]]
version = "0.1"
# ...other info as discussed above
[common_packages.versions.sdist]
origin = "https://<path to host download location>/example-0.1.tar.gz"
# ...other info as discussed above
[[common_packages.versions.wheels]]
origin = "https://<path to host download location>/example-0.1-py3-none-any.whl"
targets = [ "recent_python", "older_python" ]
# ...other info as discussed above
[[conditional_packages]]
name = "example_dependency"
targets = [ "older_python" ] # Note: not included for "recent_python" targets
# ...other info as for common packages
[[conditional_packages.versions]]
version = "1.0"
targets = [ "older_python" ]
# ...other info as for common packages
[conditional_packages.versions.sdist]
origin = "https://<path to host download location>/example_dependency-1.0.tar.gz"
# ...other info as for common packages
[[conditional_packages.versions.wheels]]
origin = "https://<path to host download location>/example_dependency-1.0-py3-none-any.whl"
targets = [ "older_python" ]
# ...other info as for common packages
I think this example illustrates the key benefit of defining a [[targets]] list as part of the format spec: it allows either projects or a sufficiently smart locker to partition the space of potential installation environments according to the distinctions that matter for that particular project rather than attempting to define a generic target selection scheme that would work for any arbitrary project. Since the latter problem appears to be intractable, I consider this to be a genuinely useful simplification of the problem space.
If a clean partitioning isn’t feasible, then installations using that lockfile won’t be supported on some platforms (unless the format spec includes the partial locking support that would allow multiple versions to be tagged against the same target environment).
Package sources are missing (per package). (You don’t see them in Poetry’s own lock file because PyPI is the default, which is not recorded, and Poetry itself does not use other sources.) If it’s not PyPI, we record the source type (index/url/git/file/directory), the url and (if relevant) the reference, resolved reference and subdirectory.
IIRC, it’s just to display descriptions of locked packages when running poetry show ... (convenient for users). If it was not in the lock file (and the cache), we would have to query the index and the command would take longer.
Sure, but the discussion about PEP 665 made it clear people wanted to also record what was used to build the sdist.
Depends on the which parts; some will be required, some will be optional.
That will be outlined in the PEP.
Playing devil’s advocate, in that scenario why couldn’t you simply run a script to update your hashes to a new value? The hash algorithm is encoded in the string in both PDM and Poetry, so I don’t think updating to a new algorithm would be a massive burden since you will need the tools to support thew new hash to begin with anyway.
How big of a problem is that, really? And I don’t mean it as a rhetorical question; I don’t recall seeing many (maybe any) pyproject.toml files where dependencies, or even optional dependencies, where dynamic. So, in real terms, how big of a deal is excluding sdists where dependencies are not static, including setup.py-based ones?
With the lock format standardised, lockers won’t know the capabilities of the installers up front.
However, I realised after your comment that locker config options to force a particular hash algorithm would be enough to solve that problem, since the project maintainers will know which installers they care about and can override the locker’s default hash choice if necessary.
Given that, +1 for the format simplification to just have a single top level hash algorithm field and record artifact hashes as a simple string rather than allowing multiple hashes per artifact.
Are there good arguments against including a hash of the contents of project.requires-python, project.dependencies, and project.optional-dependencies.* from the pyproject.toml file? Seems very useful to have.
)