FWIW, to me this proposal is pretty clear about all of these questions:
It wouldn’t interact with locking, pip-compile shouldn’t look at lock files, it’s resolving libraries for an application, not installing an application
Nothing, dependencies should be treated as libraries and their lock files ignored
Follow the lock file specification, i.e. follow the lock file and don’t try to resolve the dependencies
There should no concept of “nested lock files”, the lock file is for when you are the one primary tool or application, if you are a dependency your lock file isn’t read
Follow the lock file specification or the default behavior the installer handles this with regular lock files
In practice, probably something similar was being achieved by pinning exact versions in a package’s requirements, even if that is explicitly discouraged.
At my organisation, we currently get there by using a mixture of a lockfile, a dynamic pyproject field and a build plugin (hatch-requirements-txt); we export a uv.lock file to requirements.txt, and that plugin is responsible for translating it into the Requires-Dist entries in the wheel metadata. This is because we wanted to support less technical users doing data-science work, ensuring consistency between their local setup and the production environment; just installing our in-house packages is enough to guarantee that.
While that’s a strictly internal use-case, there are definitely cases where libraries or applications for public distribution could also benefit from the guarantees of locking. For example, applications doing numerical analysis probably want to pin dependencies to ensure consistency and reproducibility of results (libraries like scikit-learn for example don’t guarantee reproducible models between minor version changes).
I think what locking offers over just pinning exact dependencies, besides just side-stepping the resolve step, is that they can also include security-related metadata like hashes and annotations.
Edit: Delete line where I hedged way too much about security based on the phrase “However, this PEP does not solve all potential security concerns.”; thank you to @pradyunsg for the clarification.
The suggestions from Stephen and Paul seem like a promising way to start experimenting with the possibilities in the tooling space before diving into writing a full standard… It could also be a great way to gather feedback and assess what works best.
I’m trying to fully understand the concerns raised, particularly in the light of Stephen’s and Paul’s proposals. It feels like some of these could potentially be addressed within their suggested approaches.
Regarding these requirements, my thoughts are the following:
writing another package manager
In principle it is not necessary to write another package manager. Instead, there are 2 options (a) writing a tool that wraps existing package managers or (b) collaborating with maintainers of current tools to incorporate this use case. I imagine a similar effort would be needed even with a PEP.
and it’s service counterpart
If we look at Stephen’s comment, using PyPI would be a direct reuse, similar to the proposed PEP, so there might not be a significant difference in that regard. With Paul’s idea, modern code repositories like GitHub already offer somewhat decent ways of distributing files (either viw raw URLs or release artifacts), potentially negating the need for a completely new service.
teach users how to use it
My understanding is that users would need to learn specific conventions or standards regardless of whether we pursue a PEP or a tool-based approach. The teaching effort might be somewhat equivalent across all these options. Users would need to be aware of how a “cli call” is translated to a “installation/execution procedure” in either a potential standard or simply a tool-specific convention, as well as how to write/distribute them.
define what a release it and how to sync it with the wheel package release
Again, Stephen’s suggestion could leverage PyPI, just like the proposed PEP. For Paul’s concept, GitHub Actions is a good example of how to use platforms like GitHub to specify release versions — this information can often be embedded directly in the URL or via shortcut syntaxes like actions/setup-python@v5.
Here is a toy example of a (not necessarily good) way of parsing that information:
import os
import sys
import argparse
from urllib.parse import urlparse
SHORTCUT = {
"gh": ("https://raw.githubusercontent.com/{repo}/refs/heads/{ref}/pylock.toml", "main"),
"github": ("https://raw.githubusercontent.com/{repo}/refs/heads/{ref}/pylock.toml", "main"),
"bitbucket": ("https://bitbucket.org/{repo}/raw/{ref}/pylock.toml", "main"),
"heptapod": ("https://foxx.heptapod.net/{repo}/-/raw/{ref}/pylock.toml", "branch/main"),
"file": ("{repo}", ""),
"http": ("{repo}", ""),
"https": ("{repo}", ""),
}
DEFAULT = "github"
def locator(file: str) -> str:
if os.path.isfile(file):
return file
if os.path.isdir(file):
return os.path.join(file, "pylock.toml")
if file.startswith(("./", "../", "/", f".{os.sep}", f"..{os.sep}", os.sep)):
raise FileNotFoundError(file)
parts = urlparse(file)
template, default_ref = SHORTCUT[parts.scheme or DEFAULT]
repo, _, ref = file.partition("@")
return template.format(repo=repo, ref=ref or default_ref)
def parse(args: list[str]) -> tuple[str, str]:
"""
>>> parse(['hello/world'])
('https://raw.githubusercontent.com/hello/world/refs/heads/main/pylock.toml', 'world')
>>> parse(['hello/world@v5'])
('https://raw.githubusercontent.com/hello/world/refs/heads/v5/pylock.toml', 'world')
"""
parser = argparse.ArgumentParser()
parser.add_argument("file", help="URL where to find the pylock.toml file")
parser.add_argument("-e", "--entrypoint", help="Name of the entrypoint to run (must be in the `console_scripts`or `gui_scripts` group)")
params = parser.parse_args(args)
directory, _, _ = params.file.replace("pylock.toml", "").partition("@")
entrypoint = params.entrypoint or os.path.basename(directory)
address = locator(params.file)
return (address, entrypoint)
(A real implementation could expand on that and actually use APIs instead of template URLs). The advantage of this approach is that it actually increases the possibilities of how to distribute applications (and side steps historical limitations of PyPI like name collisions).
Good security defaults is definitely a thing that pylock.toml was explicitly designed for. From the rationale in the PEP:
The file format should promote good security defaults.
Yes, and that’s because it’s an explicitly unsolved problem (as already discussed above).
I share that assessment.
I also feel like we should not solve this problem in the standards until we have a non-standard solution for this problem – namely, if uvx or pipx or some other tool implements the handling of the lock file.
To be very clear, a lockfile encodes specific environments and IMO it doesn’t need to be coupled strongly with a wheel to be able to install your custom package. In fact, you don’t even need the lockfile to end up on PyPI, if you instead have it be something like “fetch the lockfile from URL, install the lockfile’s environment, run cmd in the environment” without being coupled with the wheel file or whatnot.
After reading some of the replies, I realized the discussion has shifted more toward the topic of application deployment.
Because of that, I’ve rewritten the original post (please have another look before continuing the discussion), starting more strongly from the use cases and clearly limiting the scope.
I’m also considering reaching out to uv and pipx about supporting lock files included within the package folder itself, not just in the dist-info directory.
(Though to clarify: while there can be multiple package folders, the standard does not forbid placing files in the dist-info folder — only subdirectories are reserved.)
What I still don’t fully understand is: How would a completely optional standard harm the packaging ecosystem?
I genuinely don’t have your depth of experience, so I’d appreciate any concrete examples or past situations where optional features have caused harm — especially if there’s a story/example you could share. That would really help me understand your concerns better.
From my perspective (as detailed in the revised proposal), there are several strong reasons to include lock files inside wheels:
Security & reliability: No dependency on external services during installation
Reproducibility: As the build system includes the lock file used to pass tests tests pass, for the most pipelines it is ensured they will work.
Simplicity: It’s easy to associate a lock file with the package it belongs to, and tools (as welll as humans) can find and use it without needing additional APIs, tokens, or infrastructure
So far, the only downsides I’ve seen mentioned are:
The confusion between “wheels” and “applications” (which I’ve tried to address more clearly in the rewritten proposal)
Slightly increased wheel size
If I’ve missed any additional concerns about including a lock file in the wheel, or if there are others you’d like to raise, I’d really appreciate if you could point me to them.
P.S. I’m a slow writer — the initial and revised versions of the proposal took me more than 10 hours in total — so I may only be able to follow up or respond toward the end of the week. Thanks for your patience!
Please don’t do this. It makes the discussion confusing for anyone reading in future as now the replies are written as replies to something that isn’t there any more. If the proposal is to be completely rewritten then it is better to start a new thread.
I still think that the arguments made in are very weak. They’re phrased quite persuatively (which LLMs are good at) but they’re not particularly strong arguments.
The need to trust an additional source of truth is not a problem for most users - they’d be willing to trust GitHub or an internal-to-my-company artifact and the fact that certain URLs will be mutable is arguably a feature.
I will revert the changes and start a new thread (and edit this message to point to the new thread). @pradyunsg please copy your answer to the new thread once it opened.