Pre-PEP: Include pylock.toml files inside wheels

Packaging
20

I wouldn’t call the timing of this coincidental. Abstractions matter, and standardized lockfiles give us a better ability to discuss these topics than we had before the standard was accepted just days-- er, months ago!

The original phrasing of this thread presents itself as a standards-track discussion. I think that’s premature. We should establish a shared notion of what users want and need before trying to build out a specific solution.

And I have some fundamental concerns about this specific proposal. It mixes an environment specification (pylock.toml) with the contents of an environment (distributions/wheels). So it’s “inside-out” – the thing which goes “inside” constains a spec for the thing which is on the “outside”.

I think that inversion leads to a number of problems. For example:

  • How would locking work with this? Would pip-compile or a locker have to unpack dists and read their contents, rather than just their metadata?
  • What should a locker (e.g., uv lock) do when it finds a “nested lockfile” while trying to update, not freshly generate, a lock?
  • How should tools behave when installing packages which have conflicting lockfiles? What if those packages have compatible dependencies? What if they don’t?
  • Should tools ever ignore nested lockfiles? Does that match the intent of the maintainer who put a lockfile into their distribution?
  • When updating/reinstalling a package with a lockfile, are tools expected to do any differential analysis of the lockfiles from before and after the update?

I have confidence in the (often amazing!) abilities of our community of developers. I’m sure we could make it work, but if the design is wrong, the end result will be worse for users than finding a more suitable design.

And if “making it work” really is good enough, why don’t we try that before writing a standard?
I see three relatively simple ways that installation of a locked application via a lockfile could be supported by tools today:

  • Bundle a pylock.toml in your package as in-package file data (like py.typed, if you need an example) and make a tool which uses that file
  • Use VCS/HTTPS installs from a source repo, and make a tool which looks for pylock.toml in the repo root
  • Provide a Project-URL with a link to the associated pylock.toml (served/hosted separately)[1]

You can do this with a completely new tool, a pipx or uv fork, or by wrapping those tools.

(EDIT: I just noticed that the OP mentioned putting it in dist-info above, even without a standard, since the spec technically doesn’t forbid you from including extra files. I think that’s reading the spec a little too loose, since it does reserve rights to all subdirectories of dist-info, meaning it reserves pylock.toml/. I’d rather not play with fire and just not touch that dir.)

This caught my eye because I think it’s also interesting to ask what community-owned distribution channels we should have. Should there be a Python community container repo? A zipapp repo? If PyBI work resumes and gets accepted (ref), where would that go?

PyPI is being used to distribute applications today. If we want to shift the balance of usage away from that, we need to provide or describe an alternative (read: superior) way to deliver the tools which are distributed that way. And that includes tools like mypy/pyright/pytype which may interact with the contents of a user-controlled Python environment. So that sounds like a package, unless we have some new ideas about how to distribute those tools.

I consider the status quo to be: “PyPI is a valid distribution channel for applications, if those applications are happy with the benefits and limitations inherent in being a Python package.”

This proposal is trying to alter what the tradeoffs are for a Python package.
That’s fine, but it’s not the only degree of freedom which we have as a community: as an example, we could allow packages to serve a “recommended lock” alongside their dists on PyPI.

But of course, any proposal needs to be well grounded and meet the usual standard of quality for community adoption. I think we’re having the early discussion, before it’s even clear if anything should be done.

  1. maybe dicey for security, but no worse than a setup.py ↩︎

3 Likes