You could publish a package <pkg>-app that has pinned dependencies, and then tell people to install that instead of <pkg> if they intend to use it as an app?
(And if that works well, then we could consider some metadata on <pkg> that points to <pkg>-app for any tooling that wants to reinterpret the context on behalf of the user.)
How would I upload the additional py.lock for <pkg>-app if PyPI / Warehouse does not support storing such a file?
This approach works for the āembed py.lock inside the wheelā strategy, but not for the alternative approach of publishing a separate sidecar *.whl.lock file next to the original wheel ā potentially also exposed via the simple repository JSON project detail, similar to core-metadata.
To clarify my previous post:
The idea is to publish a py.lock as a sidecar artifact at {file_url}.lock, analogous to {file_url}.metadata / core-metadata, so installers can optionally consume it.
My question is mainly about process: how could such a feature be introduced without first having a full PEP?
For example, would it be acceptable to add support in Warehouse for uploading a py.lock sidecar file associated with an existing wheel ā without changing the Simple Repository API initially ā so installers could attempt to fetch the lock file when explicitly instructed?
This way, usage would remain completely optional and experimental.
Would that be an acceptable way to go forward from here?
I donāt think there is a way as once itās on Warehouse people will come to rely on it. The PEP will have to demonstrate a desire/need for the feature to motivate it to be served from an index or shipped inside a wheel file.
You can have an entrypoint script for your app that verifies the parts needed for reproducibility prior to then launching the app.
Itās inelegant, but it allows sidestepping the chicken-egg problem for the version that includes the lockfile in the wheel as a means of demonstrating demand prior to standardization allowing a more elegant version.
It seems like youāre fixated on a particular solution, rather than thinking about just solving your problem.
Wheels/sdists have dependency metadata. By convention, that is unpinned in libraries, but itās entirely possible to just pin it and publish a regular wheel (either metadata-only, or perhaps you put the CLI in a separate package). When someone installs it, the other packages willmatch exactly what it requires, reproducing the dependencies.
If youāre insistent on creating something different, then you really need to set it up and demonstrate it, before arguing that it should be merged into the existing tool (where it becomes incredibly difficult to change or remove, as Brett says). You donāt need a PEP to set up your own index, or to encourage tools to support your index (though some may want it to be a community standard before theyāll support it - that just means you havenāt encouraged them enough ).
Or you can use an approach that already exists, already works, doesnāt require anyone to change anything more drastic than the one install command that currently doesnāt do the thing they want it to do. Iām not forcing anything, Iām just ignoring your solution and offering you a much easier way to solve your problem.
A possibly interesting point of prior art here, is that cargo install has a --locked flag that will use the Cargo.lock from the the thing that is being installed instead of using the unlocked dependencies in Cargo.toml.
That ecosystem is different from ours though. IIRC cargo install can only install a single āthingā, and itās use case is pretty much entirely installing a CLI into the system, and itās not part of the regular day to day functionality of writing a Rust project and adding/removing dependencies. cargo install is closer in spirit to something like pipx install.
In python pip/uv/etc install typically accepts multiple things, and it more closely related to something like cargo add than cargo install.
Instead of doing that, you have to somehow build a package with pinned dependencies in its METADATA, there are many ways of doing that. Not exhaustively:
setup.py where the install_requires= is generated from the contents of a requirements.txt file with pinned dependencies. That file can be generated with pip-compile or uv pip compile, for example.
Coming from the Java world, where dependency resolution (as performed by Maven) is both reproducible and composable, I often feel frustrated that many other dependency management ecosystems lack both of these features simultaneously. The tradeoff is typically framed as reproducibility (lockfiles, āfor applicationsā) versus composability (loose versioning, āfor librariesā), but it does not have to be this way. I finally got around to writing a blog post about it:
The Python community is being continuously hurt by the toolingās inability to deliver reproducible and composable environments intuitively by default. I hope threads like this one can lead to some consensus on how to fill this gap.
I publish a few applications (and libraries) on pypi and I canāt count the hours Iāve been wrestling and analyzing issues with dependencies. Some say the real reason docker has been invented was because python packaging is such a mess and every time I stumble on a dependency issue I (again) think of yanking my applications from pypi and publish docker images only.
Maybe Iām not smart enough or donāt understand everything correctly but it would be really nice if there was a straightforward method of pinning dependencies (and sub-dependencies) which result in reproducible installations per python version.
I also teach python and and all the questions I get asked after my courses are (exclusively) package installation related, e.g. dependencies that not longer work because of newer sub-dependencies, python version mismatch, sometimes missing build dependencies.
In theory you are right - itās entirely possible to write a dependency file which pins all dependencies and sub dependencies.
But letās say I have a dependency A, which has version A1 for python 3.12, A2 for python 3.13 and A3 for python 3.14. It also has a sub-dependency version SA1 for python 3.12, SA2 for python 3.13 and SA3 for python3.14. Iām also supporting 3.11 and 3.10 because there is no real reason not to but on my machine Iām developing with 3.12.
There is no easy way to generate an exhaustive list of pinned dependencies across python versions AND supply them as package metadata in a standardized way so they are properly used when a user installs my application. And even if I managed to do it (manually) once, updating these dependencies is again very difficult.
Itās very confusing to me that I need seven additional tools and three custom scripts for this simple use case (numbers made up).
All I want is that the initial installation is done with the dependencies I tested.
You should publish a lockfile. Unfortunately, the ecosystem support for lockfiles is still developing, so there are a bunch of problems with that recommendation right now, but we are getting there. The problem is ājustā implementation, though, not lack of a solution.
Python packaging is much stronger around publishing libraries rather than applications. Thatās a known weakness, unfortunately. But we are working on it.
I tried that, but itās impossible to install e.g. from a uv/poetry lock file with pip.
Itās also unclear to me how the user would use the data from the lock file for installation.
Where could I read up on this?
Iām aware and thatās awesome!
I just wanted to give additional perspective on the pain points and point out that from my perspective packaging seems to be one of the bigger issues - especially for newcomers.
I donāt believe uv or Poetry use standard lockfiles as their native format. Youād probably have to export to the standard format if you wanted a tool-independent solution. I donāt know if they have plans to switch to standard lockfiles as their native format. As I said, itās early days yet.
On the pip tracker, weāre talking about a pip sync command, or maybe pip install -r pylock.toml. But itās still very much at the āwork out what the UI would look likeā stage. And I have no idea what commands uv or Poetry would have to install from a standard lockfile - youād have to ask them.
The definitive information is the lockfile spec but itās intentionally low on UI details, because the UI is up to individual tools to decide, and most tools are (I believe) still working on that. Thereās also a question around whether PyPI would allow publishing lockfiles, which is again still at the āthinking about itā stage, I believe.
Understood, and thanks for that Progress is slow, but itās happening - lockfiles took many years to standardise, and will likely take a few more years to implement, but they are a key piece of functionality, and should make things a lot better for application distribution (among other things).
Itās interesting that the three ecosystems you mention as scoring well (JVM, .NET, go) all struggle to integrate with other ecosystems, whereas the other ecosystems have traditionally been good at integration (and conda-forgeās main purpose is multi-ecosystem integration).