They can still be used, you’d just need to specify the starting location in the dependency graph as part of the installation command (in addition to specifying the lockfile to use). It makes the lockfile more like a stricter pip
constraints file (where it specifies the set of what’s allowed to be installed) than like a requirements file (which specifies the set of what must be installed).
Yes, sorry - I immediately afterwards realized what you had actually meant and then tried to erase the evidence of my stupidity
Correct.
No, it’s the opposite; the installer would know upfront that an install would fail because some dependency would be missing in the lock file. Basically you start at your root(s), figure out if you have what you need, and if you don’t have everything you fail and otherwise you download and install the files.
Yep, and I am treating it as a requirement. The question I was asking was how to expose it in the lock file? What I’m planning on doing is the most straightforward, but it could be considered noisy if a project has a ton of dependencies but that the lock file only really needed to say to the user, “you can’t install this for this environment” in some other manner.
You could if there’s a single root. In that instance you can just infer that’s what someone would want everything from the root down that applies to their environment.
Depends on what you think of pip figuring out if there’s an obvious root in the dependency graph? And what do you do if there are multiple roots (e.g., PEP 735)?
Yep. The single root scenario can be inferred, but not the multiple roots one.
It depends on how hard of a stance we want to take that pyproject.toml
is the source of metadata? Right now there isn’t a rigid relationship, just a presumed one that it’s how tools will work. That makes it more of an installer question. But if we had a hard connection then you could have lock files record the path to the pyproject.toml
file that was used as input. And that’s ignoring alternative metadata files from other tooling where the dependencies may be recorded in a completely different file.
But they have a requirements.in
file? Otherwise if you’re hand-writing your requirements.txt
file then that’s more analogous to pyproject.toml
.
This isn’t a question of burden as much as whether it’s useful and what UX we are expecting for folks? And what is the default?
And that’s what makes having a default tricky. Toss in PEP 735 and it gets even messier if you don’t have a [project]
table to act as an implied root.
As an example, let’s take an example of the project spam
that has
You can delete your post if you want.
From reading this feedback I think the “should lock files work on their own?” depends on whether we think they should be tied to pyproject.toml
or not? If yes then that solves this issue. But if we don’t, then we either have to say goodbye to having separate lockers and installers (since in order to know where to get the metadata of what the roots might be and their logical names we need coordination between locking and installing), or it has to be recorded in the lock file itself. I suspect @anon62990384 and Pants prefer not to require pyproject.toml
.
I think I have a design for recording the roots that I like, but I will have to write it out to see. It might also negate the need for supporting multiple lock files and allow for a single pylock.toml
, but I have to think through the edge cases on that still to make sure what I have in the PEP covers all the cases that necessitated supporting separate files.
I don’t like the idea of pip (and other installers) “figuring it out”. I’d rather read it from the lockfile. Of course, there’s a big range of possibilities between those two, and I’m not clear how hard you expect it to be to “figure it out”…
Fail unless the user specifies a root, I guess. But then the question becomes what is a “root”? I’d be OK with something like pip install --lockfile pylock.toml --root=dev
where a “root” is an explicitly named starting point in the lockfile (although I’d prefer a better name than “root” for this). I would not be happy with pip install --lockfile pylock.toml <list of requirements>
, where the lockfile is effectively a very strict form of constraints file.
I’m looking at this from the POV of a tool (pip) and a user of that tool who’s been handed a lockfile but who may not routinely use workflow managers and hence not know what terms like “root”, “workspace”, “sync”, etc., mean. As an interoperability standard, lockfiles should be usable as a means for those users to interact with people who use a tool like uv
to manage their environments.
That brings us back to the question of whether a lockfile is a standalone specification of an environment (i.e., it can be shipped to someone without any other data) or it’s a component of a project (linked to pyproject.toml
).
IMO there are compelling use cases for both, so we have to find some way of either having a spec that handles both, or we have to create two different lockfile formats. I don’t think it’s out of the question to have a lockfile format that records within itself which type it is (that could be a key that records the location of pyproject.toml
, with the case where that key’s missing being the standalone case). But I think the UI for workflow managers would need careful consideration, as they would use project-based lockfiles internally, and standalone lockfiles would essentially be nothing more than an export format.
To be clear, I’d see the priority for pip to be support for standalone lockfiles. I don’t see an advantage for pip in acting as a competitor to workflow tools when it comes to managing projects - if you have a uv-managed project, use uv to install from the project lockfile.
Something with a logical name, e.g., the name of a dependency group.
I think such a tie likely makes sense for lock files with multiple roots intended to help manage local development environments, but not for standalone single root lock files intended for deployment.
That said, if you can come up with a way to handle multi-root lock files without limiting the input format to pyproject.toml
, that’s probably still a neater solution.
I would say yes, definitely. The reason is that it would be nice to be able to use lockfiles when there is no “project” in the pyproject.toml sense.
As I’ve mentioned before, the grain of the Python packaging system is pretty strongly in the “project-first” or “package-first” direction: that is, it is oriented towards a setup where you start by thinking “I want to make a project that does X”, and then you set up some scaffolding for that, and then you put your code into that scaffolding. One thing I’d hope for from lock files (but that I continue to hope for in other tooling as well ) is a “project-last” workflow in which the first step is writing code that does X. That code may be developed iteratively, gradually extended, cleaned up, and so on. It may be shared along the way in an amorphous state which is not intended for installation as a fully-fledged package, but where you still want to give someone a clean way to get it up and running. Then at the end, as part of the release process, you would wrap the core code in the necessary metadata and so on to release it as a package.
Conceptually, a lock file fits that model for me better than many existing Python packaging tools and standards, because it suggests that you could just lock “how things are” in an environment and thus enable someone else to recreate the environment, without needing to install the package under development as such. Although I admit I’ve lost track of a few of the twists and turns this proposal has taken, I have a sense that, on independent grounds, it may not meet the goal I describe.[1] Still, yes, I’d say it’s good for lock files to be self-contained.
For instance, it’s still unclear to me whether there is agreement that the lockfile locks a total environment state and not “a set of packages” or “a project” or some other thing. ↩︎
This would require PEP 710 in order to have sufficient information to create a lockfile from just an environment, but I agree that it should be a goal of the PEP to support this.
This somewhat extends my view of the ways in which pip would support lockfiles:
pip install --lockfile=pylock.toml
. A general installer, supporting single-root lockfiles, and maybe supporting multi-root lockfiles (depending on how “roots” end up being specified).pip install --dry-run --report pylock.toml --report-format=lockfile
. Create a lockfile that captures what apip install
run would do.pip freeze --format=lockfile
. Once PEP 710 lands, record the exact state of an environment as a lockfile.
Items (2) and (3) could be implemented as 3rd party tools (and in my view, probably should, at least initially, both so that their utility can be proven before they are added to pip, and so that they aren’t blocked by pip maintainer resource availability).
I want to offer an alternative vision for how the non-package workflows should work with locking. I don’t mean that locking an existing environment isn’t valid – it’s just not the workflow I would aim to guide people towards.
I expect it to start pretty early on with a pyproject.toml, but ideally without needing the [project]
table. i.e.
[dependency-groups]
main = ["numpy", "matplotlib"]
(that’s the whole file)
And then your workflow tools support you in your work:
$ $tool install main
Locking main to pylock file... ok
Creating .venv virtualenv... ok
Installing main from pylock... ok
$ . .venv/bin/activate
The lockfile would help you locally reproduce the environment, and would be available for redistribution when you want to share your work.
Because I see the non-package workflow in this way, I don’t think that this case demands a strong separation between pyproject.toml and the lock.
I also think it’s interesting that this suggests a single-entry-point lockfile where that entry-point is not [project.dependencies]
.
(All of this is predicated on Dependency Groups, of course!)
Hatch wouldn’t use this if that was the source/a new requirement. Arbitrary user-defined environments are the source, as Paul mentioned here:
I think this is similar to how uv currently manages non-package “applications” with pyproject.toml, by (sort of unofficially) using the lack of a [build-system]
key to signal it shouldn’t try to install your project, just the dependencies.
(Apologies for being slow to respond here. I’ve been traveling and am now at a Rust conference for the next few days.)
I personally think it’s useful for the lockfile to be useable as a standalone artifact. We don’t quite support it in uv today, but we could and perhaps should (even if we don’t need to read from it, we rely on the pyproject.toml
to find the project “root”).
I have a solution for this that I’m generally happy with, so assume the major update that’s coming has lock files work standalone.
New thread at PEP 751: now with graphs! .