I hope not, and I can’t imagine why (apart from the naming conflict over “lockfile”). But as I’ve said, I’m totally okay with referencing files within the lockfile’s own scope (directory) without a hash/version, and I’m okay with tools adding additional metadata provided it isn’t necessary for a trivial “restore” operation.
I don’t think editable mode is a normal enough thing to specify it in a lockfile, not without elevating it to a core Python feature (i.e. language feature), and I don’t see that happening. But if someone wants to say “the environment should also include ./” instead of them making ./ into a real package, then I would prefer to stop them by advice (and policy, if they work for me) rather than specification
Then that’s happening outside of an installer already, so I don’t see how that project’s code participates beyond you somehow generating a list of dependencies to lock against.
It depends on scope because things like uv’s workspaces (I believe) need projects listed when they are a root in the dependency graph in case they are relied upon in some other way. It also lets you lock for extras for a project since it’s represented in the lock file ATM.
Yeah, so it’s basically what the state of things were in mid-September and before this entire topic (and to be frank, roll back changes made for uv’s benefit).
Correct, it would either be optional or entirely left out. Pro of optional is if you want an editable version to line up to detect changes and thus maybe signal you need to recreate the lock file, con is you would then need UX to make the version optional sometimes.
But if the lock file is already resolved then why do installers care about what version a source tree says it is? I can see if you want some extra stale check, but it isn’t required from a lock file perspective, just like requirements files don’t know when they are out of date today. It’s definitely a nice-to-have
It could, but for flexibility to allow for uv to use this format with their workspace concept it lead to listing even the project itself so it was locked for itself and potentially as part of a larger, monorepo-like lock. It also allows for representing extras cleanly since the package itself is listed.
See above as to why this concept was pulled in. I believe Poetry also asked for editable install support (along with the lock file using source locations to be part of specifying what’s ); @radoering can correct me if I’m misremembering.
I’ll let them answer for themselves, but @charliermarsh already said uv won’t use the PEP for their canonical lock file format and just as an export target without this sort of support (he listed everything they support in PEP 751: now with graphs! - #55 by charliermarsh).
I kept going back and forth about trying to ask questions about bigger picture stuff, but having been thinking about this for well over 5 years, I’m waiting until we either all agree on what to do with this editable install topic and whether that’s going to necessitate rewriting the PEP again (and even going back to an older version is a rewrite as I’m sure it’s going to require plenty of updates). And if we do go around yet again, we will be having a very serious, time-limited discussion on scope of lock files that we agree to and we will not being changing later on; I don’t know if I can rewrite this PEP more than one more time w/o coming out of this whole experience extremely bitter (but I will see this through thanks to work supporting me getting this done).
Yes I totally own this. I said at the time that I thought the right thing to do was push for a single format that could replace uv.lock, poetry.lock, etc., but also that it would be difficult. We’re now in the phase where we’re grappling with a lot of that difficulty and I’m feeling it. I know it must be a frustrating to hear me push in different directions and I’m sorry for that. I haven’t given up on this approach yet though.
I think part of my hesitancy here is that I feel like I’ve been asking for a lot of changes that are motivated almost entirely by how uv works. Is that good? I don’t know – maybe not? I like our lockfile and our concepts but I’m also wary of coupling a standard to a tool. Do other people want these changes? I feel the same way when others come in and ask for changes that are motivated by their own tool. I find myself worrying that the proposal will continue grow in scope to try and accommodate specific tools, which could be a sign that it’s (maybe?) too closely coupled to the tools themselves?
Yeah, I just don’t see how we could use a lockfile that didn’t support source trees. Which, of course, gets back to the question of what we’re trying to design and accomplish here. If we want to come up with a format to replace uv.lock, poetry.lock, et al, then we have to support this stuff. If we want to come up with a format to enable installer interoperability, Dependabot interoperability, etc., then maybe we don’t? Although I’d still argue that we should. Even as an installer-only format, people want to install a project along with its locked dependencies. Why should the project itself be excluded?
I feel like this is going to end up in two outcomes:
tools will continue to use their own lock files, some people will export a python.minlock (made up name), some will not. The main use will be tools like dependabot being able to utilize it. Over time one of the many lock files will win out and we standardize on it.
we will not standardize any form of lock file for another year or two until the ecosystem will naturally figure things out.
I tend to think that 1 is probably better than 2, but presumably it means that every tool in the space will gain ways to read/export these lock files which does not with extra complexity for the user, and some users might not even see the benefits of it over the native lock files.
A third possible outcome is that we go back to the most fundamental question leading to this discussion - “given a collection of files presumed to be a Python project, how do I install its dependencies so I can develop/run it?” - and come up with some standard way for projects to specify what tool they require and how it should be invoked.
And yes, this is much the same as a number of other areas (build backends, task runners, etc.), because they’re all fundamentally the same problem. We don’t particularly care what the tool is, provided it’s easy to find out how to run it and get the required environment back. Like in the other cases, the front-end here is not pip or uv pip, but more likely an IDE, a CI system or a tox-like runner.
Definitely worth trying to align on specifying the actual lock file first, but if that’s going to collapse, I think a lot of the demand would be satisfied by a standard way to confirm whether the project has a uv file or a Poetry file etc.
I know this is to a certain extent just dumping the problem on someone else, but surely that problem can be relatively easily solved by someone writing a tool that introspected a project, using whatever messy heuristics they wanted, determined the “project type” from that, and then presented a standard interface for running the project-specific command to “make an appropriate project environment”?
The point here being that we don’t need a standard to provide a unified interface to execute a particular task. It can be done by a 3rd party wrapper. And if maintaining that wrapper becomes too much work, we can then take the lessons learned from doing so and design a standard around that.
The reason that same approach doesn’t work for lockfiles themselves is that there are many things people want to do with lockfiles (build an environment, audit them, generate reports, etc). So the lockfile standard is a genuine interoperability exercise, whereas a “build an environment” tool is more of a unification exercise…
I think this is drifting off-topic, though. It’s clear[1] that a lockfile standard would be beneficial, even if it has to be of limited scope. It’s much less clear that there’s a useful “project management protocol”[2]. But if we want to discuss the latter, it should be a topic of its own.
I don’t think we should be too negative here. Brett has put a lot of effort into this, and I think we’re very close to a solution. Maybe we’ll have to scale back on the idea of letting tools use the standard as their internal format, but the desire for lockfiles has been around since long before uv, PDM and Poetry developed their approaches, and even something that only replaced the basic “locked, hashed, requirements file” approach that’s been around for years would benefit a lot of people.
I also want to point out that no time or effort was wasted here. Even knowing just the scope of the problem is incredibly helpful because it tells us a lot. It also has shown that other than getting hung up in details. We have learned a lot. We understand that this is a solvable problem, there are some disagreements or unknowns on the edges. At this point comes down to having to make decisions and tradeoffs and maybe narrowing that scope down which just takes time.
From where I’m standing this is mostly just a question of time and we will arrive there eventually.
Yeah, the analogy is that this is roughly how requirements.txt is used today from the perspective of tools like uv and Poetry. You can “export” to requirements.txt, then use that format with pip, or the uv pip CLI, or anything else that operates on a requirements.txt. To some degree it already accomplishes the goals of an interoperable format… but the format itself is not standardized and limited in ways that we would want to solve here.
I don’t want to be too negative either! I am still open to and hoping that we standardize a format here that can replace uv.lock. I do think we would benefit from more alignment on what we’re trying to accomplish before we continue iterating on the format itself. If the goal is to replace uv.lock, poetry.lock, etc., then we should keep pushing in this direction. But I think there’s disagreement about whether that’s a critical outcome.
Just to share openly, from my perspective, I think there were two things that made me question whether it can / should happen:
I felt like I kept coming back and asking for more things, as motivated by how uv works. To some degree this is the system working as intended (I should play the role of advocate for the uv users), but I couldn’t tell if anyone else wanted this stuff. Then I saw others coming in with other requests based on their own workflows and based on other tools, and I was worried that the proposal was expanding too much in scope and coupling itself too closely to the tools (uv included). My general attitude has been: if we’re going to pursue a standard, I desperately want the standard to work for uv, so I’m going to push for the things we need. Am I right to be pushing like this? I need feedback (genuinely).
I couldn’t come up with a clear answer as to what the expectations for implementers should be and, relatedly, what the CLI would look like. The answers felt like they either were too prescriptive for tools or made the standard less valuable.
My go-to example for this is: a user wants to install a package along with one of its dependency groups. What does the standard say about this? Does the standard say that installers have to enable this workflow? Does it prescribe a specific API? Does that mean that all resolvers need to use the same [[group]] representation within the format itself? (This would then suggest that we get rid of [[group]] in favor of a set of first-class concepts.) Maybe we say that an installer “just” needs to be able to install a [[group]] by name. Then, for uv at least, we’d leave all of our uv APIs as-is, and probably add some separate API to “install a group by name”, to facilitate installs from other lockfiles. Is that a good outcome? I’m not sure.
Maybe to help anchor the question of what we’re trying to accomplish here, I’ll share an anecdote from a company I spoke with recently. They have a central team that supports Python infrastructure, and that team promotes a Happy Path set of packaging tools, but teams across the company can choose to use other tools if they want. So there are teams using uv, pip, Poetry, Conda, etc.
That central infrastructure team would like to be able to build unified tooling that can operate on all of those different projects. For example, they might want to ship a CLI that can install any of those projects. Or a CLI that can analyze the dependencies of any of those projects. Regardless of whether the underlying projects are using uv, or vanilla pip, or Poetry, or whatever. So what would a standard look like that supports this?
It also feels responsible for me to mention that there are a number of issues in uv that I’ve put on pause as we figure out the standard. In the past week, they include:
Dynamic versioning (mentioned here after that discussion in the uv repo)
Locking of build dependencies (very common request)
I’ve been hesitant to solve these problems in uv (and in uv.lock) without knowing how they’d be impacted by the PEP (and there’s some world in which we won’t be able to solve these problems in uv in the future without amending the standard itself – maybe tool.uv can solve that though). I don’t know what this means exactly – maybe, that there are still unsolved problems here? That the space is still in flux? – but I thought I would share it.
As a Python packaging user, my humble answer is yes. Many of us are really excited about the cohesion of workflow that uv is bringing (something we’ve hoped for for a long time after experiencing other ecosystems like Rust). Thank you for advocating for us.
Personally[1], I think it would be a good outcome, but it’s not critical.
I agree that we need more alignment. But from my perspective, the alignment that’s lacking is a common understanding of what locking is and how it works from the tools implementing it (uv, Poetry and PDM). The discussions feel to me like they are pulling in multiple directions because the workflow tools don’t have a shared understanding or common goal, and that’s concerning because it suggests we could end up with a format that’s either a compromise that satisfies no-one, or an over-complicated mess because it tries to cover everything.
Absolutely! I think a standard like this needs input from tool makers. But at least as importantly, it needs user representation, and the tool makers are a proxy for the users in that regard. And if the tool makers don’t have a common view, I’m worried that they aren’t distinguishing their needs from those of their users. That’s why it’s hard to decide what needs to be in the standard and what can be handled by the [tool] section.
IMO, nothing. That’s purely a matter of installer user interface, and it has nothing whatsoever to do with locking. You don’t even mention the term “lock” in the statement of what the user wants.
Standards are never going to say what tools must or must not enable. All a standard will say is if a tool wants to do such-and-such thing, it should do so in the following way. Where the “way” is stated in terms of APIs and file formats, but not in terns of UI.
There’s no API here. This standard is about a file format. Sorry, I know you know this, I just don’t quite understand what you’re thinking when you ask this question.
Dependency groups are specified in their own standard. Again, I don’t understand the relevance here. If you’re asking how a user requests that a tool creates a lockfile that will install the group and the package, that’s down to the tool UI, but the user would expect to state the dependency group name and “please include the package as well”. I feel like that’s obvious so again I think I’m missing something.
This is the first time you’ve mentioned installing from lockfiles. For me (speaking as a user) I don’t see a lockfile as something I’d pick bits from like this. I’d either install the lockfile or not. So none of your questions really make sense to me because I don’t see a lockfile as a “set of things to pick what to install from”. Instead, I’d see the correct workflow as me requesting the tool to build a lockfile that represents the install request I want to support (“install the project and dependency group X”), and then requesting the same (or some other) tool to install that lockfile.
Does Poetry have this idea of “picking a chunk of a lockfile to install”? Does PDM? What are the user workflows that need this functionality? Unless it’s something that all of the lockers acknowledge a need for, backed by actual user requirements, I’d say this isn’t something we should be worrying about for a lockfile standard. And if that means that you have to support this UI in uv by including additional information in the [tool] section, along with not supporting the feature for non-uv lockfiles, then that seems perfectly reasonable to me. No-one is suggesting that the lockfile standard should limit what uv offers its users, but conversely we shouldn’t be building something unique to uv into our standards.
It would support only the common subset of functionality that all of uv, pip, Poetry, conda, etc., support. That pretty likely means it would only support building an environment that contains exactly what’s in the lockfile (because you included pip and maybe conda in there, so something with the functionality of requirement files is essentially the lowest common denominator).
That might mean that teams using uv have to avoid certain uv features when publishing work for other teams to use, but that’s a consequence of their policy, not of the lockfile standard. After all, that same policy would prohibit a conda-using team from using a dependency that’s only available in conda, and not from PyPI, which is pretty much the same type of limitation.
Have you looked at how PDM and/or Poetry solve these issues? If there’s a common view on the “right” solution, that’s useful input to the lockfile discussion. If none of you have a solution, I’d strongly advise that we don’t try to solve them here, as that’s the worst sort of “design by standardisation” mistake. Much better to let tools explore possible solutions without the burden of a standard limiting their options.
and I can’t speak for Brett here, and I’m not speaking as PEP delegate so Brett shouldn’t feel constrained by what I am saying - disclaimers over now ↩︎
Yeah, all of uv, Poetry, and PDM support installs that are parameterized by extras and dependency groups.
For example, Poetry:
# Install the docs group, along with the project.
poetry install --with docs
# Install only the dev group, not the project itself.
poetry install --only dev
# Install the project with these extras enabled.
poetry install --extras "mysql pgsql"
# Install the project with all extras enabled.
poetry install --all-extras
Or uv:
# Install the docs group, along with the project.
uv sync --group docs
# Install only the dev group, not the project itself.
uv sync --only-group dev
# Install the project with these extras enabled.
uv sync --extra mysql --extra pgsql
# Install the project with all extras enabled.
uv sync --all-extras
The idea is that you want to lock your project along with its dependencies, optional dependencies, and dependency groups (e.g., development dependencies). Then you may want to install a subset of them depending on what you’re doing: linting your project, running tests, deploying to production, etc.
The relevance is that I’m trying to understand what we want to get out of interoperability. If each tool uses the standard format, but none of them understand how to install from it, is that okay? This assumes that installers need to be able to install subsets of the lockfile, as illustrated above. If each tool uses a slightly different representation for the lockfile internally (e.g., different naming convention for each [[group]] or something like that – my example was uv using package~group as the name for the development dependency [[group]]), then they have to translate the CLI into those conventions. Otherwise, how can uv support running uv sync --group docs on a Poetry-generated lockfile? If we want to support that level of interoperability, then we need to formalize more of the concepts in the schema (like a package with its associated groups).
No tool has an advantage. By which I mean, a project shouldn’t feel that they should choose a particular tool because it’s better supported by dependabot, or by VS Code, or whatever. That doesn’t necessarily mean that the tool must use the standard lockfile as its native format, just that it should be able to interact easily with services, IDEs and other tools that support the standard format.
Switching tools should be relatively simple. I don’t think lockfiles matter for this, though, as I’d absolutely expect that if I switched workflow tools, I’d relock my project as part of the transition process. But being able to export a lockfile from the old tool and import it into the new one could be useful, if you want to avoid relocking.
Things I consider not to be goals include:
Using multiple tools at the same time on a project. Project maintainers should decide on their workflow as part of setting things up for the project, and that means picking a tool and committing to it. They may choose to support external contributors using different tools, but it’s OK if that takes a bit of extra work. Again, export/import of a lockfile might be useful, but being able to run Poetry on a PDM-managed project isn’t what I mean by interoperability.
I’m unclear on this, as I don’t really know whether uv sync --group docs or poetry install --only dev should be considered a “workflow task” (and hence something that only maintainers do, which means it’s fine if it’s tool-specific) or if it’s something that an external user of a project might want to do (meaning the project needs to make it possible in a tool-agnostic way).
OK, but do we want to support that? I don’t - as I said above, using different tools on the same project simultaneously feels like it’s going way beyond my concept of interoperability. We don’t require every build backend to be able to build every project, so why should we require that of workflow tools?
If you think that level of interoperability is what people want, do you have evidence of that? You mentioned the users who have a “happy path” set of tools, while allowing teams to choose different ones if they want. But I honestly doubt that even they would expect to support a team using both PDM and uv on the same project, at the same time. That feels totally unsustainable to me, and if they did allow that, it would feel like weak management to me, to be honest.
Okay this is super helpful because it highlights some of the different assumptions we’ve been making.
My assumption had been that we did want to enable an installer to install from a lockfile produced by another tool in a first-class way. IIRC there were comments very early on about PDM users who wanted to use pip to install their dependencies in CI or when deploying. And I could imagine users that want to use uv to install in CI or similar even when the project itself is managed by Poetry or DPM. But it sounds like you’re envisioning this as more of a special case? Or like… installers need to be able to install an entire lockfile, but not anything more nuanced?
Can you clarify what you mean by “external” user of the project? When is an external user installing a project’s dependencies?
That’s certainly the only scenario that I, personally, think makes sense. But I’m in the “lockfiles are environment definition files” camp. It’s you (and to a lesser extent @radoering for Poetry) that I recall saying that being able to install bits of a lockfile is important. But I’ve never fully understood the underlying use cases for this. And in particular I don’t see how it’s useful to anyone except a developer who’s working solely in uv (or Poetry, whatever his tool of choice is) on the project itself - and hence, not in an “interoperability” sense.
Anyone who isn’t a project maintainer, following the project development guidelines…
A web hosting service which, given a project repository URL, downloads that project, creates an environment based on the project lockfile, installs the project into that environment, and starts the project as a web service.
But essentially that’s my point - no-one other than project maintainers should care about “dev” dependencies, or “docs” dependencies. And project maintainers should be using the project’s chosen workflow tool, so interoperability doesn’t apply.
One possible exception might be something like readthedocs. They might download the project and install the docs dependency group. But again, I’d imagine that being done by creating a separate lockfile that captured the environment definition for the docs build. I don’t see any reason why the docs environment needs to be compatible with the “dev” environment, for example, which is typically the argument I’ve seen here for putting everything in one “do everything” lockfile.
I have a use case which requires that two tools be able to interact via the lockfile.
I would like to lock my dependency groups via a locker, and then install the locked versions with my test runner (tox).
In this context, I’m using tox as an installer. (It may be backed by pip in fact, but I don’t know if that’s relevant.)
Both the locker (e.g., uv) and the runner (e.g., tox) should be components I can swap out in this workflow, with appropriate adaptations.
I think this use case is representative of other users’ desires as well, given how I’ve seen pip-compile used. But it’s hard to be certain. This use case should not be read as overly constraining. For example, the way the named groups are produced in the lockfile is relevant to me, but doesn’t need to be prescribed by the spec. As long as the output of the locker can be documented by that tool, I’ll be able to use it as input to the runner.
So I don’t agree with the idea that the file shouldn’t support “multiple tools”, but notably I also don’t think perfect and seamless interoperability between distinct tools which can act as lockers is a goal. uv may be able to provide a better CLI by assuming that it was the locker used, and likewise for pdm and poetry. But “pure installers” like pip will need to be able to consume any locker’s output – possibly with a more verbose specification of which group to install.
Included in this is the notion that the lockfile is not a single “install plan” for a single environment, but that it covers multiple target environments for a project. I think having multiple environments covered by a single lock is implied but having a required filename.
Ah yes, I’d forgotten this. Previous iterations of the PEP allowed things like pylock.dev.toml. I still think that “one lockfile per project” is an unnecessary restriction (and my recollection is that hatch, in particular, wanted a lockfile per environment where a project could have many environments) but it’s not something I’m going to worry about given the more significant issues we have to deal with first.
OK, I’ll concede that if we stick with the one lockfile per project restriction, multiple environments per lockfile is necessary. Although I think that’s backwards - it’s the multiple environments per lockfile requirement that means it’s not worth having multiple lockfiles per project. And for what it’s worth, multiple environments per lockfile imposes another (IMO unwarranted) requirement, that everything is locked as one set - you can’t (for example) lock the docs group independently of the test group.
Yeah. Here are things people do with uv that rely on this, off the top of my head:
Install only the development dependencies, e.g., when linting or formatting a project.
Install only the documentation dependencies, e.g., when building documentation.
When building a Docker image, install a project’s dependencies without the installing project itself, to create a highly cacheable Docker layer.
In a workspace, install a single local package (like a library) so that you can run a command in that package (or test that package) without installing the other packages in the workspace (like an application that depends on that library).
It is valuable for these use-cases to share a single lockfile. For workspaces, I think this is obvious? When testing the library, you want to use the same transitive dependencies as when testing the application, which depends on the library. For the development tooling: even if the documentation dependencies are never installed alongside the project itself, most of the time, you’d prefer to use a similar dependency tree for the documentation as for the project itself (i.e., you wouldn’t really want to use two different patch versions of pydantic when one would suffice – it’s simpler and more efficient).
That is correct. Poetry supports to define editable path dependencies in the pyproject.toml and thus also includes them in the lock file. I haven’t used this myself yet, but if I remember correctly this is particularly useful for mono repo use cases.
Poetry does not support workspaces as uv does so it is indeed a bit simpler. In the following cases, a package from the lock file is not installed (flat package, no graph approach):
The resulting marker evaluates to false when inserting the values of the target environment. [1]
The package is not required by any dependency group the user requested to install. [2]
The package is only required by an extra of the root package, which is not requested. [3] Actually, this is (more or less) just a combination of 1. and 2. because extras are always in the main dependency group and can be expressed via the extra marker.
In theory, the lock file contains a solution for each possible environment. ↩︎
The lock file contains the packages required for all dependency groups so that you can install any combination of groups from the lock file. ↩︎
The lock file contains all extras so that you can install a combination of extras. ↩︎
