# PEP 751: now with graphs! **URL:** \ **Category:** Standards\ **Created:** [October 30, 2024, 10:10pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721 "2024-10-30T22:10:37Z")\ **Posts on this page:** 20\ **Page:** 7
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [November 18, 2024, 11:23pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/122 "2024-11-18T23:23:08Z")
> [@pf\_moore](#): > > Personally, I consider installing local source trees, _especially_ in editable mode, to be out of scope of what a lockfile should be for. I agree. > [@pf\_moore](#): > > But that’s basically where the debate currently lies - if we declare them out of scope, does that mean workflow tools like uv and Poetry won’t be able to use the standard lockfile format for what _they_ refer to as “lockfiles”? I hope not, and I can’t imagine why (apart from the naming conflict over “lockfile”). But as I’ve said, I’m totally okay with referencing files within the lockfile’s own scope (directory) without a hash/version, and I’m okay with tools adding additional metadata provided it isn’t necessary for a trivial “restore” operation. I don’t think editable mode is a normal enough thing to specify it in a lockfile, not without elevating it to a core Python feature (i.e. language feature), and I don’t see that happening. But if someone wants to say “the environment should also include `./`” instead of them making `./` into a real package, then I would prefer to stop them by advice (and policy, if they work for me) rather than specification 🙂 ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 19, 2024, 12:39am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/123 "2024-11-19T00:39:07Z")
> [@dhduvall](#): > > The CI build does a non-editable install of the project into the venv, and the venv is copied wholesale to `/app`. Then that’s happening outside of an installer already, so I don’t see how that project’s code participates beyond you somehow generating a list of dependencies to lock against. > [@prophile](#): > > If a decision about it can be made at the time of generating the lockfile, would it be enough for tools to just (“just”) allow not including the root project itself without other changes to the format? It depends on scope because things like uv’s workspaces (I believe) need projects listed when they are a root in the dependency graph in case they are relied upon in some other way. It also lets you lock for extras for a project since it’s represented in the lock file ATM. > [@charliermarsh](#): > > > [@brettcannon](#): > > > > So going back one revision of the PEP to the “set of packages” approach like PDM? > > Something like that, yeah. A flat list of packages, each with a marker to indicate when it should be included, would make sense to me. A lot of things get simpler if we narrow the scope – for example, I don’t think workspaces are relevant? (Users could just export a separate lock for each member.) So multiple entrypoints is gone, etc. Yeah, so it’s basically what the [state of things were in mid-September](https://github.com/python/peps/blob/40bbaaa84ace944f3085e59af1713b320c728a67/peps/pep-0751.rst) and before this entire topic (and to be frank, roll back changes made for uv’s benefit). > [@charliermarsh](#): > > > [@brettcannon](#): > > > > The other potential option is to stop thinking about source _code_ and instead think about source _locations_. > > Yeah, I think something like this is reasonable. IIUC, it’d mean that we wouldn’t necessarily record versions in the lockfile for source trees? Correct, it would either be optional or entirely left out. Pro of optional is if you want an editable version to line up to detect changes and thus maybe signal you need to recreate the lock file, con is you would then need UX to make the version optional sometimes. > [@charliermarsh](#): > > I think installers would still need to compute and “validate” versions at install time, since given a source tree that builds a package `foo`, it’s valid for other packages (even from remote registries) to declare requirements like `foo==1.0.0`. If some package in the lockfile requires `foo==1.0.0`, then the installer would need to error if the computed version of `foo` no longer satisfies the specifier. But if the lock file is already resolved then why do installers care about what version a source tree says it is? I can see if you want some extra stale check, but it isn’t required from a lock file perspective, just like requirements files don’t know when they are out of date today. It’s definitely a nice-to-have > [@pf\_moore](#): > > The specific case seems to me to be “lock everything but the project” and I don’t understand why that can’t be done with a lockfile that includes precisely that (everything but the project) plus a normal install of the project. It could, but for flexibility to allow for uv to use this format with their workspace concept it lead to listing even the project itself so it was locked for itself and potentially as part of a larger, monorepo-like lock. It also allows for representing extras cleanly since the package itself is listed. > [@pf\_moore](#): > > The canonical example here is when the “root project” is specified as a local source tree, to be installed in editable mode. Personally, I consider installing local source trees, _especially_ in editable mode, to be out of scope of what a lockfile should be for. See above as to why this concept was pulled in. I believe Poetry also asked for editable install support (along with the lock file using source locations to be part of specifying what’s ); @radoering can correct me if I’m misremembering. > [@pf\_moore](#): > > But that’s basically where the debate currently lies - if we declare them out of scope, does that mean workflow tools like uv and Poetry won’t be able to use the standard lockfile format for what _they_ refer to as “lockfiles”? I’ll let them answer for themselves, but @charliermarsh already said uv won’t use the PEP for their canonical lock file format and just as an export target without this sort of support (he listed everything they support in [PEP 751: now with graphs! - #55 by charliermarsh](https://discuss.python.org/t/pep-751-now-with-graphs/69721/55)). * * * I kept going back and forth about trying to ask questions about bigger picture stuff, but having been [thinking about this for well over 5 years](https://discuss.python.org/t/structured-exchangeable-lock-file-format-requirements-txt-2-0/876), I’m waiting until we either all agree on what to do with this editable install topic and whether that’s going to necessitate rewriting the PEP again (and even going back to an older version is a rewrite as I’m sure it’s going to require plenty of updates). And if we do go around yet again, we **will** be having a very serious, time-limited discussion on scope of lock files that we agree to and we will **not** being changing later on; I don’t know if I can rewrite this PEP more than one more time w/o coming out of this whole experience extremely bitter (but I will see this through thanks to work supporting me getting this done). ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 1:32am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/124 "2024-11-19T01:32:39Z")
> [@brettcannon](#): > > Yeah, so it’s basically what the [state of things were in mid-September](https://github.com/python/peps/blob/40bbaaa84ace944f3085e59af1713b320c728a67/peps/pep-0751.rst) and before this entire topic (and to be frank, roll back changes made for uv’s benefit). Yes I totally own this. I said at the time that I thought the right thing to do was push for a single format that could replace `uv.lock`, `poetry.lock`, etc., but also that it would be difficult. We’re now in the phase where we’re grappling with a lot of that difficulty and I’m feeling it. I know it must be a frustrating to hear me push in different directions and I’m sorry for that. I haven’t given up on this approach yet though. I think part of my hesitancy here is that I feel like I’ve been asking for a lot of changes that are motivated almost entirely by how uv works. Is that good? I don’t know – maybe not? I like our lockfile and our concepts but I’m also wary of coupling a standard to a tool. Do other people want these changes? I feel the same way when others come in and ask for changes that are motivated by their own tool. I find myself worrying that the proposal will continue grow in scope to try and accommodate specific tools, which could be a sign that it’s (maybe?) too closely coupled to the tools themselves? > [@pf\_moore](#): > > Personally, I consider installing local source trees, _especially_ in editable mode, to be out of scope of what a lockfile should be for. But that’s basically where the debate currently lies - if we declare them out of scope, does that mean workflow tools like uv and Poetry won’t be able to use the standard lockfile format for what _they_ refer to as “lockfiles”? Yeah, I just don’t see how we could use a lockfile that didn’t support source trees. Which, of course, gets back to the question of what we’re trying to design and accomplish here. If we want to come up with a format to replace `uv.lock`, `poetry.lock`, et al, then we have to support this stuff. If we want to come up with a format to enable installer interoperability, Dependabot interoperability, etc., then maybe we don’t? Although I’d still argue that we should. Even as an installer-only format, people want to install a project along with its locked dependencies. Why should the project itself be excluded? ---
**Author:** ![mitsuhiko](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mitsuhiko/32/12209_2.png) [@mitsuhiko](https://discuss.python.org/u/mitsuhiko)\ **Post date:** [November 19, 2024, 11:55am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/125 "2024-11-19T11:55:40Z")
I feel like this is going to end up in two outcomes: 1. tools will continue to use their own lock files, some people will export a `python.minlock` (made up name), some will not. The main use will be tools like dependabot being able to utilize it. Over time one of the many lock files will win out and we standardize on it. 2. we will not standardize any form of lock file for another year or two until the ecosystem will naturally figure things out. I tend to think that 1 is probably better than 2, but presumably it means that every tool in the space will gain ways to read/export these lock files which does not with extra complexity for the user, and some users might not even see the benefits of it over the native lock files. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [November 19, 2024, 12:25pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/126 "2024-11-19T12:25:14Z")
> [@mitsuhiko](#): > > I feel like this is going to end up in two outcomes: A third possible outcome is that we go back to the most fundamental question leading to this discussion - “given a collection of files presumed to be a Python project, how do I install its dependencies so I can develop/run it?” - and come up with some standard way for projects to specify what tool they require and how it should be invoked. And yes, this is much the same as a number of other areas (build backends, task runners, etc.), because they’re all fundamentally the same problem. We don’t particularly care _what_ the tool is, provided it’s easy to find out how to run it and get the required environment back. Like in the other cases, the front-end here is not `pip` or `uv pip`, but more likely an IDE, a CI system or a tox-like runner. Definitely worth trying to align on specifying the actual lock file first, but if that’s going to collapse, I think a lot of the demand would be satisfied by a standard way to confirm whether the project has a uv file or a Poetry file etc. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 19, 2024, 1:20pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/127 "2024-11-19T13:20:47Z")
> [@steve.dower](#): > > I think a lot of the demand would be satisfied by a standard way to confirm whether the project has a uv file or a Poetry file etc. I know this is to a certain extent just dumping the problem on someone else, but surely that problem can be relatively easily solved by someone writing a tool that introspected a project, using whatever messy heuristics they wanted, determined the “project type” from that, and then presented a standard interface for running the project-specific command to “make an appropriate project environment”? The point here being that we don’t _need_ a standard to provide a unified interface to execute a particular task. It can be done by a 3rd party wrapper. And if maintaining that wrapper becomes too much work, we can _then_ take the lessons learned from doing so and design a standard around that. The reason that same approach doesn’t work for lockfiles themselves is that there are many things people want to do with lockfiles (build an environment, audit them, generate reports, etc). So the lockfile standard is a genuine interoperability exercise, whereas a “build an environment” tool is more of a _unification_ exercise… I think this is drifting off-topic, though. It’s clear\[1\] that a lockfile standard would be beneficial, even if it has to be of limited scope. It’s much _less_ clear that there’s a useful “project management protocol”\[2\]. But if we want to discuss the latter, it should be a topic of its own. > [@steve.dower](#): > > if that’s going to collapse I don’t think we should be too negative here. Brett has put a _lot_ of effort into this, and I think we’re very close to a solution. Maybe we’ll have to scale back on the idea of letting tools use the standard as their _internal_ format, but the desire for lockfiles has been around since long before uv, PDM and Poetry developed their approaches, and even something that only replaced the basic “locked, hashed, requirements file” approach that’s been around for years would benefit a _lot_ of people. * * * 1. to me, at least 2. in the sense of PEP 517-style hooks ---
**Author:** ![mitsuhiko](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mitsuhiko/32/12209_2.png) [@mitsuhiko](https://discuss.python.org/u/mitsuhiko)\ **Post date:** [November 19, 2024, 2:04pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/128 "2024-11-19T14:04:46Z")
> [@pf\_moore](#): > > Brett has put a _lot_ of effort into this, and I think we’re very close to a solution. Maybe we’ll have to scale back on the idea of letting tools use the standard as their _internal_ format, but the desire for lockfiles has been around since long before uv, PDM and Poetry developed their approaches, and even something that only replaced the basic “locked, hashed, requirements file” approach that’s been around for years would benefit a _lot_ of people. I also want to point out that no time or effort was wasted here. Even knowing just the scope of the problem is incredibly helpful because it tells us a lot. It also has shown that other than getting hung up in details. We have learned a lot. We understand that this is a solvable problem, there are some disagreements or unknowns on the edges. At this point comes down to having to make decisions and tradeoffs and maybe narrowing that scope down which just takes time. From where I’m standing this is mostly just a question of time and we will arrive there eventually. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 2:38pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/129 "2024-11-19T14:38:11Z")
> [@mitsuhiko](#): > > tools will continue to use their own lock files, some people will export a `python.minlock` (made up name), some will not. Yeah, the analogy is that this is roughly how `requirements.txt` is used today from the perspective of tools like uv and Poetry. You can “export” to `requirements.txt`, then use that format with `pip`, or the `uv pip` CLI, or anything else that operates on a `requirements.txt`. To some degree it already accomplishes the goals of an interoperable format… but the format itself is not standardized and limited in ways that we would want to solve here. > [@pf\_moore](#): > > I don’t think we should be too negative here. I don’t want to be too negative either! I am still open to and hoping that we standardize a format here that can replace `uv.lock`. I do think we would benefit from more alignment on what we’re trying to accomplish before we continue iterating on the format itself. If the goal is to replace `uv.lock`, `poetry.lock`, etc., then we should keep pushing in this direction. But I think there’s disagreement about whether that’s a critical outcome. Just to share openly, from my perspective, I think there were two things that made me question whether it can / should happen: 1. I felt like I kept coming back and asking for more things, as motivated by how uv works. To some degree this is the system working as intended (I _should_ play the role of advocate for the uv users), but I couldn’t tell if anyone else wanted this stuff. Then I saw others coming in with other requests based on their own workflows and based on other tools, and I was worried that the proposal was expanding too much in scope and coupling itself too closely to the tools (uv included). My general attitude has been: if we’re going to pursue a standard, I desperately want the standard to work for uv, so I’m going to push for the things we need. Am I right to be pushing like this? I need feedback (genuinely). 2. I couldn’t come up with a clear answer as to what the expectations for implementers should be and, relatedly, what the CLI would look like. The answers felt like they either were too prescriptive for tools or made the standard less valuable. My go-to example for this is: a user wants to install a package along with one of its dependency groups. What does the standard say about this? Does the standard say that installers have to enable this workflow? Does it prescribe a specific API? Does that mean that all resolvers need to use the same `[[group]]` representation within the format itself? (This would then suggest that we get rid of `[[group]]` in favor of a set of first-class concepts.) Maybe we say that an installer “just” needs to be able to install a `[[group]]` by name. Then, for uv at least, we’d leave all of our uv APIs as-is, and probably add some separate API to “install a group by name”, to facilitate installs from other lockfiles. Is that a good outcome? I’m not sure. * * * Maybe to help anchor the question of what we’re trying to accomplish here, I’ll share an anecdote from a company I spoke with recently. They have a central team that supports Python infrastructure, and that team promotes a Happy Path set of packaging tools, but teams across the company can choose to use other tools if they want. So there are teams using uv, pip, Poetry, Conda, etc. That central infrastructure team would like to be able to build unified tooling that can operate on all of those different projects. For example, they might want to ship a CLI that can install _any_ of those projects. Or a CLI that can analyze the dependencies of _any_ of those projects. Regardless of whether the underlying projects are using uv, or vanilla pip, or Poetry, or whatever. So what would a standard look like that supports this? ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 2:50pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/130 "2024-11-19T14:50:55Z")
It also feels responsible for me to mention that there are a number of issues in uv that I’ve put on pause as we figure out the standard. In the past week, they include: 1. Dynamic versioning (mentioned here after that discussion in the uv repo) 2. Locking of build dependencies (very common request) I’ve been hesitant to solve these problems in uv (and in `uv.lock`) without knowing how they’d be impacted by the PEP (and there’s some world in which we won’t be _able_ to solve these problems in uv in the future without amending the standard itself – maybe `tool.uv` can solve that though). I don’t know what this means exactly – maybe, that there are still unsolved problems here? That the space is still in flux? – but I thought I would share it. ---
**Author:** ![johnthagen](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/johnthagen/32/3780_2.png) [@johnthagen](https://discuss.python.org/u/johnthagen)\ **Post date:** [November 19, 2024, 3:21pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/131 "2024-11-19T15:21:18Z")
> [@charliermarsh](#): > > Am I right to be pushing like this? I need feedback (genuinely). As a Python packaging _user_, my humble answer is _yes_. Many of us are really excited about the cohesion of workflow that `uv` is bringing (something we’ve hoped for for a long time after experiencing other ecosystems like Rust). Thank you for advocating for us. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 19, 2024, 4:08pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/132 "2024-11-19T16:08:04Z")
> [@charliermarsh](#): > > But I think there’s disagreement about whether that’s a critical outcome. Personally\[1\], I think it would be a good outcome, but it’s not critical. > [@charliermarsh](#): > > I do think we would benefit from more alignment on what we’re trying to accomplish before we continue iterating on the format itself I agree that we need more alignment. But from my perspective, the alignment that’s lacking is a common understanding of what locking is and how it works from the tools implementing it (uv, Poetry and PDM). The discussions feel to me like they are pulling in multiple directions because the workflow tools don’t have a shared understanding or common goal, and that’s concerning because it suggests we could end up with a format that’s either a compromise that satisfies no-one, or an over-complicated mess because it tries to cover everything. > [@charliermarsh](#): > > Am I right to be pushing like this? I need feedback (genuinely). Absolutely! I think a standard like this needs input from tool makers. But at least as importantly, it needs user representation, and the tool makers are a proxy for the users in that regard. And if the tool makers don’t have a common view, I’m worried that they aren’t distinguishing their needs from those of their users. That’s why it’s hard to decide what needs to be in the standard and what can be handled by the `[tool]` section. > [@charliermarsh](#): > > My go-to example for this is: a user wants to install a package along with one of its dependency groups. What does the standard say about this? IMO, nothing. That’s purely a matter of installer user interface, and it has nothing whatsoever to do with locking. You don’t even mention the term “lock” in the statement of what the user wants. > [@charliermarsh](#): > > Does the standard say that installers have to enable this workflow? Standards are never going to say what tools must or must not enable. All a standard will say is _if_ a tool wants to do such-and-such thing, it should do so in the following way. Where the “way” is stated in terms of APIs and file formats, but _not_ in terns of UI. > [@charliermarsh](#): > > Does it prescribe a specific API? There’s no API here. This standard is about a file format. Sorry, I know you know this, I just don’t quite understand what you’re thinking when you ask this question. > [@charliermarsh](#): > > Does that mean that all resolvers need to use the same `[[group]]` representation within the format itself? Dependency groups are specified in their own standard. Again, I don’t understand the relevance here. If you’re asking how a user requests that a tool creates a lockfile that will install the group and the package, that’s down to the tool UI, but the user would expect to state the dependency group name and “please include the package as well”. I feel like that’s obvious so again I think I’m missing something. > [@charliermarsh](#): > > Then, for uv at least, we’d leave all of our uv APIs as-is, and probably add some separate API to “install a group by name”, to facilitate installs from other lockfiles. Is that a good outcome? I’m not sure. This is the first time you’ve mentioned installing _from lockfiles_. For me (speaking as a user) I don’t see a lockfile as something I’d pick bits from like this. I’d either install the lockfile or not. So none of your questions really make sense to me because I don’t see a lockfile as a “set of things to pick what to install from”. Instead, I’d see the correct workflow as me requesting the tool to build a lockfile that represents the install request I want to support (“install the project and dependency group X”), and then requesting the same (or some other) tool to install that lockfile. Does Poetry have this idea of “picking a chunk of a lockfile to install”? Does PDM? What are the user workflows that need this functionality? Unless it’s something that all of the lockers acknowledge a need for, backed by actual user requirements, I’d say this isn’t something we should be worrying about for a lockfile standard. And if that means that you have to support this UI in uv by including additional information in the `[tool]` section, along with not supporting the feature for non-uv lockfiles, then that seems perfectly reasonable to me. No-one is suggesting that the lockfile standard should limit what uv offers its users, but conversely we shouldn’t be building something unique to uv into our standards. > [@charliermarsh](#): > > For example, they might want to ship a CLI that can install _any_ of those projects. Or a CLI that can analyze the dependencies of _any_ of those projects. Regardless of whether the underlying projects are using uv, or vanilla pip, or Poetry, or whatever. So what would a standard look like that supports this? It would support _only_ the common subset of functionality that all of uv, pip, Poetry, conda, etc., support. That pretty likely means it would only support building an environment that contains exactly what’s in the lockfile (because you included pip and maybe conda in there, so something with the functionality of requirement files is essentially the lowest common denominator). That might mean that teams using uv have to avoid certain uv features when publishing work for other teams to use, but that’s a consequence of their policy, not of the lockfile standard. After all, that same policy would prohibit a conda-using team from using a dependency that’s only available in conda, and not from PyPI, which is pretty much the same type of limitation. > [@charliermarsh](#): > > It also feels responsible for me to mention that there are a number of issues in uv that I’ve put on pause as we figure out the standard. Have you looked at how PDM and/or Poetry solve these issues? If there’s a common view on the “right” solution, that’s useful input to the lockfile discussion. If none of you have a solution, I’d strongly advise that we _don’t_ try to solve them here, as that’s the worst sort of “design by standardisation” mistake. Much better to let tools explore possible solutions without the burden of a standard limiting their options. * * * 1. and I can’t speak for Brett here, and I’m not speaking as PEP delegate so Brett shouldn’t feel constrained by what I am saying - disclaimers over now 🙂 ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 6:20pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/133 "2024-11-19T18:20:29Z")
> [@pf\_moore](#): > > Does Poetry have this idea of “picking a chunk of a lockfile to install”? Does PDM? What are the user workflows that need this functionality? Unless it’s something that all of the lockers acknowledge a need for, backed by actual user requirements, I’d say this isn’t something we should be worrying about for a lockfile standard. Yeah, all of uv, Poetry, and PDM support installs that are parameterized by extras and dependency groups. For example, Poetry: ```python # Install the docs group, along with the project. poetry install --with docs # Install only the dev group, not the project itself. poetry install --only dev # Install the project with these extras enabled. poetry install --extras "mysql pgsql" # Install the project with all extras enabled. poetry install --all-extras ``` Or uv: ```python # Install the docs group, along with the project. uv sync --group docs # Install only the dev group, not the project itself. uv sync --only-group dev # Install the project with these extras enabled. uv sync --extra mysql --extra pgsql # Install the project with all extras enabled. uv sync --all-extras ``` The idea is that you want to lock your project along with its dependencies, optional dependencies, and dependency groups (e.g., development dependencies). Then you may want to install a subset of them depending on what you’re doing: linting your project, running tests, deploying to production, etc. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 6:24pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/134 "2024-11-19T18:24:05Z")
> [@pf\_moore](#): > > Dependency groups are specified in their own standard. Again, I don’t understand the relevance here. If you’re asking how a user requests that a tool creates a lockfile that will install the group and the package, that’s down to the tool UI, but the user would expect to state the dependency group name and “please include the package as well”. I feel like that’s obvious so again I think I’m missing something. The relevance is that I’m trying to understand what we want to get out of interoperability. If each tool uses the standard format, but none of them understand how to install from it, is that okay? This _assumes_ that installers need to be able to install subsets of the lockfile, as illustrated above. If each tool uses a slightly different representation for the lockfile internally (e.g., different naming convention for each `[[group]]` or something like that – my example was uv using `package~group` as the name for the development dependency `[[group]]`), then they have to translate the CLI into those conventions. Otherwise, how can uv support running `uv sync --group docs` on a Poetry-generated lockfile? If we want to support _that_ level of interoperability, then we need to formalize more of the concepts in the schema (like a package with its associated groups). ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 19, 2024, 8:02pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/135 "2024-11-19T20:02:04Z")
> [@charliermarsh](#): > > The relevance is that I’m trying to understand what we want to get out of interoperability. Fair point. I think that for me, what matters is: 1. No tool has an advantage. By which I mean, a project shouldn’t feel that they should choose a particular tool because it’s better supported by dependabot, or by VS Code, or whatever. That doesn’t necessarily mean that the tool must use the standard lockfile as its native format, just that it should be able to interact easily with services, IDEs and other tools that support the standard format. 2. Switching tools should be relatively simple. I don’t think lockfiles matter for this, though, as I’d absolutely expect that if I switched workflow tools, I’d relock my project as part of the transition process. But being able to export a lockfile from the old tool and import it into the new one could be useful, if you want to avoid relocking. Things I consider _not_ to be goals include: 1. Using multiple tools _at the same time_ on a project. Project maintainers should decide on their workflow as part of setting things up for the project, and that means picking a tool and committing to it. They may choose to support external contributors using different tools, but it’s OK if that takes a bit of extra work. Again, export/import of a lockfile might be useful, but being able to run Poetry on a PDM-managed project isn’t what I mean by interoperability. > [@charliermarsh](#): > > This _assumes_ that installers need to be able to install subsets of the lockfile, as illustrated above. I’m unclear on this, as I don’t really know whether `uv sync --group docs` or `poetry install --only dev` should be considered a “workflow task” (and hence something that only maintainers do, which means it’s fine if it’s tool-specific) or if it’s something that an _external_ user of a project might want to do (meaning the project needs to make it possible in a tool-agnostic way). > [@charliermarsh](#): > > Otherwise, how can uv support running `uv sync --group docs` on a Poetry-generated lockfile? If we want to support _that_ level of interoperability, then we need to formalize more of the concepts in the schema (like a package with its associated groups). OK, but _do_ we want to support that? I don’t - as I said above, using different tools on the same project simultaneously feels like it’s going way beyond my concept of interoperability. We don’t require every build backend to be able to build every project, so why should we require that of workflow tools? If you think that level of interoperability _is_ what people want, do you have evidence of that? You mentioned the users who have a “happy path” set of tools, while allowing teams to choose different ones if they want. But I honestly doubt that even they would expect to support a team using both PDM and uv on the same project, at the same time. That feels totally unsustainable to me, and if they did allow that, it would feel like weak management to me, to be honest. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 19, 2024, 9:12pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/136 "2024-11-19T21:12:42Z")
Okay this is super helpful because it highlights some of the different assumptions we’ve been making. My assumption had been that we _did_ want to enable an installer to install from a lockfile produced by another tool in a first-class way. IIRC there were comments very early on about PDM users who wanted to use pip to install their dependencies in CI or when deploying. And I could imagine users that want to use uv to install in CI or similar even when the project itself is managed by Poetry or DPM. But it sounds like you’re envisioning this as more of a special case? Or like… installers need to be able to install an _entire_ lockfile, but not anything more nuanced? > [@pf\_moore](#): > > I’m unclear on this, as I don’t really know whether `uv sync --group docs` or `poetry install --only dev` should be considered a “workflow task” (and hence something that only maintainers do, which means it’s fine if it’s tool-specific) or if it’s something that an _external_ user of a project might want to do (meaning the project needs to make it possible in a tool-agnostic way). Can you clarify what you mean by “external” user of the project? When is an external user installing a project’s dependencies? ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 19, 2024, 9:25pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/137 "2024-11-19T21:25:55Z")
> [@charliermarsh](#): > > Or like… installers need to be able to install an _entire_ lockfile, but not anything more nuanced? That’s certainly the only scenario that I, personally, think makes sense. But I’m in the “lockfiles are environment definition files” camp. It’s you (and to a lesser extent @radoering for Poetry) that I recall saying that being able to install bits of a lockfile is important. But I’ve never fully understood the underlying use cases for this. And in particular I don’t see how it’s useful to anyone except a developer who’s working solely in uv (or Poetry, whatever his tool of choice is) on the project itself - and hence, not in an “interoperability” sense. > [@charliermarsh](#): > > Can you clarify what you mean by “external” user of the project? Anyone who isn’t a project maintainer, following the project development guidelines… > [@charliermarsh](#): > > When is an external user installing a project’s dependencies? A web hosting service which, given a project repository URL, downloads that project, creates an environment based on the project lockfile, installs the project into that environment, and starts the project as a web service. But essentially that’s my point - no-one other than project maintainers should care about “dev” dependencies, or “docs” dependencies. And project maintainers should be using the project’s chosen workflow tool, so interoperability doesn’t apply. One possible exception might be something like readthedocs. They might download the project and install the docs dependency group. But again, I’d imagine that being done by creating a _separate_ lockfile that captured the environment definition for the docs build. I don’t see any reason why the docs environment needs to be compatible with the “dev” environment, for example, which is typically the argument I’ve seen here for putting everything in one “do everything” lockfile. ---
**Author:** ![sirosen](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sirosen/32/8478_2.png) [@sirosen](https://discuss.python.org/u/sirosen)\ **Post date:** [November 19, 2024, 9:48pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/138 "2024-11-19T21:48:27Z")
I have a use case which requires that two tools be able to interact via the lockfile. I would like to lock my dependency groups via a locker, and then install the locked versions with my test runner (tox). In this context, I’m using tox as an installer. (It may be backed by pip in fact, but I don’t know if that’s relevant.) Both the locker (e.g., uv) and the runner (e.g., tox) should be components I can swap out in this workflow, with appropriate adaptations. I think this use case is representative of other users’ desires as well, given how I’ve seen pip-compile used. But it’s hard to be certain. This use case should not be read as overly constraining. For example, the way the named groups are produced in the lockfile is relevant to me, but doesn’t need to be prescribed by the spec. As long as the output of the locker can be documented by that tool, I’ll be able to use it as input to the runner. So I don’t agree with the idea that the file shouldn’t support “multiple tools”, but notably I also don’t think perfect and seamless interoperability between distinct tools _which can act as lockers_ is a goal. uv may be able to provide a better CLI by assuming that it was the locker used, and likewise for pdm and poetry. But “pure installers” like pip will need to be able to consume any locker’s output – possibly with a more verbose specification of which group to install. Included in this is the notion that the lockfile is not a single “install plan” for a single environment, but that it covers multiple target environments for a project. I think having multiple environments covered by a single lock is implied but having a required filename. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 19, 2024, 10:14pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/139 "2024-11-19T22:14:28Z")
> [@sirosen](#): > > I think having multiple environments covered by a single lock is implied but having a required filename. Ah yes, I’d forgotten this. Previous iterations of the PEP allowed things like `pylock.dev.toml`. I still think that “one lockfile per project” is an unnecessary restriction (and my recollection is that hatch, in particular, wanted a lockfile per environment where a project could have many environments) but it’s not something I’m going to worry about given the more significant issues we have to deal with first. OK, I’ll concede that if we stick with the one lockfile per project restriction, multiple environments per lockfile is necessary. Although I think that’s backwards - it’s the multiple environments per lockfile requirement that means it’s not worth having multiple lockfiles per project. And for what it’s worth, multiple environments per lockfile imposes another (IMO unwarranted) requirement, that _everything_ is locked as one set - you can’t (for example) lock the `docs` group independently of the `test` group. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 20, 2024, 1:53am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/141 "2024-11-20T01:53:28Z")
> [@pf\_moore](#): > > It’s you (and to a lesser extent @radoering for Poetry) that I recall saying that being able to install bits of a lockfile is important. But I’ve never fully understood the underlying use cases for this. Yeah. Here are things people do with uv that rely on this, off the top of my head: - Install only the development dependencies, e.g., when linting or formatting a project. - Install only the documentation dependencies, e.g., when building documentation. - When building a Docker image, install a project’s dependencies without the installing project itself, to create a highly cacheable Docker layer. - In a workspace, install a single local package (like a library) so that you can run a command in that package (or test that package) without installing the other packages in the workspace (like an application that depends on that library). It is valuable for these use-cases to share a single lockfile. For workspaces, I think this is obvious? When testing the library, you want to use the same transitive dependencies as when testing the application, which depends on the library. For the development tooling: even if the documentation dependencies are never installed alongside the project itself, most of the time, you’d prefer to use a similar dependency tree for the documentation as for the project itself (i.e., you wouldn’t really want to use two different patch versions of pydantic when one would suffice – it’s simpler and more efficient). ---
**Author:** ![radoering](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/radoering/32/8399_2.png) [@radoering](https://discuss.python.org/u/radoering)\ **Post date:** [November 20, 2024, 5:17pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/142 "2024-11-20T17:17:59Z")
> [@brettcannon](#): > > I believe Poetry also asked for editable install support (along with the lock file using source locations to be part of specifying what’s ); @radoering can correct me if I’m misremembering. That is correct. Poetry supports to define editable path dependencies in the pyproject.toml and thus also includes them in the lock file. I haven’t used this myself yet, but if I remember correctly this is particularly useful for mono repo use cases. > [@pf\_moore](#): > > It’s you (and to a lesser extent @radoering for Poetry) that I recall saying that being able to install bits of a lockfile is important. Poetry does not support workspaces as uv does so it is indeed a bit simpler. In the following cases, a package from the lock file is not installed (flat package, no graph approach): 1. The resulting marker evaluates to false when inserting the values of the target environment. \[1\] 2. The package is not required by any dependency group the user requested to install. \[2\] 3. The package is only required by an extra of the root package, which is not requested. \[3\] Actually, this is (more or less) just a combination of 1. and 2. because extras are always in the main dependency group and can be expressed via the extra marker. * * * 1. In theory, the lock file contains a solution for each possible environment. 2. The lock file contains the packages required for all dependency groups so that you can install any combination of groups from the lock file. 3. The lock file contains all extras so that you can install a combination of extras. [Previous page](https://discuss.python.org/t/pep-751-now-with-graphs/69721.md?page=6) [Next page](https://discuss.python.org/t/pep-751-now-with-graphs/69721.md?page=8)