I think we’re referencing my earlier post here: PEP 751: now with graphs! - #211 by charliermarsh
Sorry, yes, this is all true!
We can support (1) + (4) and reject any lockfiles that are not installer agnostic. But the opinion I’m trying to express is that (independent of uv) I don’t see this as a good outcome. If we have this file format that’s standardized but isn’t actually interoperable, what is it buying us? Concretely, what are the benefits? If we can take an installer-agnostic lockfile, and any tool can install it, that seems like a win. But if it’s not tool-agnostic, why introduce it? Why encourage tools to use it? I don’t think the Dependabot example is very strong given that the spec wouldn’t enable update support (i.e., it’d be less than what Poetry gets today, unless Poetry was again special-cased in Dependabot as it is today), so I’m looking for other justifications, since adding this installer-specific section will come at a cost: it adds complexity to the format + spec, and it means that users need to understand when / where they can use pylock.toml (and where / when it’s interoperable vs. not).
As I’ve said before: I’ll do my best to use and support whatever is standardized. I can say with confidence that we’d support (1). So, in some sense we’d also be fine to support (4), at minimum by rejecting non-agnostic lockfiles (as every tool would, if I understand correctly). But personally I’m not in favor of the expanded scope and adding these installer-specific customizations. I think it dilutes the standard. If that’s the direction we take, though, so be it.
1 Like
That’s a bit of a pivot from:
So you’re changing your mind?
You yourself suggested that Dependabot could work in this scenario as Dependabot could update versions since that would be standardized (and I’m still trying to get confirmation from someone who works on Dependabot).
If you didn’t create it, then yes.
It isn’t interoperable if your users need something from uv that can’t be expressed in the standard. That isn’t a guarantee and I would hope not the common case since a standardized requirements.txt file isn’t going to give you more flexibility either.
So for a requirements.txt v2, there doesn’t seem to be any explicit disagreement from the tool providers (beyond whether groups should exist or not), but some users have said that’s not the outcome they prefer as they don’t want to have to keep a requirements.txt file manually synced with a tool’s custom lock file.
Trying to be a bit more accommodating to allow for independent tool evolution while not painting ourselves into a corner where some requirements.txt v2 conceptually can’t ever be a lock file has the following tool support:
- PDM:
, but no groups - Poetry: , but with groups
- uv:
since doesn’t want [tool]
Here’s what I’m going to do. If I can get Dependabot or some other equivalent consumer of lock files to tell me that having some data be tool-specific isn’t a showstopper, we will put in some installer-agnostic flag (and if people have suggestions of whom else I can talk to for feedback, please let me know). Otherwise I will basically roll back PEP 751 a major version to what I had before (the set of packages approach), I will make sure all major features for what requirements.txt can do is covered, see what we can add that people generally think is useful, and then I can hope it’s enough for some tools to use as their lock file format anyway and that people care enough to help evolve it so it can handle everyone’s lock file needs in the future once they feel they know what’s missing.
But I’m approaching 6 years on this topic come February, so I do not plan to go past CY Q1 2025 – i.e. by April – before submitting PEP 751 for pronouncement.
3 Likes
It can be addressed by two entries for A:
[[packages]]
name = "A"
groups = ["dev"]
[[packages]]
name = "A"
groups = ["test"]
marker = 'sys_platform == "win32"'
Does that work for you?
3 Likes
Sorry, I didn’t mean to suggest that Dependabot could update versions. I’ve assumed that was off the table for a while, since we gave up on resolver interoperability (I don’t see how Dependabot could update the lockfile without asking a resolver to resolve with the updated dependencies?). What I meant was that Dependabot could read it the in-use versions and give you diagnostics based on those in-use versions (e.g., security advisories). My most recent comments are questioning whether that’s a worthwhile outcome.
Just to clarify, my current expectation is that every uv lockfile would need information in tool. I just want to make sure we’re not talking past each other there. (At minimum, we need the full dependency graph to power things like uv tree, which IIUC the previous iterations did not include.)
Yes. I thought on it more, talked to people on our team, and ultimately had trouble coming up with justifications for it. But I feel heard on my arguments, I won’t repeat them. If we go with the [tool] extension variant, then we’ll try to use it, and I’ll likely make some requests as we get into the details to make that possible. (And if we ultimately find that we can’t use it, then we’ll have to stick with uv.lock.)
1 Like
I haven’t fully engaged with this discussion, but a comment from the peanut gallery:
I would say it provides a huge milestone in terms of solving a concrete issue in a standardized way (which can be extended in the future), while leaving tool-specific modifications on the table where necessary.
More importantly, things don’t happen in a vacuum. If the homogeneity is prized as highly as you hypothesize, then users will mostly use these lockfiles in a tool-agnostic manner, and shun the tool-specific variants for things like their dependabot setup[1]. And it still lets people opt into more features (which could later be standardized!) where they prefer that over the tool lock-in. That seems like a massive win to me.
I think this is a very hard-won higher equilibrium and a “but it’s not as universal as it could be” is kind of moot, because we don’t have anything more universal on the table (that people are in agreement on), and holding out for that risks killing the progress outright for the foreseeable future. So IMO it’d be more than deserved to take that win, observe the usage patterns for 1-2 years, and then come back with fresh eyes.
presumably most tools would provide an option to produce a tool-agnostic file if demanded by the user ↩︎
9 Likes
As a maintainer who uses lock files, I agree with @h-vetinari. I’m not a fan of maintaining two “lock files”, e.g., a requirements.txt exported from uv.lock or poetry.lock. If we can get a standard location (pylock.toml) that acts as both a requirements.txt for other tools (say pip) while also providing extended features with the [tool] table, I think it would be a win for the ecosystem. As things mature, more and more features can be moved out of the [tool] table without requiring a full blown lock file PEP (it would still require a PEP but the PEP would only have to justify a feature, instead of a whole lock file standard).
4 Likes
I also support option 4.
Just to throw in my use case: I package a few scientific apps where being able to reproduce the environment after several years is considered important.
My single source of truth for the Python package is pyproject.toml and I have a Conda env.yml that includes pins for non-Python tools and some core packages (like numpy) that get pulled in by the dependency graph. I use pip compile (now uv pip compile) to generate a requirements.txt that I update and check in at release time, and include it in the env.yml as follows:
name: myenv
channels:
- conda-forge
dependencies:
- python=3.11
...
- pip
- pip:
- -r requirements.txt
As long as I keep any overlapping pins consistent between env.yml and requirements.txt at the time of release, I am able to keep regenerating these environments for years, and even release bug-fixes of years-old branches where all our dependencies are out of support.
I don’t want, or intend, to start committing uv.lock (or other tool-specific lock) files in addition to pyproject.toml, env.yml and requirements.txt. If I can do pip install -r pylock.toml, then I’ll migrate my requirements.txt and call it a day. If that new file has a [tool.uv] section, then I might start caring about what advantages there can be to having a checked-in uv.lock.
2 Likes
If I introspect a bit, a lot of my pushback comes from a feeling that introducing this format will actually discourage innovation.
To me, the fact that uv already can’t use the proposed standard despite trying hard to do so is a sign that it’s too tightly coupled to specific product features.
Like, we view workspace support as tablestakes functionality, and the standard can’t support workspaces without becoming non-tool-agnostic. So suddenly, users are penalized for using workspaces, despite the fact that workspaces are a product feature, and not something that should need to be covered by a standard.
Similarly, if uv starts including build dependencies in the lockfile, users will be penalized for including those too, since it will make the lockfile non-installer-agnostic.
You might say, “Yes, but why is that worse than you just sticking with uv.lock?” The issue is that if we standardize this, the ecosystem will build around it, and we’ll effectively be penalized for going beyond the first-class support from the standard. We’ll need to petition through the standards process to support product features. That seems off to me. And if we fail to move things out of [tool], what happens then?
4 Likes
As a last thought…
(1) has evolved in the last few comments past requirements.txt v2. I originally described this as capturing a single scenario, but it’s now been suggested that it could support extras and dependency groups.
So, I thought (and my preference was) that we’d have pylock.toml alongside poetry.lock, pdm.lock, and uv.lock. In that case, I did not want pylock.toml to have a non-installer-agnostic mode. I wanted to clearly separate the “list of packages to install” from the “resolver lockfile”.
But if it’s now the case that the format would work for PDM and Poetry, then it probably is better to include the non-installer mode, so that we can use it in uv. We may be able to generate an installer-agnostic lockfile in some cases (e.g., single-project workspaces), which would mean we can use the same pylock.toml file universally, just installer-agnostic in some cases and non-installer-agnostic in others.
So, my preference is (1) (without support for extras or groups), then (4), with a caveat that if we do (4), we may end up keeping uv.lock – I’m not sure.
I know it doesn’t compare to Brett, but I’ve put many hours into this over the past few months and at this point feel somewhat defeated. So I’ll try to move forward and support whatever Brett decides on given these preferences 
3 Likes
I think that’s completely backwards, to be honest. Workspaces are currently a tool-specific feature. Maybe they will always remain that way, maybe not. But for now, it’s right that standards shouldn’t try to support them, outside of allowing tools the freedom (via the [tool] namespace) to support them within the standard. Conversely, requirements.txt is a widely used feature, that’s essential for many users’ workflows, and it absolutely should be standardised.
Users aren’t being penalised for using workspaces, they are simply being required to use a tool that supports them.
At the moment, uv writes requirements files as an export format. You can continue to do that for lockfiles. Maybe the user experience for users trying to work with uv’s locking capabilities and standard lockfiles is suboptimal, but to be blunt, that’s an issue uv has to deal with. If we don’t standardise anything, you’ll still have to deal with it (just replace pylock.toml with requirements.txt).
Your complaint seems to be that you can use lockfiles with [tool] as your lock format, but by doing so the lockfile won’t be portable. Sure - that’s because you’re offering tool-specific features that benefit your users. But if you do so, your users will still need tool-agnostic lockfiles, so you will have to support exporting to that format. What’s the problem?
Other tools seem comfortable working with the proposed format. So scaling it back (or worse, not standardising at all) because uv can’t seems like it’s blocking those tools from making progress on standardisation, just because uv needs capabilities that we can’t agree on yet to get past the “requirements v2” level of interoperability.
At this point, I think everyone has invested a huge amount of time into this. And everyone’s contributions are appreciated. The fact that we have had to scale back our ambitions shouldn’t be seen as a defeat - if you look back at the early proposals, even the most limited of the proposals we’re now discussing would significantly exceed the ambitions we had at that point. In the interim, tools have become more sophisticated, and not all of the new capabilities are ready for standardisation. But that’s fine - standards will always lag behind innovation, but they are still valuable nonetheless.
6 Likes
I disagree (with the characterization around standards and workspaces) but I appreciate your reply. I’m happy with any of the outcomes I described above.
1 Like
I originally proposed recording what’s needed to reproduce the lock file to help with this.
If that’s what uv ends up doing I’m personally fine with that. It’s at least a start, and I hope that as uv learns what they need that future PEPs can be written to update the spec to add support for what’s missing.
Actually both dependencies and dependents were optional data before the graph change. I find them useful when pip-tools provides that sort of information in comments (along with the command used to create the lock file), so I included them from the start of the PEP.
To be clear, a file will probably not do both at once; it would be one or the other depending on whether the tool generating the lock file flagged the file as needing a specific installer. Now it’s not impossible, but it’s up to the tool generating the file to say that no critical data for installation is hidden in a [tool] table somewhere.
I specifically suggested “groups” which could be used for extras and dependency groups. It’s really more of a way to make requirements.txt files that use -r actually be a single file instead multiple files, all without duplicating data. As an example, if your requirements.txt file was:
urllib3==3.2.3
and your requirements-dev.txt had:
-r requirements.txt
pytest==8.3.4
then I don’t see why you can’t have urllib3 be part of the “default” and “dev” groups and pytest in the “dev” group in a single file. That seems like the same outcome to me, just written in a single file instead of multiple ones (i.e. I don’t see how it expands the scenario of what’s currently possible with requirements files). Unless you were thinking more of a “requirements.txt v2” as self-contained requirements files only (i.e. no -r functionality), and so with less features than what requirements files currently have (which would make sense if you view it as an export target as you may not be exporting to multiple files)?
I was honestly planning to do the next version of the PEP with all data justified by either being supported in requirements.txt files, helping with security (e.g. requiring hashes), or helping with auditing/code reviews (e.g. recording the package version when it can be specified reliably to make @hynek happy 
). I would then move a bunch of stuff to open issues and if 2 or more tools out of PDM, Poetry, and uv say that something would be useful then it would be added to the PEP, otherwise it would end up being a rejected idea. I was then also going to include something that provided an escape hatch for tools that needed to record non-standard data in [tool] that would affect installation if that’s what a lot of us agreed made sense to support (which sounds like a possibility).
3 Likes
Genuinely wondering, do you think this will be confusing for users? More so than the status quo where lockfiles are either obviously interoperable (requirements.txt) or not (<tool>.lock)?
Could you clarify how these groups would be defined? My concern is that if groups are just arbitrary strings, then the interoperability goes out the window — i.e., in that case, a tool can’t install based on extras and PEP 735 dependency groups without knowing how the locking tool names the groups. I’m not quite sure what the plan is for dependency groups, extras, and markers. Reading further back…
it seems you settled on this instead of having group and marker labels on packages? but these were still present in a subsequent comment, e.g., PEP 751: now with graphs! - #226 by frostming.
I think including requirements files from other files is important for the inputs to the locker but not the output. For example, people nest requirements.in and requirements-dev.in as you’ve described, but it doesn’t really make sense to do so for the compiled requirements.txt output. The lockfile should be self-contained. Combining arbitrary locks seems prone to failure and rather confusing.
This makes sense to me, thanks Brett!
1 Like
Yes, this should work. I wonder if all fields (like wheels) have to be duplicated, but since this should only be an edge case maybe even that is fine.
1 Like
One point I’m taking from @charliermarsh’s concerns is that there is a bit of ambiguity with what it means for a lock file to be “installer agnostic”, and I think that’s where the uv workspace handling is encountering friction.
Fundamentally, pylock.toml corresponds to a single application (or library dev environment), such that “Install this with its dependencies” is a valid request to make (in many cases, “this” will be an adjacent pyproject.toml file). In that context, “installer agnostic” refers to the ability to correctly update the file with revised dependencies - merely installing the default set of packages is expected to work even when the file as a whole is installer specific. This makes me think “installer agnostic/specific” is a poor naming choice - maybe we should instead describe these files as “resolver agnostic/specific”? All lock files are going to be installer agnostic to some degree (that’s the essential point of the entire exercise!), even if the installer side of the corresponding resolver can offer extra features (like selective installation).
Since PDM & poetry work with a single pyproject.toml at a time, it is plausible that they could entirely replace their existing lock file formats with a resolver-specific lock file that still allowed interoperable installation with other tools.
Due to the workspace feature, this is not the way uv lock files work. Even a resolver-specific pylock.toml file would only correspond to a single project within the workspace, it cannot reasonably represent the workspace as a whole. This means that uv.lock would still need to exist at the top level (with an entirely uv-defined format), and uv could at most emit resolver-specific pylock.toml files for the individual workspace entries that referred back to the parent workspace from a [tool.uv] table entry.
That still feels like an overall interoperability win to me, since the individual workspace components could still be installed independently of uv at that point. Actually updating them would require uv, but that is the basic restriction on any resolver-specific lock file.
Maybe some day we decide to define pyworkspace-lock.toml for managing composite Python projects, and then uv could consider switching their main lock file over to that, but we’re a long way from that point (and we may never reach a point where standardising that level seems worthwhile).
4 Likes
That is just a UX detail I guess, but I see mentions of scenarios where we would have both a pdm.lock, poetry.lock, or uv.lock, and a pylock.toml and I wonder if it really has to be two files. Would it be possible to have the whole proprietary lock data in (a sub-table of) the proprietary [tool.tool-name] table of pylock.toml instead, and a “sensible default export” (that can be specified by the maintainers of the project) in the standard tables of pylock.toml?
Rereading what I said, maybe I could better express it by talking in terms of “monorepos” rather than “workspaces” (because my impression is that workspace support is how uv handles monorepos, but “monorepo” is the broader term not focused specifically on uv).
Monorepo support is well known to be weak in the packaging ecosystem at present. It would be good to improve that (and uv’s workspace support is a great example of how we might do that) but it’s too early to think in terms of standardising monorepo support. However, the fact that we’re not yet able to standardise locking for monorepos, doesn’t mean we shouldn’t standardise lockfile support for standalone Python projects. We know that many tools use requirements.txt as an adhoc substitute for lockfiles, so the need is clearly there, and given the well-known limitations of requirements files, we absolutely should be standardising a lockfile that can replace requirement files.
If that means that tools supporting monorepos have to maintain some form of tool-specific extensions to the lockfile format to handle monorepos, this isn’t the end of the world. Users won’t be able to share monorepo lockfiles between tools, but that’s no different from the current situation. And as long as tools use the standardised format for standalone projects, interoperability will be possible in that case.
It appears that uv might decide not to use the standard lockfile for standalone projects, but stick to the proprietary format for all projects, leaving the standard format as merely an export option. That’s a shame, as it means that even for standalone projects they won’t have the interoperability benefits. But I’ve no idea how the internals of uv work, and can’t judge the cost/benefit of a two-format approach. I can easily imagine designs where the overheads and complexity would outweigh the benefits. So if that’s the way uv chooses to (or has to) go, then that’s perfectly fine. Equally, if we can make changes to the format in a way that would make it easier for uv to support it natively for standalone projects, that would be great. What we shouldn’t do though is try to cater for uv features that are needed specifically for monorepo (workspace) support, as the ecosystem’s position on that isn’t mature enough yet to warrant standardisation.
Does that formulation of what I was trying to say sit any better with you, @charliermarsh? I didn’t mean my previous comment to sound like I was pushing back against uv’s input - but on rereading, it did come across that way, and I apologise for that.
8 Likes
Yep, this is the essence of the “installer specific” (which is really more “resolver specific”) lock file idea. The standardised fields will still be enough to let other tools install from the file, but only the original resolver will be able to update it reliably (since the tool table needs to be updated as well).
It’s still scoped to a single project, so it can’t cover uv’s workspace feature, but other than that it seems like a plausible format for project management tools to use.
2 Likes
I tend to think it would be better for all tools to opt for that option until it’s clearer how a long term solution should look like. It think it’s a bad idea to adopt a limited format as a standard. The result will almost immediately be that people put proprietary extensions into that format in the same way as tool.x is growing in scope in pyproject.toml.
I like the idea of this not being a lock file but a more modern requirements.txt.