That’s not for us to say. Individual tools can (and will) make that decision for themselves.
What you describe as “limited” I could describe as “carefully scoped”. It doesn’t matter what the PEP ends up proposing, it will be “limited” in terms of some people’s expectations. What’s more important (speaking at least in part as PEP delegate here) is that the PEP clearly defines its scope, and proposes a standard that addresses that scope.
What you’re suggesting is precisely the mistake that the phrase “the perfect is the enemy of the good” warns against.
I think that whatever your personal views might be on the [tool] namespace in pyproject.toml, it’s been a huge success. Without it, every build backend would have its own configuration file. I find it hard to see why a [tool] section in a lockfile is so drastically different. And even if it is, there’s no requirement for tools to use the namespace - if it’s a bad idea, simply don’t use it.
I don’t think it’s particularly useful to debate terminology here. It’s a lockfile in the sense that many people use the term, and it solves the problems that those people need a lockfile for.
It could be for us to say, by omitting the tool section from the specification.
I actually think these use-cases are quite different. The pyproject.toml is the home for all Python project configuration. A tool specific section here makes a ton of sense. This file is edited by users. In contrast, the lockfile is a generated file. It’s focus is consumption by machines, and interoperability is much more critical for it to achieve its purpose.
I think omitting the tool section is drastically different because it means that the lockfile is always interoperable. Otherwise, users will not be able to depend on interoperability — the lockfile will only achieve its goals in a subset of use-cases and we’ll have to teach them why that is.
I think it’s great that tools like flake8 and co can leave stuff there. I do not at all like that core packaging tools (including my own rye or now uv) are putting core behaviors in there that fundamentally change how the rest of the file works. However that is still better than what proprietary extensions in a lock file format would do, since at least nobody re-generates pyproject.toml files so evicting unknown information does not happen in practice all that much.
Thanks Paul – I appreciate the follow-up, and yes, I think that’s a reasonable perspective. My hope is that we will be able to use the standard lockfile for standalone projects, but as you guessed, there’s some calculus to do around the cost/benefit that will only be clear once we’ve hashed out the specifics of the spec.
Personally, I’d prefer that the format explicitly encode concepts like extras and [dependency-groups] (or omit [dependency-groups] for now if needed), rather than use a more flexible “groups” concept. (But I’m trying to balance “giving feedback on specifics of the upcoming revision too soon” with “giving feedback that will help inform the upcoming revision”.)
For me, when I say “installer-agnostic”, I mean “every tool can install from this lockfile (and every tool will install the exact same set of packages when they do so)”.
So, personally, I would view any lockfile whose behavior varies at install-time when installed by a specific tool as non-installer agnostic. For example, if we wrote [tool.uv] information to a package in a lockfile, and when installed with uv, we read that [tool.uv] and somehow selected a different package if the Poetry was installing from the same lockfile, I’d view that as a non-installer agnostic setup. While Poetry might be able to install from that lockfile, Poetry and uv would install different packages (in that scenario). This is partly why I was pushing a bit for [tool] to be omitted from the spec – it makes non-installer agnostic behaviors impossible. But the spec can also be clear about what’s “allowed” here. Like, we could say that (in this scenario) uv is violating the spec, or that if uv is going to do this, it has to mark the file as non-installer agnostic, etc.
I would expect the standard to specify precisely what to install when given a lockfile. Indeed, I can’t imagine accepting a standard that didn’t specify that. So in your terms I would expect every lockfile to be installer-agnostic in that sense.
Yes, tool-specific settings could change (for example) the default group to install, but that is simply a UX issue, as someone using an installer that doesn’t recognise that setting could manually request the same group.
Perhaps to some, but I honestly don’t expect most users to mix and match tools.
I’m currently thinking raw strings just like you don’t want it to be.
But they can’t do that with requirements files either; that’s all just some filename part that we have all decided is either *-requirements.txt or requirements-*.txt.
Nope, I’m thinking packages.groups is an array of strings that filters the initial list of packages and then packages.marker is the next level of filtering, leaving you with the set of packages to install.
That’s actually not what I meant. I specifically was thinking that “installer-agnostic” would mean the PEP would specify how installers operate. But an “installer-specific” lock file would have some data that could be understood, but other that was inside a [tool] section. As such, it would mean other tools couldn’t install from it. But based on comments that I will address below from @pf_moore , it seems my idea is not going to happen anyway.
Yes, it is rather different. And yes, some tools may have to explain to users that using some feature means they can’t expect other tools to use their lock file (and lets be honest, we know that onus will definitely hit uv). But based on what I say below, you don’t have to worry about it.
Thanks, although people are already misinterpreting some brain dumps of mine regardless, so it’s a losing battle regardless.
In that case the concept I had in mind for “installer-agnostic” isn’t viable as that was specifically to try and let uv have a format they could use which provided some data, even if it is was incomplete in terms of what other tools could understand about it.
Here’s where things seem to stand based on the above and other conversations I have had with folks:
uv will use whatever PEP 751 defines as an export format, but not as its own lock file format (i.e. the “installer-specific” concept is dead)
PDM and Poetry may be able to use PEP 751 as a lock file format if it has the right features
People would prefer having explicit support for extras and dependency groups instead of a generic “groups” concept
I think I would need to see a concrete proposal to understand what “precisely what to install” means as opposed to “tool-specific settings”, but my current hope is to have a format such that:
We can use it in uv, in the same way as PDM and Poetry, for standalone projects (i.e., projects without a workspace). Ideally, this is, like, “fully” (?) installer-agnostic?
We can use it in uv with some kind of non-installer-agnostic extension for workspaces.
I did misunderstand what you meant, but I don’t think you have to drop the idea of including the tool table completely, as my misinterpretation was based on a variation that I think will work (and I read @pf_moore and @charliermarsh’s responses above as also judging it plausible).
Specifically:
the default behaviour of installing from a standard lock file is always installer agnostic (fully defined by the specification)
installers are permitted to define opt-in behaviours which aren’t covered by the specification (and some of those may only work if the relevant tool table entry is present)
tools emitting lock files with tool table entries have a way to mark the files as resolver specific when there is information in the tool table that needs to be kept in sync with the standardised fields
Your naming choice makes sense now that I understand what you meant, but I agree allowing lock files that are installer specific to use the same name as the standardised installer agnostic format wouldn’t be a good idea.
Edit: in addition to the default installation behaviour, the other piece that should be fully standardised is what it means for resolvers to use a standardised lock file as a constraints file. (primarily that extras declarations aren’t relevant in that context)
This is what I understood as well. And in particular, I didn’t mean my comments to imply that Brett’s proposal wasn’t going to work, although I think I need an explicit example to be sure I understand it.
