As a user, I like the idea of a compromise @brettcannon I think I understand the pyproject.toml comparison. I think it may take a while for the mental model to shift from current discussions.
Fwiw, pip does have some of the machinery towards a v2 requirements lockfile. The installation report.
And maybe the tool maintainers we need to consider are:
(pip + pipcompile)
poetry
PDM
hatch
uv
One way to help consolidate the ecosystem, might be for this initiative to roll pip and pipcompile together. I know it will be controversial and my perception so far is that the pip maintainers are unlikely to welcome the idea. Iām just speaking as a user that it feels like pip holds this unique place that itās important, it has a huge responsibility to not break backwards compatibility, but that itās also not a very complete tool for a lot of users. Thatās completely understandable and reasonable of course. Not a criticism. pip + pipcompile is more comparable to the rest in the list. Just my 2c.
I prototyped generating a lockfile from the pip installation report, very early in this discussion. At the time, it wasnāt hard. I donāt know if the more recent āwith graphs!ā versions of the standard would be as easy, but Iād hope itās not impossible.
From pipās perspective, I see three places that might gain lockfile support:
Installing from a lockfile. This is relatively uncontroversial, with the only potential issue being whether installing while ignoring the [tool] section of the lockfile is allowed.
pip install --report --report-format=lockfile. This seems like it would be easy enough. It would be equally easy for someone to write a tool that read the existing pip report and generated a lockfile. So this one will probably come down to whether anyone feels itās worth the effort to write a PR for pip.
pip freeze --format=lockfile. This is potentially possible if PEP 710 gets accepted. But it will likely be unreliable until enough time has passed for PEP 710 metadata to be universal. It would also be easy enough to write as a 3rd party tool, so I wouldnāt expect it to be added to pip particularly soon.
The other question is pip-tools, which you grouped with pip. I donāt know if itās reasonable to do that - yes, many users use pip and pip-tools in combination, but the two projects are independent, and we donāt collaborate in any way to provide a āsharedā workflow. I donāt see any reason why this would change for lockfile support.
I certainly would be against this. If nothing else, I donāt think we need yet another workflow tool, which is what āpip plus pip-toolsā would inevitably become. But merging pip and pip-tools is offtopic for this discussion[1], so I suggest we donāt worry about it for now.
Thatās a big ask for an idea I came up with while trying to fall asleep.
Think about the spec covering what it currently does, but providing flexibility to have tool-specific ways to say how to get code or what to install, and in such a way other tools can know what parts are tool-specific.
The latter. I think @charliermarsh once pointed out that anything in [tool] would have to be basically harmless āFYIā info or else you break installers w/ the current locker/installer dichotomy. Iām saying open it up so itās okay for tools to have stuff that ultimately mandates that same tool be used for installing.
I think others pointed it out, but a lock file that didnāt have any tool-specific info could still be read by pip (and potentially produce a lock file).
Thatās one possibility. Another is we keep a graph format but provide ways to specify tool-specific ways to make relationships between packages and/or how to install the code.
Thatās up to the tools, but I assume that they would only reach for tool-specific stuff when necessary. As well, I would hope they would contribute back with future PEPs to expand what they have found to be useful.
Correct.
Yes, I assume a basic level of interoperability. Think of the PEP either in its current state or back in its āset of packagesā state, but where tools can have tool-specific details that they need for some extended support they provide which others simply wonāt understand. It might make that lock file not fully usable by other tools, but I also donāt assume it will make it an opaque file format either (e.g., package name and version is pretty well understood at this point ).
I donāt think it would be.
Probably not; you would error out. But hopefully either thatās not going to be the common case or we come up with some way to say, āyou wonāt understand this properly for installing, so donāt even tryā and either you export out to a lock file with no [tool] or you have to give up because it was never going to work due to some exotic thing the tool can do.
I think a āset of packagesā approach is sufficient for Poetry. [1] If I remember correctly, the only thing that was missing was a means to specifiy the resulting dependency groups a package is required for and the resulting markers per group. We can still put this in the tool section of each package but it will mean that another installer, which does not look into tool.poetry, will not install the correct set of packages under certain circumstances.
Considering my previous point, I suppose the tool that produces the lock file can tell if the tool section is harmless or not. Maybe, it is worth to add a field for this information (outside the tool section) if we continue to pursue this approach?
So a way to have named groups where the group itself had marker requirements overall? And still have marker requirements per package entry? Iām personally fine with that. Would you expect to consolidate markers to the group and then only have markers for packages if they are e.g., OS-specific?
I was thinking the same thing, like having an installer-agnostic key that signifies if the thereās critical information in any [tool] section which would impact installation.
Not exactly. More something like package A, which might be a transitive dependency, is required for groups dev and test. For group dev it is always required, but for group test it is only required for sys_platform == "win32". (That is not something specified in the pyproject.toml but the result of dependency resolution.) You need something like
[[packages]]
name = "A"
groups = ["dev", "test"]
marker = {"dev": "*", "test": 'sys_platform == "win32"'}
If marker is just a single marker string, we can only put something in that is either wrong for dev or wrong for test. Alternatively, we put it in the tool section of the package and specify installer-agnostic = false.
No, because there is an extra marker. If there was a marker for groups, we would only need one marker string per package. However, I suppose it does not make sense to have a marker for groups because groups are not published in the packageās metadata.
Right, and youāre only dealing with a single project, so the extra is unambiguous ā got it.
While weāre not yet aligned on the goals, I think thereās sort of a gradient of options being considered here:
A new requirements.txt: captures a single scenario, across many platforms.
The PDM-style lockfile from the previous iteration of the proposal. Itād still be focused on a single project, but it would presumedly support installing extras and dependency groups for that project.
The lockfile in this current iteration, which allows packages to be tagged as part of arbitrary groups, with the intent of supporting multiple roots (e.g., workspaces) in addition to extras and dependency groups.
I find a fourth option (4) sort of intriguing, inspired by Brettās most recent comments, which is: the standard just covers [[packages]], a flat list of packages, but any metadata connecting the packages or indicating when they should be installed, etc., needs to go in [tool]. That way the standard just covers ālisting a set of packagesā, which makes it very flexible while still enabling tools like Dependabot to operate on the standardized format. Then, uv, Poetry, PDM, etc. could decide what they want to write in [tool] to support their CLIs atop those lockfiles. And we can continue building out functionality in uv (like locking build dependencies) without being constrained by the standard.
My current opinions are as follows:
As I said before, if we want to do (1), that seems fine with me. It seems useful to standardize this, even as an export format.
If we do (2), I wouldā¦ try to find a way to make it work for uv. Maybe weād write a separate lockfile for each workspace member? Then weād be in āthe sameā position as Poetry and PDM. I think it would be a worse experience for uv users and would create a lot of complexity for us, but Iād still try. (Or maybe weād find a way to solve this with tool.uv? Iām not sure.)
If we do (3), Iād push for several changes to the schema, but Iām trying to ignore those for now. I remain worried that (3) is a little too closely coupled to the tools themselves, as per my previous commentsā¦ But if we no longer view installer interoperability as a goal, then I think itās possible for us to align on something here, and Iām game to try.
If we do (4), thatās also fine with me. In fact it might be my preference, since it seems to nicely balance āthe things we want from standardizationā with not constraining tools.
Finally, it was suggested above that one outcome here could be that Poetry and PDM use the standard as their first-class formats, but uv just exports to it. Honestly, Iād be really disappointed by an outcome like that, and I think itād be a big loss for the ecosystem to standardize on something that willing implementers already canāt use today. So, as my comments above reflect, Iām trying very hard to avoid that while still sharing my honest opinions
Reading the options youāve laid out, I think itās worth noting that youāve mentioned how (3) is ātoo close to tool implementationsā, but (2) has that same problem as well. Itās too close to the way that PDM and Poetry lockfiles work, which makes it hard to adapt for uv.
Hereās a particular experience which I (and I bet many other users) would like:
I have a variety of dependency lists (groups or requirements files)
I have a range of supported interpreter versions
I ask the locker to lock āthat matrixā of things
I later ask an installer to āinstall a dependency group or requirements file, but use the lockā, and the installer can find the locked data, lookup the current interpreter version, and give me a reproducible install or a hard error if it canāt be done
I think that sort of workflow can be satisfied under several possible lockfile specs, but it strikes me as easier if thereās a single lockfile with all of the data, rather than a file per possible environment.
Or you list the project twice with different markers (and Iām not saying I think this is better, just that itās an option). After the whole āversion numbers with source trees are a painā discussion I have been thinking about how to state the requirement that each listed package needs to be unique overall, but without being so simplistic as saying just āname and versionā. In this instance it could include differing markers and groups.
I believe @ncoghlan once suggested introducing a marker for groups.
Yeah, I think Iāve hinted at that possibility. I would probably want to tweak it a little, though, and have it piggyback on option (1) by saying the simple requirements case is covered when installer-agnostic = true ā bikeshed later on details ā is set and thus the lock file is something anyone can install from (e.g., pip could freeze an environment and also do an install from the same file as could anyone else). I would think [[packages]], markers per package, and thatās it; no extras, no groups, etc. Think requirements.txt but that it can be cross-platform thanks to the markers. Otherwise we can toss the markers idea out and have a top-level environment key that specifies under what circumstances the list of packages blindly apply to. Either way, it would be the greatest common denominator that anyone could directly work with and still act as an export target. It also doesnāt expand the feature set either way of what requirements.txt files currently supports, so we know the baseline feature set does meet a need.
But when installer-agnostic = false then all bets are off and tools can do whatever they want with no expectations anyone will do more than read the list of packages that may get installed somehow.
But I donāt know what @radoering or @frostming think of this idea (nor have I run this by GitHub yet to see what they think; Iāll go see if I can chase someone down after I post this). I also donāt know if you like this tweak, @charliermarsh . Iām personally happy to go with this idea if the tool maintainers think itās a good compromise.
Yep, but making everyone happy sure is hard.
I may have suggested it as itās a bit easier to reach that goal since everyone gets something out of this in that instance compared to right now where we are at an impasse. But hopefully the option (4) above works out.
But @ofek has said the exact opposite, so thereās no clear answer here.
Just to clarify, I think he was saying so in the context where the locker and installer are (or are wrapped by) the same tool.
āSame toolā vs ādifferent tools interoperatingā is one of the axes of this problem which I should probably be more careful to specify, since it dramatically changes things.
Youāre working hard on it though! We should try to be easier to please.
I donāt speak on behalf of the maintainers, this is just my opinion, but I feel almost certain that the current proposal is a requirement for pip to ever support proper cross-platform resolutions. Itās unclear to me (I think not currently) that there is a desire for pip to gain more functionality similar to a full-fledged project manager like Hatch or UV, and I agree with that preference. As such, I think the previous requirements.txt-style spec is more compatible with the current and future architecture of pip.
I was a bit saddened by that realization but I think itās fine after more contemplation. pip ships with Python and supports easy bootstrapping of dependencies targeting the interpreter with which it is installed. It is very good at this and serves that forever-critical need.
edit: To be explicit, Iāve updated my preference to the current proposal rather than the previous requirements.txt-style spec. This is the opposite of Charlieās update
I mostly havenāt been following the thread for the past few weeks (the size of the āunreadā number scared me), but I wanted to chime in to say I really like this idea due to the way it allows the ecosystem to evolve:
in the installer agnostic variant, we get our āstandardised cross platform requirements.txtā replacement
in the installer specific variant, we can still get standardised discovery of what the expected installer is (and how to install and invoke it)
over time, the capabilities of the installer agnostic variant can be expanded (similar to the migration of dependency group support from tool table entries into standardised fields)
Iād paint the bikeshed as a full [installer] table rather than as just a boolean flag, though. That way in addition to the simple declaration of the installerās name, it could also include a full transitive lock of the installer and its dependencies, as well as the command to execute for the default install.
I think a combination of (1) and (4) will be fine for Poetry.
As already said, extras can be covered via markers. (Not sure if every installer will understand it, but anyway.) I wonder if groups should not be considered at least? (Just the simple case where you do not need different markers per group.) Otherwise, each lock file (with all groups) from a pyproject.toml with groups will not be installer agnostic, correct?
Exactly. With the file format being versioned it should definitely allow for expansions in the future as experience tells us what should be expanded as something every installer should support.
Yeah, I realized after I posted that people were probably going to at least want the tool name, although you can at least partially infer it from whatever tool has a [tool] table.
Great! So that leaves us at:
Poetry:
uv: maybe (waiting to hear from @charliermarsh about the option (1) modification)
PDM: unknown
And Iām trying to find someone at GitHub from Dependabot and/or the dependency graph to tell me this idea works as an external validation.
Iām personally fine with that, but @charliermarsh seemed a little skittish on pushing too far out past what a requirements.txt file does today. But honestly, if we drop the custom markers then itās no different than saying we just came up with a way to squish multiple requirements.txt files together and I donāt think thatās controversial.
So I think I just talked myself into liking the idea. I will probably say there is a required āDefaultā group ā bikeshed on the name later ā so that itās always clear what should be installed by default. Whether we want to make it implicit when thereās just the one group is for a later discussion.
The spirit of PDM is to adopt Python packaging standards as much as possible, so it is also very likely that PDM will use the new lock file to replace pdm.lock.
I agree not to introduce groups into the new lock file standard, otherwise it will become too complicated. (1) + (4) too here but I tend not to put too many stuff inside.
My current stance is that we should pursue (1). requirements.txt is popular, and replacing it with a standardized, secure format would be impactful for users of all tools.
On the other hand, I donāt really see value in pursuing (2), (3), or (4) right now. Iām just not convinced that theyāre solving problems in a way thatās worth the costs. Specifically, what are the benefits? The fact that weāre not pursuing installer or resolver interoperability meansā¦
Dependabot will only partly work. Dependabot wonāt be able to put up PRs to upgrade your dependencies, since that will require tool-specific commands and knowledge. Thatās most of the value of Dependabot, at least in my experience. In the current framing, Dependabot would be limited to security advisories (unless Iām misunderstanding). So I donāt think these formats really solve the āfairnessā problem discussed above. Dependabot would still need per-tool support to be useful.
Users wonāt have to learn about poetry.lock, pdm.lock, uv.lock, etcā¦ Butpylock.toml wonāt actually be interoperable. So users will have to learn that despite using the same filename, theyāre not the same format, nor are they interchangeable between tools. I could actually see this being even more confusing for users than the status quo.
I understand the desire to standardize, but is diversity in this specific problem space actually harmful? Is it actually causing problems or confusion for users? I know this might be an unpopular opinion, but if we standardize on (2), (3), or (4), weāll be inherently placing constraints on these tools. Suddenly, weāll have to think about the interaction with the standard for every new feature that weāre pursuing (conflicting dependencies across extras and groups, locking of build constraints, etc.).
So, if we do (1), Iām confident that uv will fully support it. If we do (2), (3), or (4), Iām not sure. I suspect weāre more likely to stick with uv.lock.
Two addendums, thoughā¦
I do think (3) could be worth doing if we can solve the installer and resolver interoperability questions. But I donāt think itās feasible to do today. At least for us, the format, capabilities, and understanding of the problem are still evolving dramatically. I would prefer to revisit it in the future, even if that is not the ānear futureā.
I know that the line between (1) and (4) is fuzzy, but the way I might frame it is: I would prefer that we not put tool-specific metadata in the file.
Iām in favour of (1) + (4). I donāt think (1) removes the [tool] section from the specification, and I wouldnāt want it to. So all (4) is really adding is a way for tools to provide some metadata about whether the data in the [tool] section affects installability or not.
If thatās your preference, then even under (1) + (4), I believe that uv can simply not use the [tool] section, and refuse to import lockfiles that do use it.
Iām not at all clear why youād be unable to do with (1) + (4) exactly what you would do with just (1). In particular, if you can support the standard if itās (1), then surely you can support it just as well in the case of (1) + (4). Just treat it as if it were (1).
Maybe Iām misunderstanding (4), though. As far as I can tell, all (4) seems to add is the possibility to set installer-agnostic=false, and uv can just reject all such lockfiles unless they were produced by uvā¦
).
