Preferable but not required ever (feel free to provide an example to the contrary) so I think the single lock file resolution approach is not worth the extra complexity.
Okay. What about Mypy? Thatās an example where you need to install a development dependency atop your projectās existing dependencies. Would you want one lockfile for the project, then a separate lockfile for the entire project tree plus Mypy?
(I think your comment need more nuance, I donāt fully understand what you mean by āsingle lock file resolution approachā ā does that include extras? Dependency groups? Multiple platforms? Resolver configuration?)
1 Like
For what itās worth, in uv, users can define groups that are known to be conflicting (i.e., mutually exclusive). So the entire project does not have to be lockable as a single unit. We solve them separately and then error install time if they try to install conflicting subsets.
1 Like
My ideal scenario would act upon the use cases themselves, which Hatch refers to as just environments. So if you have an environment for type checking then the resolution of that would have inputs as the cumulative project dependencies and Mypy. I think itās perfectly acceptable to have a newly resolved lock file have a newly released version of a transitive dependency which differs from another lock file/section, in exchange for the simplicity of this approach.
I mean basically the current version of this PEP as a whole compared to the previous draft.
1 Like
Okay. To clarify, though, what youāre describing also includes direct dependencies. So the dependencies you type-check against would not match the application dependencies at runtime, in the scenario youāre describing.
1 Like
The direct dependencies of all environments would match when the inputs themselves have pins like what one would do in requirements.txt. They would not match for libraries (e.g. using pyproject.toml) because hopefully they do not have strict pins. That is fine because libraries should not be using lock files generally speaking so they catch issues with dependencies before their users do.
edit: to be extra specific, a libraryās runtime dependencies should not be pinned but I do imagine setting explicit versions for formatters and type checkers and so those non-test environments would use a lock file
edit 2: now that I think more about libraries, type checking environments probably should be treated like test environments because you want to catch issues with newly released type updates before your downstream users do so perhaps that environment also should not have pinned Mypy/etc.
Okay. What youāre describing seems empirically out-of-sync with how these projects are managed today, at least with uv and Poetry. In my experience, it is not common for applications to use strict equality for their dependencies (and neither uv nor Poetry default to strict equality). It is also not uncommon for libraries to use lockfiles for development. I donāt expect you to agree. This may just be further evidence of misalignment that we need to work through.
3 Likes
Thatās quite intriguing. Just to be sure we are talking about the same thing, by application youāre thinking about something like a Django app right? If thatās the case then I am supremely surprised by what youāre witnessing because everywhere Iāve worked you would never be able to merge a PR for an application (not library) that added a dependency without a strict == pin.
1 Like
My experience is that. I mostly work on something closer to an application in that it generally is installed alone and not used as a dependency by another library. We strongly avoid == pin as they tend to lead to dependency conflicts and being manual resolver. We did have == more often in past and found it headache especially when our total transitive dependencies was close to 300ish (about 70-80ish top level dependencies total for monorepo app).
I do also expect my ci test/type checking dependencies to use same versions as without them. Iād be fairly disappointed if jobs we run fail but passed ci which used test lock due to version skew.
Today we handle this by first producing lock like file without test/type checking dependencies. Then we produce second lock file whose input is first lock file + test/type checking dependencies.
4 Likes
This seems contrary to the purpose of a lockfile? Why use one at all if your dependencies are already pinned? I think a major goal of lockfiles (and, historically, requirements.txt) is to manage pinning dependencies for you.
4 Likes
You would use them because of transitive dependencies, those wouldnāt be pinned in the input file.
1 Like
I see, thanks for explaining that. I still think this is contrary to the broad use of requirements.txt files ā as @mdrissi mentions, itās quite painful to pin direct dependencies and it seems like a stretch to say this is the norm.
5 Likes
Pretty sure there is some misunderstanding here. eg hatch would be an example of an application that does not pin its dependencies (which I suppose is not a surprise to you)
Thatās a fair point! However that is actually only because of the plugin system. Were it not for that then I actually would have strict pins (itās something Iāve thought about).
Iām completely out of my depth with all this, but I have been following and reading all the many hundreds of posts. Forgive me for chiming in!
Need this be true? Canāt a locker just lock whichever groups it wants, chosen via whatever API, and then the installer just needs to support choice from whatever groups it finds? Why does an installer need to know what groups were originally available? Why is a locker obliged to lock all of them? (Itās entirely possible Iām missing some technical constraint.)
For example Iām thinking uv or poetry could choose only to write each group after itās actually been synced by a user for the first time.
I have had a thought at times when following this thread, and it may be a complete non-starter but I figured Iāll mention it anyway.
There seems to be a tension between two camps, one wanting a simple set of instructions of what to install, and the other wanting some information on how things were resolved. What about if that information was available but was not in the lockfile and installers werenāt required to understand it?
My idea is essentially to
-
Not have
groups, or any indication as to whether the project itself should be installed, but to have arbitrarysetsinstead. They would subsume anything that is used to specify which edges should be followed other than things contained withinmarkers i.e. anything that is influenced by choices that users have made. -
Offload the information on how the locking was actually done, the instructions on how to lock, to
pyproject.toml.
Thing is, Iām kind of in @pf_mooreās camp here of thinking that an installer really shouldnāt have to care about anything, it should just be given a list of packages (or obtain one by navigating the graph) and install them. So with sets, installers donāt need to know anything about how the locking was done. To them, sets are arbitrary strings that indicate what to install. They either install the default set, or the user specifies one of the other available ones.
Then lockers can use them for whatever purposes they want, and have the keys be whatever they want. If in future lockers want to lock based on more things, more user options, installers donāt need to implement anything new.
I also figure it should be specified in the lockfile what the āentry pointā for each set is too. Then if a locker wants to create different sets with different entry points it can, regardless of the reason it wants to.
Things like uv, poetry and so on are naturally focused on how uv sync or equivalent would work i.e. making sure the lockfile and pyproject.toml and the installed environment all match, but these operations rely on the presence of a pyproject.toml file. So the idea is that what each set actually represents would be recorded in pyproject.toml.
How that would look in pyproject.toml I donāt actually know. The PEP could also have a go at standardising that too, or tools could be left to do their own thing for now and it could be standardized in future. But the point is that then weād have a standardized, future-proof, interoperable, lockfile that all installers will hopefully always be able to understand. And even if interoperability of āenvironment managersā like uv, poetry, pdm is then not a solved problem, nothing needs to be changed about lockfiles in order to solve it later on.
As I said, I have no expertise here whatsoever, and may have understood various things that have been discussed, so if this is a complete nonsense idea it can be readily dismissed.
2 Likes
No, but depending on the tool thatās whatās currently done (e.g., uv locks everything together so thereās consistency in the versions used).
As I have a lot going on in the real world (and itās positive, but stressful), Iām going to be rather occupied for the rest of the month. As such, donāt expect me to chime in heavily until December (I will try to chime in as appropriate and stay up-to-date, though). But I did want to quickly say two things.
One, donāt worry about one lock file versus several. Thatās a format detail that I think is driven by the use cases we choose to solve.
Two, I think we have to lines of thought about what the purpose of lock files are. One is locking for a specific scenario; historically this is what requirements.txt files did. The other is locking for any and all scenarios; Poetry and uv have done this by locking for all optional dependencies and dependency groups at once. Both are valid use cases, but they do lead to different design choices.
To use an example, a pyproject.toml has a couple of potential inputs into a lock file. Imagine you have:
[project]
name = "proj"
...
dependencies = [...]
[project.optional-dependencies]
ex = [...]
[dependency-groups]
dep-grp = [...]
The possible combinations here are:
projproj[ex](although you could skip installingprojin this case if you wanted to)dep-grpproj+dep-grpproj[ex]+dep-grp
The ālock for the scenarioā folks would say if you need a lock file for all of those combinations, then itās 5 lock files, but it might be less as you may only have e.g. 2 scenarios you care about and the lock file is simpler. The ālock for all scenariosā group would have one lock file covering all 5 possibilities, but obviously the lock file is a bit more complex.
Once again, I donāt think either group is wrong and both approaches have plenty of use in the community and happy users. But this is the key question being asked: are we trying to do ārequirements.txt 2ā or āuv.lock and poetry.lock and pdm.lock all convergeā? PEP 751 currently targets the latter case, but previously tackled the former case. As such, I canāt continue working on the PEP until we come to an agreement as to which approach we targeting. But once we do, then we can talk about goals, etc. within the scope of the approach we have chosen to go after.
8 Likes
I think this is a very good characterization of the two paths.
These two are also distinguished by āwhoā gives inputs to define a scenario. Either itās the locker or an installer:
# single scenario per lock
$LOCKER lock . --group docs docs.pylock.toml
pip install --lockfile docs.pylock.toml
# many scenarios per lock
$LOCKER lock
pip install . --group docs --lockfile pylock.toml
The scenario per lock is easier to use at installation time and easier to integrate into tox/nox/hatch as well as pip.
But if we start with a scenario per lock, I think we wonāt be able to evolve it to the point that it replaces the tool-specific lockfiles without a major revision to make it handle multiple scenarios. So I tend towards the unified lockfile in the current draft, on account of the fact that itās closer to what poetry, uv, and pdm already produce.
Some ideas and questions to bat around:
- should there be a way of encoding scenarios into the lockfile, for easy reference from installers?
- is there any value in having a default install scenario for a lockfile, which would be provided by the locker (e.g. current project + dev dependencies)?
- how do we expect the pip install interface to look? This question privileges pip but I think itās important to have some shared understanding of at least what it could be
1 Like
I think the other distinguishing feature is what constraints are on the lock. In the āsingle scenario per lockā case, thatās simple - thereās one scenario, it will be installed into one environment. Everything in the lockfile must be consistent.
In the āmany scenarios per lockā case, thatās missing. In theory, each scenario could be resolved independently. Even now, I remain unclear as to what the consensus is on this. Is the idea that the whole lockfile must be consistent, or is consistency on a per-scenario basis? In theory, thatās a locker UI question, but it matters for the file format in terms of whether the format needs to handle recording two different versions of a package because they are needed in different scenarios.
My concern as a user is that if I want to use one version of (say) Sphinx in my ādocsā scenario, and a different version in my ātestā scenario[1], will I be allowed to do so? On a couple of occasions, people have (in effect) claimed Iām wrong to want this - a recent case was here:
(Sorry Charlie - I know you said āmost of the timeā there, and therefore youāre not making a hard and fast rule, but I wanted to call out the implied assumption anyway, and yours was simply the first comment I found that made my pointā¦)
Maybe Iām misunderstanding, and thereās more flexibility in the format than there is in (for example) uvās user interface. If so, then thatās fine. But I do not want a format that prohibits users from doing what they want, even if itās not the preferred workflow of the currently popular workflow tools.
For example, when building the docs I want the latest version, but in my tests I want a specific version that triggers a bug Iāve fixed and have a regression test for ā©ļø
Just to address this concretely ā you are allowed to do so in uv (but not in Poetry, I believe). We agree people should be allowed to do this. Itās just not the default user experience weāre trying to provide.
Charlie mentioned this here and the documentation on conflicting groups is here.
1 Like
I believe your are correct about it not being supported in Poetry today. It prevented me from using Poetry in a project as recently as last year. But Iām not sure how Poetry handles extras, which seem to me to be capable of producing similar scenarios.
I know this thread isnāt specifically about locker interfaces, but I think uv is correct to treat this as a more advanced use case. IMO itās okay for the default experience to be consistent versioning of dependencies across a lockfile, and for handling of conflicts to require some more care from the user.
I would like the installer side of this to remain simple, however. Suppose I have two conflicting dependency groups: test and test-mindeps. How does that data get rendered in the single lockfile variant? Is it trivial easy enough for an installer to detect a conflict when the two are requested in a single command?
Also, a question about uv today: does the lock contain a complete resolution for each mutex group, pairwise with all other groups? i.e. Is there a separate resolution for docs,test alongside docs,test-mindeps, and likewise for any extras, etc?
1 Like