The “Security Considerations” heading on the Direct URL page is now live, so PEP 751 can reference it directly when covering URL fields : Direct URL Data Structure - Python Packaging User Guide
(these paragraphs were already there, they just didn’t previously have their own linkable heading)
I made the same suggestion a jillion posts ago, but it’s really nice to see there’s an actual (potential) use case for it, rather than just my intuition or aesthetics.
I’m 60+ messages behind due to vacation, so perhaps this has been already mentioned, but if the proposal evolves in this direction, and it seems interesting to me, then would variants contribute to supported-environments and if so, how? It seems like they’d have to.
Also, could py3-none-any be a default wheel-tag so you DRY? Is there a use case for not including that wheel tag in a supported environment?
I dislike doing this, but have occasionally been forced to when tag-triggered CI cannot effectively be tested locally before the tag is pushed. CI fails, tag on the remote is deleted, bug is fixed (hopefully) and pushed, and new tag is created. Yeah, it’s a short lived bad tag, but there is a window of opportunity.
This also mirrors at least the way I think about dependencies inside of hatch environments, which seems like a nice parallel.
40 posts still to catch up on, I really like the way you’re framing this here @ncoghlan.
They are for extras, but PEP 735 dependency groups we don’t know yet.
I intended the latter since you would be asking for a specific lock, and extras may not be independently composable as they might have conflicting version requirements (@pf_moore brought up not wanting a consistent version like what uv currently does).
That looks like what I have in PEP 751, but w/ expanded metadata and support for dependency groups. Is that a fair assessment?
It should be 1:1 and I should have used name in the latter. 
Filename; package name and version get you to the appropriate entry for [[packages]], and then file name gets you what you need in packages.files.
That’s the intent!
I would expect the user would decide what they want from the locker.
If an installer chooses to not follow your guidance, then sure. But then you shouldn’t use that installer either.
You might trust “deployment platforms” to provide such a config more than I do. 
I more expect them to just use some installer already in use by the community.
But do note that Charlie later said …
I don’t see how they wouldn’t, but w/o knowing how they are going to be implemented it’s hard say “how” ATM.
It’s redundant across all environments? But you also haven’t gotten a PEP accepted saying Python 3 is the last major version, so it might be useful some day? I honestly think a redundant tag is not a big deal.
OK, I’m going to start forcing folks to express their views as I’m getting conflicting feedback and this whole endeavour is pointless w/o buy-in. So, the first question is should we have single file format which works for open-ended environments and specific environments, or have two separate file formats (they could build off of each other).
“open-ended” in the sense of being a package lock but not a file lock? Or in only containing a partial specification (and so the installer has to resolve any additional dependencies required by packages that are listed)?
Correct, that’s what I’m thinking (although if they are separate files it potentially opens things up, but I don’t think anyone really wants to deviate that much from what’s been discussed).
I voted “single format” on the assumption that it won’t be permissible for tools to only implement part of the spec (e.g., we don’t allow an installer to claim PEP 715 support but only implement file locks). But I’m not sure if that was how you intended it to be read.
Probably not in practice, but I guess we’ll have to see how PEP 2026 shakes out!
Fair enough, of course. I just want to keep variants in the mix because of the (in my mind) cross-connections between the proposals.
In some sense though, for exact file locking, taking variants into consideration would be a job of the locker, and the resulting fully resolved environment may not be portable, e.g. because you transport the lock file to a machine with a different GPU. It might still be installable, but it would likely be broken in some fundamental way at run time. So then you get into package locks (I think) where variant resolution has to happen at install time and the locker has to leave enough flexibility in the lock file to do variant resolution at install time.
But anyway, like I said, this is overlap just has to be either handled at some point, or explicitly punted, once variants (and to some extent, lock file) proposals are finalized.
I think synthesised group names can work, but if that’s the intent, I’d advise against reusing the full “dependency groups” name (since those are composable, with no certainty that any given combination is valid until you try it).
Maybe “resolved groups” for the synthesised groups that correspond to resolved environments?
However, I suspect a lexically ordered array of group names would be simpler overall than defining rules for synthesising names for different input group combinations.
I think that’s the right way to read it. A file lock only installer would at best be able to claim partial support, and I suspect we’re more likely to see opt-in --require-resolved-environment CLI flags that some deployment platforms enable unconditionally than we are dedicated installers that are outright incapable of handling package locks.
@brettcannon I think I have found my preferred colour for that bikeshed: require-resolved-environment = true rather than the double negative in unresolved-environments-allowed = false
I would happily use an installer which:
unresolved-environments-allowed is truepackages), verifying hashesI think many would trust the reliability of such a simple app.
Yeah, there’d be no problem with such client apps existing, they’d just be expected to qualify their support of the lock file standard as being limited to fully resolved locks.
The problematic UX scenario @pf_moore is worried about is platforms that only support fully resolved locks claiming that they support “Python lock files” (with no caveats), and then users of those platforms complaining to the various Python packaging related open source projects when their open ended locks don’t work with that platform (rather than either adding the missing fully resolved lock, or asking their platform provider to support open ended locks).
Not sure what “expanded metadata” means exactly but yes, I think it is pretty much what is currently in PEP 751 (+ dependency groups), but it is different from the idea in PEP 751: lock files (again) - #120 by brettcannon. From a package-lock perspective (i.e. unresolved-environments-allowed=true in the new proposal), I prefer the current state of PEP 751 to the new proposal, which I think will not work in all cases without backtracking at install time.
I think I’m advocating for using “package” and “file” instead of “name”. That way when you’re reviewing a lock file diff you don’t need to know where you are in a deeply nested structure (with GitHub’s customary single-digit number of lines of context) to know what you’re reviewing? Either way this is a minor point.
Indeed I would be against removing the unresolved-environments-allowed flag from the standard for this reason. As a user I want to be absolutely sure that what I reviewed is what gets put into production, and if that means a tool errors out along the way because it doesn’t implement that behavior then I want to know that, platform config or not.
Yeah, I would strongly prefer that we settle on a format that doesn’t require any sort of resolution at install-time. I’d be happy with either a flat list of package-versions (with markers), or a graph of nodes with dependencies that you can unambiguously traverse. Without this guarantee, the output resolution for a given lockfile becomes implementation-dependent and could vary from installer to installer.
I want to speak to what I see as one of the biggest risks to this PEP, which is: we don’t yet have a robust, well-proven tool that can produce the kinds of resolutions that this lockfile is designed around – i.e., a tool that both (1) does not require a resolution at install-time, and (2) can handle disjoint dependency graphs.
(I mean no disrespect here, and authors of other tools should correct me if I misrepresent their work.)
To skip ahead: I would love to see a real-world implementation of the Package Locking proposal prior to a decision on the PEP. And, as someone that wants this proposal to succeed, I’m happy to put my money where my mouth is by implementing it in uv once we have rough consensus on the schema.
(N.B. What follows is a longer explanation on why I consider this is so important – skip to the end if you find this uninteresting.)
Without a working implementation, we run the risk of standardizing something that doesn’t actually work. And I think that risk is non-trivial. We, as an ecosystem, are still figuring out how to resolve for these Package Locking environments – at least, in the form envisioned by the PEP. I’m not surprised that @frostming called it “impossible or extremely difficult to do correctly”.
Concretely, this proposal draws on a few references:
mousebender, which performs File Locking but not Package Locking.
PDM, which does generate a Package Locking resolution (with marker propagation), but has limited support for resolving disjoint dependency graphs. For example, PDM can resolve this:
anyio > 3 ; sys_platform == 'darwin'
But it cannot resolve this:
anyio > 3 ; sys_platform == 'darwin'
anyio < 3 ; sys_platform == 'win32'
Which leads to:
ERROR: Unable to find a resolution for anyio
because of the following conflicts:
anyio<3; sys_platform == "win32" (from project)
anyio>3; sys_platform == "darwin" (from project)
That second set of requirements represents a step up in difficulty. (In uv, we refer to this general setup as “forking”, since we fork the resolver based on the disjoint markers and then unify those forks later on.)
Poetry, which supports the cases described above, but writes out a list of resolved packages without any markers, and then performs a resolution at install time.
Based on my own experience, I think it will be non-trivial for Poetry to migrate from the current solution to the PEP proposal. (I apologize if I’m off-base here, I don’t feel fully comfortable commenting on other tools but I’m doing my best to describe the state of the landscape.) [1] My understanding is that Poetry’s current approach has a higher tolerance for error, since it doesn’t need to guarantee that a single resolution is produced for every possible environment – only that a resolution is available for every possible environment. This too represents a step up in difficulty.
uv, where we’re working on a resolver that can solve these disjoint dependency graphs (as in the anyio case above), without requiring a resolution at install time. And those goals are well-aligned with the motivations of the Package Locking proposals.
We’ve built this, it’s public, and early adopters are using it, but it hasn’t been “announced” or proven in the wild – so it likely has its own problems! It is capable of resolving highly complex dependency graphs with hundreds of package nodes… but we don’t yet know where it will fall over. Just last week, we shipped a bunch of significant changes to the lockfile format based on new information from early testing.
In building uv’s universal resolver, we’ve also had to confront a bunch of hard problems that we didn’t anticipate in advance. Here’s an example…
In order to generate these kinds of resolutions (especially when “forking” is required, as in the anyio case), you need to do a lot of algebra on marker expressions.
For example:
anyio case above). I believe this is actually an NP-complete problem, equivalent to SAT (checking if A and B are disjoint is equivalent to checking if A and B is not satisfiable).We started off doing these manipulations naively (just OR and AND), then made a series of changes to perform marker normalization and simplification (1, 2, 3, 4).
Even with all of that, for complex cases like “Transformers with all extras enabled”, we ended up with marker expressions that weighed in at 10s of KBs per dependency (5). They were just ridiculously long. In the end, Ibraheem (from our team) implemented an entire SMT solver (6) to support robust simplification of marker expressions and disjointness checks. It’s a hard problem!
I’m happy to talk more about the marker problem (maybe we’re over-thinking it), but I mostly intend for it to serve as an example of something that we didn’t anticipate ahead of time…
My feeling from the above is that this isn’t yet a solved problem. So I’m wary of standardizing on an output format for something that we’re still trying to understand…
For me, the solution is to ensure that we have at least one working implementation prior to a decision. (I recognize, of course, that this is not my call.)
If we can pass uv’s current test suite with the proposed format, and continue to pass whatever additional tests we add over time once the resolver is released, I would have far more confidence in whatever we decide on here. And I’m personally willing to put in the time to see that through.
I know there’s a draft PR for this, and I did run it through some of our test cases prior to posting here. ↩︎
As PEP delegate, I would almost certainly require this in some form. That might mean simply provisional acceptance of the PEP subject to implementation of at least one locker and one installer, although I’m not particularly comfortable with doing that (I was burned by PEP 708, which is still in limbo 
)
I had been assuming that “simple” implementations (basically, file locking) would be sufficient to prove the specification, but you make a strong case that package locking is where the problems are most likely to lie.
At this point, assuming we get consensus, I’m not sure what the best way of handling the PEP would be:
While it’s ultimately my call on how to handle PEP acceptance, I’d be very grateful for community feedback on what works best for people here.