# PEP 751: now with graphs! **URL:** \ **Category:** Standards\ **Created:** [October 30, 2024, 10:10pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721 "2024-10-30T22:10:37Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 1, 2024, 2:50pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/21 "2024-11-01T14:50:54Z")
> [@charliermarsh](#): > > I tend to view this as “okay” in the context of uv I agree - I consider it as acceptable in the context of pip as well. But the PEP describes the operation as “syncing a pre-existing environment to match the lock file” and I’m not comfortable with claiming that pip’s behaviour corresponds to that description. So it sounds like neither pip nor uv (in its `uv pip install` form) will implement either of the SHOULD requirements of the PEP. @brettcannon does this affect your thinking on those two points? ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 3, 2024, 9:49pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/22 "2024-11-03T21:49:14Z")
I’ve had a detailed read through and see two issues detailed below as well as an odd duck I feel uniquely qualified to call out as such. > [@brettcannon](#): > > pants: @anon62990384 Side note: I only speak for Pex. Pants is a big user of Pex lock files, but I can’t speak for Pants. # Specification - File Format ## packages.source-tree IIUC both of these can be locked: 1. “foo @ file:///this/loose/project/here” 2. “foo @ git+https://github.com/org/repo@\” Right now 2 is ~directly addressed with mention of the `@` being required to be an immutable commit (no tags or branches). How does one hash source trees like 1? This is how Pex does it, and it does the same for VCS requirements which lifts the immutable commit restriction, allowing tags and branches as well: ```python source-tree -> build sdist -> unpack -> standard for hashing a directory ``` ## \*.url As far as I can tell, there is no facility for dealing with `file:///right/here/a.whl` right now. A common real world case I’ve run into: A `--find-links` repo is part of lock and that find links repo is an NFS mount on dev machines in their home dir. Each dev has a different home dir and each may have the repo using the lock checked out at an arbitrary location in relation to their home dir. Pex uses “path mappings” to handle this. WHen creating the lock you say `... --find-links /right/here --path-mapping "FL|/right/here" ...` and “file://${FL}” is then used as a placeholder in the lock file in place of all occurences of “file:///right/here”. To use the lock file later, you must specify `--path-mapping "FL|/over/there"` where `/over/there` is the appropriate path on the install machine hosting the find links repo. ## Expectations for Lockers > Lockers MAY want to provide a way to let users provide the information > necessary to lock for other environments, e.g., supporting a JSON file > format which specifies wheel tags and marker values. This feels like it comes out of left field in this spec. Perhaps worse, it goes so far as to soft-specify things including format and fields! ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 3, 2024, 10:58pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/23 "2024-11-03T22:58:57Z")
Sorry - forgot about this one: # Specification - File Format ## hash-algorithm This seems like big trouble for a universal lock with multiple indexes where the indexes use different hash algorithms. Say the lock gets all the big boys from the index with the un-desired hash algorithm. This means the locker must download 10s of wheels per locked big boy just to re-hash them to the single hash-algorithm. I’m thinking `torch` + `nvidia-*` here. This is alot of downloading! ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [November 4, 2024, 3:37am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/24 "2024-11-04T03:37:38Z")
> [@pf\_moore](#): > > And it means that someone who simply wants to audit a lockfile to check what’s going to be installed, will have to reverse-engineer the pseudo-code to do so. This is my concern as well. It might be mitigated by the fact that I should be able to make a lock file where everything in `[[packages]]` is installed, provided: - only one entry in `[[groups]]` - this group entry has `requirements`, not `project` (as the latter must support all extras) - no requirement specifies `marker` It’s this correct? If so, can that be stated explicitly in the PEP as an invariant? It would simplify my lock file reader. In addition, this boils down to the file-lock from the previous proposal of there’s only an `sdist` or one entry in `wheels` for each package. This is my primary use case for using lock files. * * * The PEP isn’t too explicit about this, but the requirement to list each package’s requirements from `Requires-Dist` metadata makes the package entry very verbose as it includes requirements for extras which aren’t even referenced. This is obvious in the PEP’s example lock file. Is it possible to allow lockers to drop requirements for unused (especially unreachable) extras? This would make the lock easier to visually audit. * * * Having `version` in requirements be a version specifier means that installers and lock file readers need functionality to parse and compare version specifiers. It makes sense to support multiple package entries across different platforms with the one requirement entry, but it does mean I will have to use `packaging` to read the lock file. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 4, 2024, 2:52pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/25 "2024-11-04T14:52:48Z")
> [@pf\_moore](#): > > And it means that someone who simply wants to audit a lockfile to check what’s going to be installed, will have to reverse-engineer the pseudo-code to do so. (Aside: in uv we ship tooling to make this easier. You can run `uv tree` or even `uv tree --python-version 3.11` to view the packages that would be installed for a a given Python version or platform, etc.) > [@EpicWink](#): > > It’s this correct? If so, can that be stated explicitly in the PEP as an invariant? It would simplify my lock file reader. If this is allowed, then we need some way for the lockfile to encode the platforms that it supports. Otherwise, users can no longer assume that a successful install from a lockfile gives them the intended set of packages for the current Python platform. (Aside: we record this in uv with an `environments` field, which we validate against at install time.) ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 4, 2024, 10:35pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/26 "2024-11-04T22:35:15Z")
> [@charliermarsh](#): > > we do treat entrypoints differently than other packages in uv, since we fully resolve _all_ of their extras (though we don’t write their lockfile entries any differently – we don’t record them as roots in any way). OK, so we seem to be aligned in general on that then. That’s actually why I had the difference in project and dependency groups in `[[groups]]`; to signal which packages had all of their extras resolved. But here’s a question: I have only ever seen a single `uv.lock`, which suggests everything goes into a single file. So, how do you encode potentially different results when using `uv lock --resolution`? E.g., if I want to have my CI test both the newest and oldest versions of my dependencies, is that all in one file, and if so how are the packages separated from each other? > [@charliermarsh](#): > > we key off name, version, and source. The source could be a registry index, a Git URL, a direct HTTPS URL, or a local file path. If I remember correctly you and @pf_moore had a back-and-forth on that since it’s a bit different than how the rest of packaging does things. I think to make such an approach work you would need the dependencies/edges embed the source part as well as the version (the name is inherently taken care of). Very quickly thinking about this, you could either create a unique key per item in `[[packages]]`, have a bunch of different keys in the inline table to represent the possible source types, some single key that specifies the type and then option details in other keys to help differentiate, or you don’t use an inline table for dependencies and you use an inline table for the source type details. I’m not sure how uv does this or what Poetry/@radoering had in mind when they mentioned they like this idea. > [@charliermarsh](#): > > > [@brettcannon](#): > > > > Not if you skip extras that you didn’t lock for. Let’s say you lock for `spam[fast]`, but skipped `spam[compat]`. This would be recorded in `[[groups]]` by only recording the one case. But if you left that detail out and tried to enter the graph at `spam`, you would have to walk the graph first to know that the lock file won’t succeed in your case. > > Can you expand on this a little bit? What is the “one case” that gets recorded? Is there a group for each extra, for each package? I’m just trying to square this with the [pseudo-code](https://peps.python.org/pep-0751/#pseudo-code). The extra that’s used. > [@charliermarsh](#): > > Both projects define their own dependency groups. I want users to be able to install: > > - `root` > - `child` > - The `test` group in `root` (with or without `root` itself) > - The `lint` group in `child` (with or without `child` itself) > - `root` with the `stubs` extra enabled > - `child` with the `async` extra enabled > > Is [this](https://www.notion.so/astral-sh/PEP-751-13148797e1ca805ba97cce83af9a544a?pvs=4) roughly the correct representation for that setup? Yes, I believe so. > [@pf\_moore](#): > > In the specification of `[locker]`, there’s a statement > > > Enough details SHOULD be provided such that the lock file from the details in this table (provided the same I/O data is available, e.g., Dependabot if only files from a repository is necessary to run the command). > > This seems incomplete. Should it be “the lock file **can be reproduced** from the details…”? Yep! > [@charliermarsh](#): > > > [@pf\_moore](#): > > > > I imagine pip would _not_ make sdist usage opt-in. > > Ah I missed this – I don’t think we’d do this either. Or at least, I find it surprising, and I don’t see how it would mesh with our interfaces. That’s actually been in the PEP since the start to avoid using what the newly built wheel lists as dependencies and thus going outside the lock file or unexpected results (as well as pushing wheels more, having less surprises when an sdist build fails, etc.). Anyway, it’s a “SHOULD” for a reason. But if it truly bothers people that much I’m open to taking it out as long as the PEP somehow says that tools should not ask the sdist at install-time what to install and installers fully rely on the lock file. > [@ofek](#): > > Hatch was going to have a lock file per environment. Would someone mind giving a TL;DR about why this is no longer supported? To be clear, it’s in the “deferred ideas” section, not rejected. And it’s because I think groups can handle a similar use-case as multiple files. But we can bring it back if others want it. > [@pf\_moore](#): > > Looking at the pseudo-code for installers, it feels rather complex, but if I assume you’ve got it correct, I don’t have a problem with pip implementing it. It’s a direct copy-and-paste from my proof-of-concept, so it seems to work. 😁 > [@pf\_moore](#): > > I also don’t think we’d want to commit to supporting syncing an existing environment. Sure, although that came from you. 😉 I think you’re reading “sync” differently now than when you suggested I put that in. You originally meant for “sync” to mean" delete everything in the environment and then install from the lock file", i.e. it’s like a new environment, just without the environment creation step. I can clarify that’s what the statement is meant to mean. I believe uv calls this `uv sync`. > [@pf\_moore](#): > > So it sounds like neither pip nor uv (in its `uv pip install` form) will implement either of the SHOULD requirements of the PEP. @brettcannon does this affect your thinking on those two points? See above. > [@anon62990384](#): > > How does one hash source trees like 1? You don’t as there’s no standard way on how to hash a directory of files. > [@anon62990384](#): > > there is no facility for dealing with > `file:///right/here/a.whl` right now. [PEP 751 – A file format to record Python dependencies for installation reproducibility | peps.python.org](https://peps.python.org/pep-0751/#packages-wheels-path) covers that case. > [@anon62990384](#): > > > Lockers MAY want to provide a way to let users provide the information > > necessary to lock for other environments, e.g., supporting a JSON file > > format which specifies wheel tags and marker values. > > This feels like it comes out of left field in this spec. Perhaps worse, > it goes so far as to soft-specify things including format and fields! Suggestions are allowed in PEPs and it clearly says it’s an idea, so there is no soft “spec” here. It might seem slightly odd because it came from when there was more concern about locking for other platforms. But I also don’t want to take it out as I don’t want to start people worrying about it either. Plus some people liked the idea, so I can’t win here. > [@anon62990384](#): > > ## hash-algorithm > > This seems like big trouble for a universal lock with multiple indexes where the indexes use different hash algorithms. If you go back a few hundred messages on a previous version of this PEP you will find the discussion on this one; it’s deliberate and consensus was reached on it. > [@pf\_moore](#): > > the lack of a “plain text” description of how to interpret a lockfile means there isn’t anything to check the pseudo-code against. As in you want a bullet list of the expected steps an installer would go through? I can add that in once I think we aren’t going to be making massive changes. I primarily skipped it because I needed the pseudo-code to make the PoC work anyway, so it was easier to reuse that than write the same concept in two different forms. > [@EpicWink](#): > > I should be able to make a lock file where everything in `[[packages]]` is installed, provided: > > - only one entry in `[[groups]]` > - this group entry has `requirements`, not `project` (as the latter must support all extras) > - no requirement specifies `marker` > > It’s this correct? Yes, as long as you didn’t artificially make that be the case (i.e. you didn’t strip out markers). > [@EpicWink](#): > > can that be stated explicitly in the PEP as an invariant? I guess, but I don’t know how that helps much. > [@EpicWink](#): > > The PEP isn’t too explicit about this, but the requirement to list each package’s requirements from `Requires-Dist` metadata makes the package entry very verbose as it includes requirements for extras which aren’t even referenced. This is obvious in the PEP’s example lock file. > > Is it possible to allow lockers to drop requirements for unused (especially unreachable) extras? This would make the lock easier to visually audit. I went back and forth on this one. I think I’m okay with having the PEP say dependencies for unused extras in the lock file MAY be left out for brevity. > [@charliermarsh](#): > > > [@EpicWink](#): > > > > It’s this correct? If so, can that be stated explicitly in the PEP as an invariant? It would simplify my lock file reader. > > If this is allowed, then we need some way for the lockfile to encode the platforms that it supports. Yeah, you can’t strip out markers you don’t use, else we are back to listing upfront all known platforms that a lock file supports (which was a contentious bit when it came up due to how to appropriately represent the support, handling the boolean logic, etc.). ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 4, 2024, 10:51pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/27 "2024-11-04T22:51:43Z")
> [@brettcannon](#): > > > [@anon62990384](#): > > > > How does one hash source trees like 1? > > You don’t as there’s no standard way on how to hash a directory of files. Ok … but the spec doesn’t say you can’t have those sorts of source trees in the lock fwict; so you must either outlaw those sorts of source trees or define the standard for hashing a directory. Pex choses the latter. > [@brettcannon](#): > > > [@anon62990384](#): > > > > there is no facility for dealing with > > `file:///right/here/a.whl` right now. > > [PEP 751 – A file format to record Python dependencies for installation reproducibility | peps.python.org](https://peps.python.org/pep-0751/#packages-wheels-path) covers that case. It refers to: ###### [`packages.sdist.path`](https://peps.python.org/pep-0751/#packages-sdist-path) - Required if `url` is not set - String - A path to the file, which may be absolute or relative. - If the path is relative it MUST be relative to the lock file. Which decidedly does not cover that case. An absolute path makes no sense in a lock file in the general case and yet the `--find-links` example I explained is very real and used by users of Pex locks. To record the absolute path of the find links repo as it happens to be on the locker’s machine leaves folks wanting to use the lock with a broken lock. > [@brettcannon](#): > > Plus some people liked the idea, so I can’t win here. Well, I clearly like the idea - it’s [~“mine”](https://github.com/pypa/pip/issues/10050#issuecomment-1101429422)! But … ok. I fundamentally have a problem with the PyPA being a standards-creation-mill for front ends instead of working towards 1 front end, and I think that frustration is just niggled by this seemingly gratuitous spec sprawl. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [November 4, 2024, 10:56pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/28 "2024-11-04T22:56:53Z")
> [@brettcannon](#): > > Anyway, it’s a “SHOULD” for a reason. But if it truly bothers people that much I’m open to taking it out It doesn’t bother me as such, it just seems odd to say that installers shoud do something when there’s a clear indication that none of them will… > [@brettcannon](#): > > as long as the PEP somehow says that tools should not ask the sdist at install-time what to install and installers fully rely on the lock file. I don’t know what that statement means. Installing a sdist means running the build backend to generate a wheel and then installing that wheel. Are you trying to say that the installer should ignore the dependencies of the wheel generated by the sdist? That seems obvious to me (in the sense that it’s what we do for wheels as well) but sure, that’s OK (assuming it’s worded a bit more clearly in the PEP). > [@brettcannon](#): > > I think you’re reading “sync” differently now than when you suggested I put that in. You originally meant for “sync” to mean" delete everything in the environment and then install from the lock file", i.e. it’s like a new environment, just without the environment creation step. I can clarify that’s what the statement is meant to mean. Ah, yes, I’m OK with that - but I definitely wouldn’t describe it as “syncing” (which I feel has too many other implications). We’ve moved a long way from a lockfile being a way to replicate an environment at this point, so I’m not sure how useful it would be to have a specific option to uninstall everything not in the lockfile\[1\]. I feel like some form of `pip uninstall-all` command might be better. Also, there are some tricky design questions - would syncing\[2\] remove pip and/or uv from the target environment? > [@brettcannon](#): > > As in you want a bullet list of the expected steps an installer would go through? I can add that in once I think we aren’t going to be making massive changes. Basically, yes. But feel free to leave it until things are settled down. Ultimately, my point is that I don’t think we should rely on implementation-defined behaviour, even if the (reference) implementation is in the specification 🙂 * * * 1. and I’m not sure how easy it would be to implement in pip - scanning a potentially large environment just to uninstall everything is potentially costly 2. I’ll use that term for now, until we have something better ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [November 4, 2024, 11:05pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/29 "2024-11-04T23:05:25Z")
> [@brettcannon](#): > > > [@EpicWink](#): > > > > can that be stated explicitly in the PEP as an invariant? > > I guess, but I don’t know how that helps much. > [@brettcannon](#): > > > [@charliermarsh](#): > > > > If this is allowed, then we need some way for the lockfile to encode the platforms that it supports. > > Yeah, you can’t strip out markers you don’t use, else we are back to listing upfront all known platforms that a lock file supports Hmm. I think I would be free to create a tool which filters a lock file for a single platform (and record that platform in the `[tool.my-tool]` table, for documentation). It would remove all platform (maybe more?) markers and then dangling packages, and unmatching wheels. This would allow me to have a lock file with the precise files to be installed. This too would have to ensure the same guarantees as a locker, but also guarantee that the lock file can be scanned linearly when visually inspected (by a human). * * * > [@charliermarsh](#): > > in uv we ship tooling to make this easier. While this tool sounds fantastic and can help with quickly reviewing dependencies during development, I don’t want to rely on any tool (including any I write!) for security auditing, as the tool may have bugs (eg omitting a package which would be installed). I suppose `cat` and `diff` may have bugs but that’s less likely. I also suppose using the same tool which outputs what would be installed, and which would actually perform the install, makes sense. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 4, 2024, 11:09pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/30 "2024-11-04T23:09:41Z")
> [@brettcannon](#): > > But here’s a question: I have only ever seen a single `uv.lock`, which suggests everything goes into a single file. So, how do you encode potentially different results when using `uv lock --resolution`? E.g., if I want to have my CI test both the newest and oldest versions of my dependencies, is that all in one file, and if so how are the packages separated from each other? Yeah we don’t support this right now. You’d have to, like, create two separate files and rename them back to `uv.lock` whenever you want to use them (obviously bad). I do think that locking for two “lowest vs. highest” is a bit different than locking for “a project alongside a set of dependency groups”, because in the latter case, we actually want to resolve the whole thing as a single cohesive set of dependencies (at least in uv). Like if your dev dependencies and production dependencies both depend on `packaging` somewhere, we’d like to use the same version if we can. > [@brettcannon](#): > > Yes, I believe so. (This is in reference to the [example lockfile](https://www.notion.so/astral-sh/PEP-751-13148797e1ca805ba97cce83af9a544a?pvs=4) I constructured.) I think the challenge for me here is: what would the CLI look like to install the `test` group from the `root` project? Today in uv, that’d be like `uv sync --package root --only-group test`. Under this standard, under the hood, we’d then construct `"root~test"` (which would be a uv-specific thing) to look up the right group. That seems fine so far, but how would a Poetry user install that same group from the uv-generated lockfile? The mapping from group to name is sort of “proprietary” to the tool. What if Poetry calls those groups something else? Would we be required to expose a CLI like `uv sync --group "root~test"`? That could work as an escape hatch (I think it’d be a step down if that were the _only_ way we exposed groups in the CLI), but it kind of hurts the interoperability. I don’t know what a good solution looks like here. I could imagine a solution whereby we get rid of the `name` field on `[[groups]]`, and a group is either a `project` (unique) or a dependency group of a `pyproject.toml` (like a file path combined with the group name). So we encode those concepts directly into the spec, rather than making it flexible. But then we lose the ability to do things like “include a resolution for the lowest and highest versions of each dependency in the same lockfile”, which I believe is a goal right now. ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 5, 2024, 12:49am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/31 "2024-11-05T00:49:31Z")
> [@anon62990384](#): > > the spec doesn’t say you can’t have those sorts of source trees in the lock fwict I don’t know what “fwict” means. And you’re right, the PEP says source trees are allowed because when I tried to leave them out people didn’t like that. And it still doesn’t address the fact we don’t have a standard to hash a directory of files. You have: > [@anon62990384](#): > > `source-tree -> build sdist -> unpack -> standard for hashing a directory` But that doesn’t state how you got a hash from that. Are you hashing the sdist? > [@anon62990384](#): > > It refers to: > > ###### [`packages.sdist.path`](https://peps.python.org/pep-0751/#packages-sdist-path) > > - Required if `url` is not set > - String > - A path to the file, which may be absolute or relative. > - If the path is relative it MUST be relative to the lock file. > > Which decidedly does not cover that case. Sorry, I read too fast and misunderstood what you were asking for. It seems you’re wanting some way to have almost a placeholder file path or something relative to as-of-yet specified path that is only know at install time. So far no one has asked for that. If you want some wording to allow you to have relative paths anchored to something other than the lock file as specified by the locker I would be open to that. > [@anon62990384](#): > > An absolute path makes no sense in a lock file in the general case I would argue it depends on the purpose of your lock file. > [@anon62990384](#): > > I fundamentally have a problem with the PyPA being a standards-creation-mill for front ends instead of working towards 1 front end, and I think that frustration is just niggled by this seemingly gratuitous spec sprawl. I’m going to ignore this as it feels off-topic and a bit of a dig at me (although I don’t think you meant for it to be taken that way). > [@pf\_moore](#): > > Are you trying to say that the installer should ignore the dependencies of the wheel generated by the sdist? That seems obvious to me (in the sense that it’s what we do for wheels as well) but sure, that’s OK (assuming it’s worded a bit more clearly in the PEP). Yes, that’s what I mean. > [@pf\_moore](#): > > Ah, yes, I’m OK with that - but I definitely wouldn’t describe it as “syncing” (which I feel has too many other implications). I’ve already clarified the PEP. > [@pf\_moore](#): > > We’ve moved a long way from a lockfile being a way to replicate an environment at this point How so? It’s been a lot of talking, but the goal is the same to me. > [@pf\_moore](#): > > I’m not sure how useful it would be to have a specific option to uninstall everything not in the lockfile I believe the original discussion was to avoid having to fully delete an environment in case it was viewed as costly. I’m also fine in saying that “syncing” can entail deleting, recreating, and then installing into the “new” environment. But I’m also fine with taking that out. As I said, it wasn’t my idea to specify this in the first place so I’m not attached to it. > [@EpicWink](#): > > I think I would be free to create a tool which filters a lock file for a single platform (and record that platform in the `[tool.my-tool]` table, for documentation). It would remove all platform (maybe more?) markers and then dangling packages, and unmatching wheels. This would allow me to have a lock file with the precise files to be installed. Sure, I can’t stop you from doing anything. But that lock file will also not be portable to any other installer as it wouldn’t understand that the lock file can’t specify on its own what platforms are supported. > [@charliermarsh](#): > > I think the challenge for me here is: what would the CLI look like to install the `test` group from the `root` project? Today in uv, that’d be like `uv sync --package root --only-group test`. Under this standard, under the hood, we’d then construct `"root~test"` (which would be a uv-specific thing) to look up the right group. > > That seems fine so far, but how would a Poetry user install that same group from the uv-generated lockfile? The mapping from group to name is sort of “proprietary” to the tool. What if Poetry calls those groups something else? It’s tricky because I know you want to support multiple projects in a single lock file for your “workspaces” concept which often complicates things as the starting point is no then no longer singular thing but several possible things. It does explain why you want dependency groups attached to a package. But dependency groups also do not have to be specified in a `pyproject.toml` that defines `[project]`, so they can be independent concepts. I think to resolve this we need to decide what “self-contained” means for lock files since that’s why I introduced `[[groups]]`. Here some key scenarios for what the inputs into a locker to create a lock file: 1. A project and its extras (e.g., `[project]` from `pyproject.toml`) 2. A group of dependencies (e.g., `[depencency-groups]` from `pyproject.toml` or freezing what’s already installed in an environment) 3. A specific way to generate the lock file (e.g., oldest versions of everything, newest of everything) 4. What environments are supported And there can be combinations of any of these things. Now, given **only** a single lock file, what are people expecting to be able to do? Is the lock file supposed to tell the user about the inputs it was locked on upfront? Or does the user specify what they want to be able to install and “self-contained” in this instance simply means `pyproject.toml` isn’t required as long as the installer can be told what root(s) in the lock file to install for? And at what point do we say, “that requires a separate lock file” (and yes, that would require bringing multiple lock files back into the PEP which I’m fine with it that’s what we want)? The answers to those questions determines what we write down in a lock file beyond the packages that have been locked. ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 5, 2024, 12:59am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/32 "2024-11-05T00:59:11Z")
> [@brettcannon](#): > > > [@pf\_moore](#): > > > > I’m not sure how useful it would be to have a specific option to uninstall everything not in the lockfile > > I believe the original discussion was to avoid having to fully delete an environment in case it was viewed as costly. I’m also fine in saying that “syncing” can entail deleting, recreating, and then installing into the “new” environment. > > But I’m also fine with taking that out. As I said, it wasn’t my idea to specify this in the first place so I’m not attached to it. I went ahead and dropped this and just left installer must support installing into an empty environment. ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 5, 2024, 2:03am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/33 "2024-11-05T02:03:54Z")
> [@brettcannon](#): > > > [@anon62990384](#): > > > > the spec doesn’t say you can’t have those sorts of source trees in the lock fwict > > I don’t know what “fwict” means. > > And you’re right, the PEP says source trees are allowed because when I tried to leave them out people didn’t like that. > > And it still doesn’t address the fact we don’t have a standard to hash a directory of files. You have: > > > [@anon62990384](#): > > > > `source-tree -> build sdist -> unpack -> standard for hashing a directory` > > But that doesn’t state how you got a hash from that. Are you hashing the sdist? I meant FWICT (from what I can tell), the lock spec does not forbid locking source trees. And in order to lock a source tree, which may have ignorable garbage that should not affect the hash of the source tree (.pyc files, pytest caches, mypy cahces, etc), the way I found to get a reproducile hash was to 1st build an sdist, then unpack that (since the sdist tarball will generally have non-reproducible timestamps embedded), then hash the dir the sdist was unpacked to. For that I was not specifying the standard that Pex happens to use internally, I figured you’d come up with a standard as the PEP author. For the record though, Pex uses: 1. os.walk the unpacked sdist dir and collect all file paths in a sorted list. 2. hash the bytes of that sorted list of names, relativized to the sdist unpack directory, joined by “” and encoded in utf-8 3. update the hash with the hash of each of the sorted files contents in order. > [@brettcannon](#): > > It seems you’re wanting some way to have almost a placeholder file path or something relative to as-of-yet specified path that is only know at install time. So far no one has asked for that. If you want some wording to allow you to have relative paths anchored to something other than the lock file as specified by the locker I would be open to that. I just want to be able to handle the example I gave. That is, a lock who has at least some locked packages coming from a local file-system `--find-links` repo, and consumer of the lock who will have that find links repo be mounted somewhere other than the absolute path it was found at when the lock was created. The case I gave, `--find-links` repo somewhere under the user home dir, the code repo using the lock somewhere else, is, again, real and repeated across different orgs. How that is handled I have less concern over, but I gave the example of how Pex handles it today, with absolute paths being able to have placeholders in a lock file. ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [November 5, 2024, 3:11am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/34 "2024-11-05T03:11:29Z")
> [@brettcannon](#): > > But that lock file will also not be portable to any other installer as it wouldn’t understand that the lock file can’t specify on its own what platforms are supported. I’ve misunderstood something. I thought that if I have a lock file valid for only one environment (platform, OS, Python version, etc), then any installer will be able to use it, and all installers would install all packages listed in the file. What happens when trying to use this lock file in another environment, I’m not too concerned (I suppose I would prefer the installer to fail, eg if some package’s marker-conditional requirement isn’t satisfied, or a wheel’s platform tag doesn’t match). Also I’m ignoring sdists here (internally we build and host wheels for all of our sdists anyway). * * * I’ve noticed an incompatibility with our workflow: [sdist URL](https://peps.python.org/pep-0751/#packages-sdist-url) (or path) and wheel URLs (or paths) must be specified, if specifying sdist/wheels. Our file download locations are subject to change (and in fact, the query parameters which include authentication change basically every time). Is discovering the download location of an sdist or wheel (when provided an index URL) considered part of resolving? If so, we can’t use this lock file as we need to provide file hashes. If not, could we make URL/path optional (but mutually-exclusive)? ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 5, 2024, 6:43pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/35 "2024-11-05T18:43:14Z")
> [@anon62990384](#): > > the lock spec does not forbid locking source trees. Correct. > [@anon62990384](#): > > in order to lock a source tree, which may have ignorable garbage that should not affect the hash of the source tree Well, it depends on how stringent you want to be in the face of source trees. The fact you’re using a directory is already dangerous as you’ve pointed out. And getting a reliable hash of the contents would be a bit finicky. > [@anon62990384](#): > > the way I found to get a reproducile hash was to 1st build an sdist, then unpack that (since the sdist tarball will generally have non-reproducible timestamps embedded), then hash the dir the sdist was unpacked to. But step 1 – creating an sdist – is not consistent if the build back-end changes on you between hashing/creating the lock file and validating/installing (e.g., a newer build back-end version could change the order of the fields in `PKG-INFO`). How do you handle that? > [@EpicWink](#): > > I’ve misunderstood something. I thought that if I have a lock file valid for only one environment (platform, OS, Python version, etc), then any installer will be able to use it, and all installers would install all packages listed in the file. Yes, **IFF** the files can be installed on the platform and there aren’t any files missing for that platform. That’s why you need to record all dependencies per package even if you don’t lock for them; to detect when you are missing something and thus can error out. > [@EpicWink](#): > > What happens when trying to use this lock file in another environment, I’m not too concerned (I suppose I would prefer the installer to fail, eg if some package’s marker-conditional requirement isn’t satisfied, or a wheel’s platform tag doesn’t match). That’s currently what would happen. > [@EpicWink](#): > > Is discovering the download location of an sdist or wheel (when provided an index URL) considered part of resolving? You mean locking and creating a lock file (which is the only time resolving occurs by design)? I don’t know how else to make it work if you can’t find the files to lock against (unless you’re talking about something else like literally just freezing the state of a virtual environment so there is no searching). Technically, how the locker makes it work isn’t specified by the PEP. > [@EpicWink](#): > > could we make URL/path optional (but mutually-exclusive)? They are already mutually-exclusive (see e.g., [`packages.sdist.url`](https://peps.python.org/pep-0751/#packages-sdist-url) and [`packages.sdist.path`](https://peps.python.org/pep-0751/#packages-sdist-path)). And sure, we can make them optional. ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 5, 2024, 8:44pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/36 "2024-11-05T20:44:23Z")
> [@brettcannon](#): > > But step 1 – creating an sdist – is not consistent if the build back-end changes on you between hashing/creating the lock file and validating/installing (e.g., a newer build back-end version could change the order of the fields in `PKG-INFO`). How do you handle that? That means either pyproject.toml changed (you literally changed it to point to a new backend) or pyproject.toml did not change, but the constraints on the build-backend requires were loose enough to allow for such a change over time or there was no change at all and the build backend produces non-reproducible sdists. In all 3 cases Pex simply fails at lock install time with a hash mismatch as it should. An example of this 3rd case in action is found here: - [Plumb reproducible build env vars more thoroughly. by jsirois · Pull Request #2554 · pex-tool/pex · GitHub](https://github.com/pex-tool/pex/pull/2554) - [Hash validation with python dependencies from private git repo failing · pantsbuild/pants · Discussion #21145 · GitHub](https://github.com/pantsbuild/pants/discussions/21145) - [build: Fix build reproducibility. by jsirois · Pull Request #3712 · Unstructured-IO/unstructured · GitHub](https://github.com/Unstructured-IO/unstructured/pull/3712) In that case Pex locks caught a non-reproducible VCS requirement lock at a fixed commit sha. ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 6, 2024, 12:39am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/37 "2024-11-06T00:39:33Z")
> [@anon62990384](#): > > pyproject.toml did not change, but the constraints on the build-backend requires were loose enough to allow for such a change over time This is the case I’m thinking about. I don’t have insight into how common of an issue this is, especially as the [default semantics are unpinned](https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-build-system-dependencies-the-build-system-table). > [@anon62990384](#): > > there was no change at all and the build backend produces non-reproducible sdists I also don’t know how common this occurs either, but I suspect it isn’t as common as it could be since dicts started having expected iteration order. What do other people think of hashing source trees as Pex does? > [@anon62990384](#): > > - os.walk the unpacked sdist dir and collect all file paths in a sorted list. > - hash the bytes of that sorted list of names, relativized to the sdist unpack directory, joined by “” and encoded in utf-8 > - update the hash with the hash of each of the sorted files contents in order. Or do people not care about worrying about this (right now in this PEP)? ---
**Author:** ![anon62990384](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@anon62990384](https://discuss.python.org/u/anon62990384)\ **Post date:** [November 6, 2024, 12:54am UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/38 "2024-11-06T00:54:12Z")
> [@brettcannon](#): > > Or do people not care about worrying about this (right now in this PEP)? I think I’m a little confused. Isn’t it the case the PEP must either outlaw source trees in locks or else describe how they must be locked? If a lock has hashes for 99 packages but not for 1 source tree it seems to me the security guarantees of the lock are all for naught. For example, before Pex gained the ability to lock source trees, it would fail fast for any input requirements that were source trees explaining why the source tree requirements were not allowed. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 6, 2024, 5:59pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/39 "2024-11-06T17:59:58Z")
Personally I don’t think the PEP should require hashes for source trees, in part because the problem of “how to hash a source tree” seems outside the scope here – I feel like that would merit its own PEP entirely (though I’m fairly new to the PEP process so that may be wrong). I don’t think any of the other candidate users of the PEP compute source tree hashes today nor expect it to be a requirement in the format. > [@anon62990384](#): > > Isn’t it the case the PEP must either outlaw source trees in locks or else describe how they must be locked? If a lock has hashes for 99 packages but not for 1 source tree it seems to me the security guarantees of the lock are all for naught. Would it not be sufficient to: 1. Make “flag to (dis)allow source trees” an expectation for installers, like it is for source distributions. 2. In Pex, write these source tree hashes to `[packages.tool]`? I don’t think that would be a spec violation since others would still be capable of installing Pex-created lockfiles. ---
**Author:** ![charliermarsh](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/charliermarsh/32/10246_2.png) [@charliermarsh](https://discuss.python.org/u/charliermarsh)\ **Post date:** [November 6, 2024, 6:10pm UTC](https://discuss.python.org/t/pep-751-now-with-graphs/69721/40 "2024-11-06T18:10:29Z")
> [@EpicWink](#): > > Our file download locations are subject to change (and in fact, the query parameters which include authentication change basically every time). > > Is discovering the download location of an sdist or wheel (when provided an index URL) considered part of resolving? If so, we can’t use this lock file as we need to provide file hashes. If not, could we make URL/path optional (but mutually-exclusive)? I’m not sure I fully understand the first question, but personally I don’t think URL / path should be optional. There’d now be a requirement that installers support the Simple API to look up available distributions, whereas they don’t have that requirement right now. In some cases we have to expect workflows to accommodate the standard rather than the other way around. [Previous page](https://discuss.python.org/t/pep-751-now-with-graphs/69721.md?page=1) [Next page](https://discuss.python.org/t/pep-751-now-with-graphs/69721.md?page=3)