PEP 751: now with graphs!

[brettcannon]

@dhduvall

The CI build does a non-editable install of the project into the venv, and the venv is copied wholesale to /app.

Then that’s happening outside of an installer already, so I don’t see how that project’s code participates beyond you somehow generating a list of dependencies to lock against.

@prophile

If a decision about it can be made at the time of generating the lockfile, would it be enough for tools to just (“just”) allow not including the root project itself without other changes to the format?

It depends on scope because things like uv’s workspaces (I believe) need projects listed when they are a root in the dependency graph in case they are relied upon in some other way. It also lets you lock for extras for a project since it’s represented in the lock file ATM.

@charliermarsh

@brettcannon

So going back one revision of the PEP to the “set of packages” approach like PDM?

Something like that, yeah. A flat list of packages, each with a marker to indicate when it should be included, would make sense to me. A lot of things get simpler if we narrow the scope – for example, I don’t think workspaces are relevant? (Users could just export a separate lock for each member.) So multiple entrypoints is gone, etc.

Yeah, so it’s basically what the state of things were in mid-September and before this entire topic (and to be frank, roll back changes made for uv’s benefit).

@charliermarsh

@brettcannon

The other potential option is to stop thinking about source code and instead think about source locations.

Yeah, I think something like this is reasonable. IIUC, it’d mean that we wouldn’t necessarily record versions in the lockfile for source trees?

Correct, it would either be optional or entirely left out. Pro of optional is if you want an editable version to line up to detect changes and thus maybe signal you need to recreate the lock file, con is you would then need UX to make the version optional sometimes.

@charliermarsh

I think installers would still need to compute and “validate” versions at install time, since given a source tree that builds a package foo, it’s valid for other packages (even from remote registries) to declare requirements like foo==1.0.0. If some package in the lockfile requires foo==1.0.0, then the installer would need to error if the computed version of foo no longer satisfies the specifier.

But if the lock file is already resolved then why do installers care about what version a source tree says it is? I can see if you want some extra stale check, but it isn’t required from a lock file perspective, just like requirements files don’t know when they are out of date today. It’s definitely a nice-to-have

@pf_moore

The specific case seems to me to be “lock everything but the project” and I don’t understand why that can’t be done with a lockfile that includes precisely that (everything but the project) plus a normal install of the project.

It could, but for flexibility to allow for uv to use this format with their workspace concept it lead to listing even the project itself so it was locked for itself and potentially as part of a larger, monorepo-like lock. It also allows for representing extras cleanly since the package itself is listed.

@pf_moore

The canonical example here is when the “root project” is specified as a local source tree, to be installed in editable mode. Personally, I consider installing local source trees, especially in editable mode, to be out of scope of what a lockfile should be for.

See above as to why this concept was pulled in. I believe Poetry also asked for editable install support (along with the lock file using source locations to be part of specifying what’s ); @radoering can correct me if I’m misremembering.

@pf_moore

But that’s basically where the debate currently lies - if we declare them out of scope, does that mean workflow tools like uv and Poetry won’t be able to use the standard lockfile format for what they refer to as “lockfiles”?

I’ll let them answer for themselves, but @charliermarsh already said uv won’t use the PEP for their canonical lock file format and just as an export target without this sort of support (he listed everything they support in PEP 751: now with graphs! - #55 by charliermarsh).

---

I kept going back and forth about trying to ask questions about bigger picture stuff, but having been thinking about this for well over 5 years, I’m waiting until we either all agree on what to do with this editable install topic and whether that’s going to necessitate rewriting the PEP again (and even going back to an older version is a rewrite as I’m sure it’s going to require plenty of updates). And if we do go around yet again, we will be having a very serious, time-limited discussion on scope of lock files that we agree to and we will not being changing later on; I don’t know if I can rewrite this PEP more than one more time w/o coming out of this whole experience extremely bitter (but I will see this through thanks to work supporting me getting this done).