Supporting sdists and source trees in PEP 665

[brettcannon]

@pf_moore

My assumptions are (roughly)

… all correct from what’s in my head.

@pf_moore

This is another round of the “how intelligent must the installer be?” question. And to an extent, it’s 100% fair to say “it’s up to the installer”. But I think the PEP should state clearly what the minimum level of behaviour is that users can rely on. Even if that’s just the bare minumum “installers must do the right thing when installing a lockfile into an empty environment, but if there’s already packages other than the installer in the environment, behaviour is at the option of the installer”.

I think that’s fair. I will update the PEP.

@sbidoul

@brettcannon

I think that would have be a choice of ours to say that’s an assumption being made as I don’t think anything currently restricts/requires this

Is it not a reasonable assumption to make ?

Maybe? I really don’t know. Some people do crazy things in their setup.py files.

@sbidoul

I’m not quite sure I understand how that would work. For instance where would those separate lock files be stored ?

That’s between you, the tool generating that lock file, and the tool doing the building. I got enough push-back from suggesting where lock files should go in terms of directories that I am not about to make that mistake again .

@sbidoul

Would project authors have to create a separate lock file for each of their dependencies that is not available as a wheel ?

Probably. As of right now the PEP has no concept of scoping per-project, and that would be required to have separate build dependencies per sdist/source tree that don’t conflict with other projects or runtime dependencies. Maybe this is that one case where it makes sense to let a lock file point to another lock file to specify the build dependencies for an sdist (although I know @stewartmiles expressed concern about files getting out of sync with each other, but maybe if there were file paths linking them that isn’t as much of a concern?).

@sbidoul

If the list of pinned build dependencies is scoped to each source distribution in the lock file, and the installer builds sdists in an isolated environment, there is no conflict.

How would that look in the file? If people want this sort of thing supported in v1 then I think we need a proposal of what it’s supposed to look like from someone pushing for this. I have an idea, but it’s ugly and so I would rather see what others propose.

Otherwise I would rather get v1 landed and make sdist/source tree support a v1.1 thing for someone else to propose/push after seeing what solutions the community comes up with.

@sbidoul

IMO, it could be fair to say that supporting sdist building is optional for installers.

It would have to be optional IMO, else the simple installer and security story gets diluted too much. I will fully admit this PEP helps push the “wheels are good” story as much as possible and letting in sdists waters that down.

I would also say anything that goes in about locking build requirements would also have to be optional for lockers.

@mdrissi

Part of the reason this issue is specific to monorepos with multiple packages is that for repository with exactly one package you can tell pip compile setup.py/pyproject.toml and it will produce a lock file of the dependencies of that package without including package itself.

I don’t think that’s inherent to monorepos if you treat the individual packages in the repo as just that; separate, individual packages (and I used to work at Google, so I have lived the monorepo life and realize it can have its own issues when you view your entire code base as a series of snapshots instead of as individual units of stuff you pull together as needed at different points in the repo’s history).

From the way I’m reading it, the issue you’re having is you’re trying to treat the monorepo as a single thing to lock against, but still developing sections of it as independent units. That just doesn’t fit with the worldview this PEP is presenting. You may need to have your own tooling to make this all work by regenerating your lock files as dependencies change at whatever project granularity you have in your monorepo.

To be honest, this PEP might simply not be a solution for you (if I’m understanding what you are specifically asking for appropriately).