Discuss PEP 662: Editable installs via virtual wheels

[pf_moore (Paul Moore)]

OK. I fairly carefully re-read the proposal here, and I’m still none the wiser on how I’d write frontend code to implement this. As a result, I came to the conclusion that I’d want any proposal that gets approved to come with a working “proof of concept” implementation.

PEP 660 has this, in the form of the editables project, @sbidoul’s POC implementation for pip, and @ofek’s use in hatch.

Looking back into the history, I note that this wasn’t an original thought with me @pganssle originally called for a PoC here:

Third try on editable installs

Here is the current specification that I am suggesting we build a PoC for: Specification of editable installation Presumably with Bernat’s additional suggestion of adding a version number. It’s pretty dead-simple, and I don’t know that it’s worth going into much more detail until we know what’s feasible and what problems could come up, but I think it’s the minimal specification that meets the requirements of: Back-ends don’t need to know anything about how to install files. Front-ends handle…

Now admittedly, he was asking for a PoC of the “virtual wheel” proposal, and what ended up being developed was the PEP 660 PoC. So, well, sorry about that

But the point still stands. We should have proof of concept code, because fundamentally we’ve been going round in circles for a long time when we just talk in the abstract. So whether it’s this proposal, @pganssle’s or some yet-to-be-agreed compromise, I think we need some proof of concept code that:

1. Implements the mechanism for taking the data returned by the hook, and making it into a working installation the current Python environment. It only needs to implement one mechanism for doing this, but it can implement more if there’s trade-offs that are worth demonstrating, and it should be clear what works, what doesn’t, and what options there are to address those limitations (see how editables handles namespace packages for an example).
2. Demonstrates how a front end UI would look under the proposal - feel free to implement it for pip, but I’m fine if the example is a different front end (I appreciate that tackling a pip PR is a big ask).
3. Demonstrates how this would be implemented in setuptools. Or another backend, but as the point of this proposal is that it makes it easier for backends to implement, I’d hope that a setuptools implementation is achievable in a PoC.

My reasoning on points (2) and (3) is that PEP 660 puts a lot of the work on the backend, so it’s fair to build a PoC that does the “easy” frontend task in pip, but the “harder” backend work in any backend, just to demonstrate it’s possible. Conversely, virtual wheel proposals should expect to do the “easy” backend work in setuptools, and the “harder” frontend work in whatever frontend it’s easiest to use to demonstrate that it can be implemented.

If we don’t get a “virtual wheel” PoC, then we’ll need to make a choice:

1. Accept the proposal that’s delivered working code.
2. Defer editable installs yet again, and continue with the status quo.
3. Abandon editable installs altogether as something we can’t unify or standardise. Make the idea of backends providing a custom editable install mechanism (such as setup.py develop and flit install --symlink) the officially supported way of accessing the functionality.

If we did (3), we’d need to decide whether to drop --editable from pip when we remove the legacy setuptools-specific code. I fully expect a huge outcry if we did that, though But that’s an internal pip issue, so not really relevant here.