The Python import system is mostly designed statically - by which I mean the design doesn’t really take much account of things like
sys.path or the contents of the filesystem changing “on the fly” while programs are running. This is the reason we typically don’t recommend executing pip from a running process, for example. It’s not exactly that the results are undefined, but they are pretty hard to reason about, and documented mostly by implication rather than explicitly.
Namespace packages are an unusual exception here, in that they explicitly do define how they behave if the filesystem content changes at runtime. And the implementation handles this by making the filesystem rescans part of the implementation of the namespace package, rather than something “higher level” that 3rd party code can hook into.
It is possible to implement them, just significantly harder. After all, we could simply take the existing implementation from importlib, modify it to handle the needs of editable installs, and ship that as support code with an editable wheel.
The question isn’t what can be implemented, it’s what trade-offs backends want to make. And more importantly, what trade-offs will be acceptable to end users. To that end, the setuptools
.pth solution is simple and proven to be “mostly sufficient”. And it’s easy to implement in a wheel - it’s just a single file to install.
But from a specification point of view, the question is how willing we are to leave the behaviour of an editable wheel unspecified. For example, if a backend returned the exact same wheel from
build_wheel_for_editable as it does for
build_wheel, is that compliant? The backend could claim, “well, you can edit the files, they are all in site-packages”.
I don’t honestly think we need to protect ourselves against deliberate attempts to subvert the spec. But conversely, if the spec is too loose, we will get people interpreting it differently - we’ve had recent examples with other specs, and it’s a pain.
I think the spec should steer well clear of providing any sort of “advice” to backends, it’s too easy for people to end up claiming that it’s a requirement, not a suggestion.
I don’t have a good answer here, but I feel like it’s important to restrict the PEP to things we can be precise about.