OK, so can you define what behaviour you would consider essential in a “limited-functionality equivalent of a namespace package” in sufficient detail that I can write some tests to check that an implementation delivers that behaviour? (Because I’ve tried, and I can’t 
)
The good news is that an editable wheel can, I believe, contain a .pth file, so this is still 100% compatible with the proposed new hook.
A strong motivator for the src layout, it seems to me. An all-around win.
-Fred
Given package a and b under namespace foo, I’d expect foo.a and foo.b to be importable as long as all packages are installed as editable.
The pre-draft PEP now lives at build_wheel_for_editable by sbidoul · Pull Request #1 · sbidoul/peps · GitHub. The phrasing defines the broad expectations of a user for an editable install but otherwise leaves a lot of freedom to backends. .pth and editables are mentioned as possible implementation approaches.
OK. So basically it’s up to the backend, that seems fair. I’m sensing a general reluctance to be too prescriptive here, which is fine.
I’ll probably omit namespace package support from editables in that case, and let backends implement those however they choose. They can still use editables for non-namespace packages. That saves me having to make policy choices on behalf of backends.
This makes me feel the root problem is in Python’s import system, and editable namespace package support should and can only be implemented by introducing new mechanisms to the system. A “proper” way to “link” a package into sys.path, instead of adding an entire directory into sys.path, for example. So I’d feel it’s good for the editable PEP to explicitly rule out namespace packages, until a PEP adding that feature is proposed (by someone) and accepted in CPython.
Why rule it out explicitly, since the .pth approach works perfectly well with namespace packages? So a solution exist that is on par with what setuptools does, and ruling this out explicitly would create another roadblock for the adoption of PEP 517, IMO.
The pth approach works under certain conditions, but far from perfectly (even in terms of editables where technical perfectness is impossible).
Do we say “it’s an especially good idea to use src/ if you are doing a namespace package”? Does a namespace package or the .pth method fail then? (If you don’t need to exclude anything under src/?)
I’ve never been thrilled with namespace packages. They are fragile because someone can come along and put an __init__.py in their version of the namespace package, and it breaks you. The installer could check that all installed packages in a particular namespace are using the same strategy?
The Python import system is mostly designed statically - by which I mean the design doesn’t really take much account of things like sys.path or the contents of the filesystem changing “on the fly” while programs are running. This is the reason we typically don’t recommend executing pip from a running process, for example. It’s not exactly that the results are undefined, but they are pretty hard to reason about, and documented mostly by implication rather than explicitly.
Namespace packages are an unusual exception here, in that they explicitly do define how they behave if the filesystem content changes at runtime. And the implementation handles this by making the filesystem rescans part of the implementation of the namespace package, rather than something “higher level” that 3rd party code can hook into.
It is possible to implement them, just significantly harder. After all, we could simply take the existing implementation from importlib, modify it to handle the needs of editable installs, and ship that as support code with an editable wheel.
The question isn’t what can be implemented, it’s what trade-offs backends want to make. And more importantly, what trade-offs will be acceptable to end users. To that end, the setuptools .pth solution is simple and proven to be “mostly sufficient”. And it’s easy to implement in a wheel - it’s just a single file to install.
But from a specification point of view, the question is how willing we are to leave the behaviour of an editable wheel unspecified. For example, if a backend returned the exact same wheel from build_wheel_for_editable as it does for build_wheel, is that compliant? The backend could claim, “well, you can edit the files, they are all in site-packages”.
I don’t honestly think we need to protect ourselves against deliberate attempts to subvert the spec. But conversely, if the spec is too loose, we will get people interpreting it differently - we’ve had recent examples with other specs, and it’s a pain.
I think the spec should steer well clear of providing any sort of “advice” to backends, it’s too easy for people to end up claiming that it’s a requirement, not a suggestion.
I don’t have a good answer here, but I feel like it’s important to restrict the PEP to things we can be precise about.
@pf_moore The current release of editables is working well for me! However, I just realized that PyCharm & VSCode are unable to provide auto-completion. I think that may be worth mentioning somewhere.
I have no idea what they might need that isn’t already present to allow them to do so. If you can identify what that is, I can look at adding it.
Sometimes there is an “execute Python code to look at these trickier modules” list that imports the module instead of using static analysis
I was trying to politely say “if you explain what code I have to write, I’ll write it, otherwise feel free to offer a PR” 
I don’t intend to bother researching how to make editables work nicely with IDEs myself right now (I mostly consider it the IDE’s problem to work with properly-written Python code, not the other way around…)
I know from experience that PyCharm doesn’t resolve dynamically-loaded modules to get the attributes of the loaded module. There doesn’t seem to be any issues posted to the PyCharm bug-tracker about this.
This seems to me the kind of thing that will likely need a specific solution, like Jinja template file resolving. I imagine they wouldn’t want to release that solution until editable has been standardised.
Ah, if you mean that modules loaded via import hooks aren’t introspected correctly by IDEs, then yes, that’s not a problem in the editables code directly. I’m pretty sure the modules have a source file loader, so the source code is accessible - if that’s not the case then that would be something that can be improved, but I’d need some means to demonstrate the issue (ideally in a way that can be added as a test) before that would be possible.
By the way, these sorts of questions would be better raised on the editables issue tracker…
Correct, and none of the auto-completion providers support this sort of thing, so no one is going to be able to make it work without some configuration of your paths in your preferred editor for it to know where to look. This is the one thing that .pth files have going for them since they edit sys.path and that can be programmatically picked up with python -c "import sys; print(sys.path)".
Cool. So simply having a loader that gives access to the source code isn’t enough? One thought I’d had was to check that I was returning a SourceFileLoader where possible, in case that helped…
I view this as a gap for auto-completion providers to implement 
Rather than force only altering the sys path.
It’s not practical for static analysis tools to get all the runtime stuff right, so they have to draw a line somewhere. I view this not different from dataclasses and metaclass magic in ORMs etc.; once it is standardised (or at least stablised), static analysis tools can add specific logic to support it.