Seems that neither IDEs I’ve run into until now supports PEP-660 editables as shipped today by pip. The current logic works out at runtime however the IDE is unaware of the package and does not provide autocomplete. I consider this an important feature gap, so I’d like to start a discussion on how we can add/help IDEs supporting it. The current mechanism as used by most backends is defined by GitHub - pfmoore/editables and maintained by @pf_moore .
In short, the functionality is:
a pth file that imports a _<project_name> module
on the import of the _<project_name> module of the package install a import hook
this import hook knows (via registration) how to map modules to paths
For example given project pyproject_fmt, the pyproject_fmt.pth file is installed with import _pyproject_fmt content and the _pyproject_fmt.py file contains:
from editables.redirector import RedirectingFinder as F
F.install()
F.map_module('pyproject_fmt', '/Users/git/pyproject-fmt/src/pyproject_fmt/__init__.py')
The discussion is can this be supported by IDEs? If yes, how? If not, how we can adjust it to make it work.
I should point out that the library also supports just adding a simple .pth file which works exactly as the existing setuptools .pth file mechanism, so if that works better for IDEs, backends could choose to use that approach instead (there are trade-offs, which is why editables offers both options).
I’m happy to take PRs to improve the situation here, although I’d want a “common” solution (ideally something that could in the longer term be made into a standard that all import hooks could use, so it’s not just a special case). I’m not keen on having individual workarounds for each IDE.
For us (Spyder) it would probably be handled in PyLS/Python LSP Server, which we also are the maintainers of, or even further upstream in Jedi/Rope. @ccordoba12 would know much more about that and be able to ask the appropriate devs, since I haven’t been too involved in that aspect lately.
For VS Code, you probably want support in Pylance (already an open issue, so people can comment there if they want to help see it prioritized), otherwise Jedi would need to add support.
I’ll point out that this breaks a common idiom where you create an extension module called _foo and a Python level wrapper called foo.py which imports _foo.
The name _foo is an implementation detail that can easily be changed. I’d happily take a PR to do so. So far, it’s not been an issue (although that may simply be because setuptools doesn’t use editables and setuptools is the only backend in common use that can build C extensions).
IIRC (I’m still not caffeine loaded for the morning yet ) I reported this to @frostming for pdm and it was fixed over there. Anything you choose has some potential for name collisions, but practically speaking, something like __foo.pth might be fine (I think that’s the fix that pdm took).
Following the discussion at PEP 660 support · Issue #2114 · microsoft/pylance-release · GitHub seems it’s unlikely any static analysis tool would be able to support the current import hook based solution, purely because anything that would require some type of runtime evaluation is an instant no go for these tools.
When writing PEP-660 and PEP-662 we seem to look over the importance of static analysis, and that was the primary feedback when talking with the VS Code developers. In hindsight seems here the declarative nature of PEP-662 would have been something much easier to adapt and support for IDEs (as they could have parsed the editables.json and registered those as additional files).
Within the scope of PEP-660 though seems currently the mechanism must be pth files. Symlinks could be also an option one day if we add that support to the wheel standard. @pf_moore given this should the editables project change the default mechanism to be a pth file, instead of the import hook?
There’s not a default approach as such - there’s simply two methods available, my_project.map("foo", "src/foo") and my_project.add_to_path("src"). Backends can use whichever they prefer, and if they use add_to_path exclusively, it will be implemented using a .pth file without the import hook. So there’s nothing for editables to do here, it’s up to backends to implement the mechanism they prefer (which is the whole point of PEP 660, of course).
The documentation isn’t 100% clear about this, but honestly that’s because the documentation isn’t very good (yet). It needs improving, but the lack of questions on the tracker suggests to me that either the README is adequate, or not many people are using the library - so I’m not rushing to address this myself.
By the way, as this is specific to the editables library, rather than to PEP 660 in general, it would have been better raised on the editables tracker (where I’d have given the same answer, of course )
I’ve also commented on the linked pylance issue. To summarise, I think this is a discussion that needs to happen between the static checkers and the build backends. Whatever approaches are agreed on are either already supported by editables or can be added if necessary.
Perhaps standardizing on some metadata, such as a comment that static analysis tools could look for in a module, could work. For example, a comment like # editable-src: /path/to/src in a generated module, the same path that would have been in a pth file.
If pyright or another tool saw such a comment in the module they were inspecting, they could redirect to analyzing the code there instead. Build tools that implement editables dynamically for correctness could output that comment at the start of the module they generate, so that static tools could find the original code, just like a pth file would have done. Static tools likely don’t care about the “correctness” of non-python files in source anyway.
And for what it’s worth, the file is now called _editable_impl_<project_name>.py in editables 0.3.
In editables 0.3, the documentation explains the trade-offs more clearly, and presents the .pth based approach first, so the emphasis is (slightly) more on that option. The choice of which API to use is still ultimately up to the backend, though.
The conclusion from the pylance developers was that there’s no way they can make import hooks work. They’d want some static definition they can consume, something similar to what PEP-662 defined with its editables.json. Both PEP-660 and PEP-662 seemed to miss mentioning on ease of supporting the mechanism by static checkers.
I’m not sure how we can solve this now within the framework of PEP-660. Other than saying to people that if you care about autocomplete don’t use import hooks. The only alternative I can see is if we’d introduce that backends need to generate a _<module_name>_editables.json containing what file to map to what path and instruct static checkers/IDEs to index those too.
To be blunt here, I personally don’t really agree with the idea that we have to simply not use a basic Python mechanism just because static typing tools can’t cope with it. I thought type annotations were supposed to be optional
But as you say, it would be pretty straightforward for backends to generate some form of “type stubs” metadata file and add that to the wheel they generate. If someone wants to create a PEP specifying this (“Metadata to expose type information for packages which use import hooks” - it should probably be a typing PEP rather than a packaging one, much like py.typed was) then backends could add support fairly easily. I would very strongly recommend, though, that the mechanism is made general, rather than being focused on how editables currently does the mapping - we don’t want to go through all this again when someone comes up with a better way to use import hooks (for example, I think setuptools might be looking at their own approach, rather than using what editables does).
Either way, I’m happy to add support to editables if needed.
The problem here is that we’re not just talking about type annotations but also structural layout. The IDEs know nothing about the project and files with import hooks. And if the IDE is telling the users that their functions/imports are incorrect then that’s a big problem, especially for new users.
Understood. But Python is a highly dynamic language, and that’s an issue the IDEs have to deal with. Telling language users not to use parts of the language is not the solution. This isn’t just about editable installs, it’s about any use of import hooks to make code available in a way that IDEs can’t introspect - editable installs are merely one example of such a situation. Can IDEs provide tooltips for code imported from a zip file? What about code imported from a sqlite database (there’s no reason someone couldn’t write a sqliteimport that worked like zipimport)?
Defining a mechanism by which developers can state “this is what the runtime mechanism I used exposes” is a reasonable goal, on the other hand. But it’s up to the IDEs to describe what information they require and then the community can agree on how to provide that.
) I reported this to 
)