I’m maintaining a tool that performs static analysis on packages specified by users. To find the specified packages, it uses some custom logic. Part of this logic is to treat .pth files and “editable modules” in a special way. Editable modules are those that usually are private (begin with an underscore) and have “editable” in their name. These names are chosen by the various tools responsible for installing packages in editable mode or providing the hooks for editable installations, and I was wondering if there is already a suggestion to standardize them, or if we should make one.
Some examples of names for such modules:
__editable__PKG.py (setuptools)
__editables_PKG.py or _editable_impl_PKG.py (installer)
_PKG_editable.py (scikit-build-core)
Sorry if this has been asked elsewhere, I couldn’t find it. Let me know if I should repost this as a comment somewhere
No, there’s no such standard. There’s not even a necessity for the editable install to provide a “helper module”, although most implementations that don’t just create a .pth file seem to do that. I know that the highly dynamic nature of using path hooks causes problems for static analysis tools like type checkers. This has been raised before - one such discussion was here, but as far as I’m aware no-one has tried to do anything about standardising anything.
The intention of PEP 660 was to allow backends to choose their own methods for implementing editable installs, precisely because there was little consensus on what the best way of doing so was, or even what guarantees an editable install should provide. I don’t know to what extent there’s been any consensus among backends on the best mechanism to use - there’s certainly not been much shared public discussion on the matter.
As far as editables is concerned, I don’t have a particular problem if someone proposes a standard name. For what it’s worth, editables currently generates a PROJECT.pth file and in some cases an _editable_impl_PROJECT.py file as well (note it’s the project name rather than the package name - I don’t know if your use of PKG was a mistake, or if the quoted backends really do somehow provide per-package redirectors). Changing these isn’t hard. But I won’t change them arbitrarily - it would need to be on the basis of an agreed standard that all backends follow.
What I will say, though, is that I’m not particularly in favour of mandating any particular behaviour of the code in the .py file. If we want to dictate how editable installs are implemented, we’re squarely in PEP territory IMO.
Also, why the heck is installer generating an editable implementation file? It’s a frontend, and has no business doing so. Or are you misunderstanding where those files came from?
Hi @pawamoy, we can address your proposal directly, but I think what you are actually looking for is a bit different, so there’s probably no need to go into editable installs details, which might unnecessarily complicate things a bit.
AFAICT what you want is a way to map importable names to packages, so we probably should be looking at maybe standardizing setuptools’ top_level.txt, and perhaps extend it too.
In case you weren’t aware, importlib.metadata has a packages_distributions method that does just that, though it isn’t super accurate[1]. This does require you to run the interpreter with the packages installed though, which I suspect isn’t what you want.
It looks for a top_level.txt file, so it should work well with setuptools, but it tries to calculate the importable names itself otherwise, which is when it might fail to find certain names. ↩︎
One option might be to traverse the installed packages’ metadata - look for a direct_url.json where dir_info.editable is true (see Recording the Direct URL Origin of installed distributions — Python Packaging User Guide) and extract the installed filenames from the package RECORD. Not everything that’s editable-installed provides “editable” functionality, but this might just be good enough for your purposes.
@pf_moore about my use of PKG instead of PROJECT, you’re right, I simply chose the wrong word for the placeholder. About installer I’m probably wrong as well, I wrote that from the top of my head but I think it’s indeed editables I was referring to (sorry for not doing my research before posting, even though you got my point!).
@FFY00 and @layday indeed I cannot look for package metadata: my tool must be able to find packages given a list of paths (defaulting to sys.path): the packages in question might therefore not be installed and importable into the current environment.
As for why I’m asking all this: I’d simply like to be able to support all tools making use of such “editable modules”, without having to hardcode some regular expressions into my code. But I think I can manage doing that by better handling pth files and following them accordingly: if they import _something_editable.py (or similar names), I just search for such a module and try to find the right info within it. Actually, having different names from different tools helps me decide how to retrieve this info, so it’s not that bad.
They don’t need to be installed or be importable in the current environment; they just need to be installed in the sense that they have a dist-info folder which is on a given [sys.]path.
Assuming that with “meson” you actually mean “meson-python”, the name of the support module used to load the editable installation is _{distribution_name}_editable_loader.py where distribution_name is packaging.utils.canonicalize_name(name).replace('-', '_') and name is the package name. Your regexp has an extra _py in it.
meson-python also installs a .pth file to load the editable module. That is named {canonical_name}-editable.pth where canonical_name is ``packaging.utils.canonicalize_name(name)`.
I do mean meson-python, thanks for making it explicit. And the regex does match the editable module that the .pth file points to. A issue on GitHub reported such a name _{dist}_py_editable_loader.py, but building locally I do get _{dist}_editable_loader.py Fixing my comment above right now, thanks.
Fixing my comment above right now, thanks.