@sbidoul perhaps you would like to collaborate on this document in git. peps/pep-editables.rst at master · dholth/peps · GitHub
* Yes, I’m aware that I’ve broken several PEP formatting rules and only have one space after sentences.
The PEP has every right to tell backends what they are allowed to do and what they are not - that’s the whole point of a PEP after all, to define rules that compliant tools must follow. Also, I don’t think the PEP needs to go into the history too much, but what you have in your current draft looks fine.
Having said that, I think we have a way of handling single-file modules (install a package containing an __init__.py that redirects to the module) so the issue may no longer be relevant.
FYI you probably also want to update __name__ and __file__ for the __init__.py during this process to behave exactly as a single file module would. Though this rewrite arguably could be a backend flag so the user can opt out of it.
I very much like the idea of using wheels for this. Users may wish to have several packages in editable mode simultaneously, one potentially being a (transitive) dependency of another. I imagine this would become possible this way. Building a wheel with pip could then look like pip wheel --editable /path/to/src? Thinking about how this would be used by Nixpkgs users, it could be a simple override for them, passing editable = true; and optionally a path to the source.
I think the PEP should not be too specific about how to create the editable wheel. Just what the end result should (broadly) look like for the user after installation of that wheel. In that sense, the PEP would recommend editables as the best known example of what to do, at the time of writing the PEP, but should not preclude other approaches.
2 Likes
I don’t think pip will expose a mechanism to expose the editable wheel to the end user. These editable wheels will remain a communication mechanism between pip and the back-end, only for the purpose of installing in editable mode.
Of course that should not preclude other front ends to do it, but the PEP should emphasizes that these wheels are meant to be transient and are not meant to be published. I’m considering to add wording in the PEP to recommend back-end to add a local version identifier to the wheel (such as +editable) so at least PyPI would reject them.
1 Like
Is there discussion on this?
Also, you’ll probably need an empty __main__.py
Yes, I agree we should not document in the PEP the pip ui although it may show an example of how it could look.
I agree also with you that it should be clear these are editable wheels, however
I don’t think pip will expose a mechanism to expose the editable wheel to the end user. These editable wheels will remain a communication mechanism between pip and the back-end, only for the purpose of installing in editable mode.
I do hope we could get the wheel out, so it is still possible to have a two step process of building and then installing. Anyway, this is for on the pip tracker by then.
Agreed. The renewed interest in this topic has meant that I’m looking at editables again, and I’m not too happy about the specifics of the existing mechanism. It works, but as I mentioned it has (fixable) problems with single-file modules, and it also has the problem that it adds a path hook to sys.path_hooks for every editable package that gets installed.
I do intend the editables package to ultimately be the reference implementation for how to do editable installs (and once it reaches a suitable level of functionality I intend to submit it to be a PyPA project), but at the moment, it is still a proof of concept implementation, and like you, I don’t think we should be stopping people from experimenting if they wish.
Having said that, I do think the PEP should be clear over the contract it expects backends to fulfil. I don’t think these need to be controversial, it’s mostly just a matter of being explicit over what “an editable version of a normal install” means. So:
- The things that get installed are actually “editable” in the sense that changes made to the source location will be visible when the package is next imported. (No, forwarding
build_wheel_for_editabletobuild_wheelis not allowed
) build_wheelandbuild_wheel_for_editableshould put the same things onsys.path. I’d prefer to leave this a little flexible for now, until we know that we can handle single-file packages properly (we may have some problems to iron out with C extensions, too), but setuptools has set the precedent that people don’t need perfection here.
I’d also at this point like to leave the door open for implementations to either install “extra” modules, or add a runtime dependency. Specifically, editables currently installs the same path hook code multiple times - being able to install it once, and just reference it, would be a useful saving IMO. The runtime dependency avoids namespacing problems, but people might not like it.
No, it was pointed out on the editables issue tracker as a limitation of that implementation, and I’m thinking about implementation. I don’t think it needs wider discussion, does it? Unless you mean whether the PEP should mandate that backends support this case, in which case this is the discussion 
IMO, the existing setuptools behaviour is the baseline here, but setuptools support for single-file modules is broken (exposes too much) so we may have some flexibility.
I don’t understand. Could you give an example of what you mean? You can demonstrate the behaviour using setuptools current editables, as that should be the baseline for what’s “good enough” here.
You can always call the hook yourself, so it doesn’t need a pip command (any more than we need a pip sdist command). If you feel a CLI invocation is important, it might fit better in the build project.
1 Like
Sounds like everything here is not controversial, which is great.
Wouldn’t all setuptools single- or multi-file setup.py develop support be broken in exactly the same way, and in exactly the same way as PYTHONPATH or .pth techniques? It adds the source root to the import path, exposing everything in that directory. Unless you use a src/ folder in which case the path method is faithful to the installation.
Clearly Python import hooks are powerful enough to do whatever we want.
I like the idea of not requiring the editable wheel metadata to be the same as the production wheel. You could depend on a helper module or flag the wheel as editable. It would be a nuisance if multiple editable installs required different versions of the helper module though.
It would indeed. That’s a good point.
My main concern here is that at the moment, every directory (subpackage) exposed by build_editable adds its own sys.path_hooks entry. In practice, though, I’d imagine most people would only have one or two editable installs in any given virtual environment, and each one would only include a single top-level package (installing pytest just adds pytest to the things you can import - setuptools adds two, setuptools and pkg_resources and is widely acknowledged as an unusual case). So this may well be a bad case of premature optimisation.
As long as the PEP doesn’t flat-out prohibit experimenting with solutions like these, I’m fine with not worrying for now.
Hooks aren’t the problem here (meta-hooks let you do essentially anything). The big issue is initialising the machinery. As things stand, there’s no way for an installed package to run any sort of “startup” code (other than abusing the “executable code” mechanism of .pth files), so we have to write modified files in the installation, that boot up the redirection and then replace themselves. That’s fragile, and frankly just ugly.
Maybe PEP 648 will be an option in future, but for now I think we either need to use .pth files or stick with self-replacing .py files.
I think the self-replacing .py is pretty cool. It doesn’t expand the search path until you import the package. What’s the rule for .pth files in subfolders of site-packages?
Simply the normal caveat when executing packages: you can’t execute a package without a __main__.py. The following is an example:
# /data/foo.py
import sys
print(sys.argv)
# /data/bar/__init__.py
import os
import importlib.util
spec = importlib.util.spec_from_file_location("foo", "/data/foo.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
del os, importlib, spec
locals().update(foo.__dict__)
del foo
python3 -m foo prints: ['/data/foo.py']
python3 -m bar prints /usr/local/bin/python3: No module named bar.__main__; 'bar' is a package and cannot be directly executed, and has non-zero exit-code.
Adding an empty /data/bar/__main__.py allows it to work.
PS: the output of python3 -m bar is ['-m'], so my implementation isn’t so hot
I now have a proof of concept of the “meta-hook” approach in the editables project under the redirector branch. See this issue for some notes on it.
It uses a .pth file to install the hook, so there’s no “self-replacing” going on. At the moment it won’t handle being used by multiple packages, and the way data gets added to the redirector is a massive bodge, but it should be enough for people to experiment with the approach. On the plus side, it seems to handle single-file modules and namespace packages without problems.
As for the PEP it seems very certain that the basic mechanism of doing an ephemeral wheel is powerful enough and could be finished. Even if the wheel contained an .egg-link exactly like the current mechanism; the PEP is primarily about supporting non-setuptools and secondarily about encouraging a higher fidelity editable experience.
We could remove the metadata directory from the hook entirely. We should make clear that the metadata should be the same as the normal package, and we should allow it to have extra dependencies.
We could add some design notes as for why we chose a wheel (the installer knows how to install it which is not the case for other formats e.g. a json manifest).
1 Like
Working some more on this over the last few days, I’m pretty sure there are some limitations that it will be difficult, if not impossible, to overcome. They involve functionality that the current setupools editable mode gets wrong, too, so I don’t think it’s a showstopper, but I do think we should be careful in the PEP to make it clear that backends do not have to guarantee that editable installs act “just like” normal installs. I don’t know precisely what the requirements should be, but the following two points have already come to my attention.
- Namespaces packages cannot be a mix of editable portions and non-editable portions
- Editable installs of single file modules may expose more than just that one file
My personal interest, as a potential downstream library for backends writing editable-mode wheels, is not to be put in a position where “your editable mode is broken because it doesn’t do XXX” is passed onto me with an implication that it’s up to my library to faithfully replicate a normal install.
What I’d like to see in the PEP is a statement of the minimal level of functionality required to be sufficient - otherwise we’ll be left in the unpleasant situation where “doing as much as the old setuptools mechanism that’s no longer available” could be what the standard in effect mandates.
At this point, I’m only flagging this up for consideration. I’ll wait to see what the PEP actually says before getting into details here.
Definitely. As you say, worst case a wheel that installs a .pth/.egg-link file will replicate the current behaviour.
Define “finished”. That’s the problem here. I now have (effectively) two implementations, but I’ve no idea if they are sufficiently complete. One only handles packages, one handles pretty much everything but won’t merge editable and non-editable portions of namespace packages. The second needs a meta-hook. Is the first sufficiently “finished”? I’ve had bug reports that it “needs” to handle namespaces (perfectly polite ones, I don’t want to suggest anyone is being pushy here), and I have no idea if I’m OK to say “nope, out of scope”.
We can finish the most difficult part to change, which is mostly the function signature; get that into pip and backends.
The part of the PEP that tells you what to put into the wheel (not the desired effect) will be a recommendation. What actually goes into an editable wheel will change over time as people learn different ways to do it.
I’ve been looking some more at namespace packages in the context of editables. The big issue is that PEP 420 explicitly requires that namespace packages recalculate their path based on sys.path every time they are accessed. This means that you cannot just “add” a development directory ~/work/project/foo as a namespace package - sure, the sys.modules entry is present, but unless the project directory ~/work/project is on sys.path at the time you import from the namespace, it will not be searched.
So the practical implication of this is that namespace packages cannot be supported properly. Note that this is a property of the import machinery, not specific to editable mode (via wheels or otherwise).
Possible workaround include:
- Just add
~/work/projectvia a.pthfile. That’s what setuptools does now, and it seems to be acceptable enough. The problem is, it exposes the whole directory, which means it’s not a precise equivalent of a normal install. - Write your own implementation of namespace packages, or steal/monkeypatch the importlib implementation. This looks like it would be tricky to get right, and it would then deliberately not be following PEP 420. I’ve no idea if this is sufficient, as I don’t use implicit namespace packages myself.
The current implementation of the editables package doesn’t support namespace packages at all, and I intend, at least for now, to keep it that way and make it an explicit limitation.
My question for the PEP is, what do we want to say build_wheel_for_editable must do? Is it OK for it to just omit namespace packages when building the wheel? Should it be required to fail, and (in effect) say “project cannot be installed as editable”? Is it allowable to implement some sort of limited-functionality equivalent of a namespace package? I’m reluctant to choose that final option, because we’ll then be sucked into writing our own equivalent of PEP 420 without the “dynamic path” feature.
I don’t see why we wouldn’t allow that, and thereby force the perpetual use of setuptools / setup.py.
I plan to revert to the pth way for hatch (based on an option in pyproject.toml).