What do you mean by “it knows how it’s been built” ? The PEP is not prescriptive wrt the content of the wheel.
This doesn’t make sense to me. Does PEP 660 allow you to distribute its pseudo-wheels? Your front-end will always need to know how to execute an editable wheel build (it’s a different hook, after all) There’s no “required frontend version” other than “a version that supports PEP 660”, which will always be the case for any editable installs not on a setuptools backend.
This is another misunderstanding of the relationship between the install mode and the project. There’s absolutely nothing project-specific about the minimum front-end version needed to do editable installs (except for whether or not the project uses a setuptools). It’s the user who wants an editable vs. normal install, and they can achieve their tasks only with front-end tools that support that operation, which means a recent version of pip or whatever their front-end is.
No, this is not a requirement. pip can choose to have one or two or three or any number of modes. The point is to make the standard have the right boundaries of responsibilities so that pip has the option to choose different installation modes (because they are more efficient or because they have different trade-offs, etc), or pip can choose one mode and stick with it and an alternate front-end can come in to use another mode. That’s the point of these standards, is to make things not so dependent on the tools itself.
1 Like
I don’t think I’ll have time for this thread at this point or at any point in the foreseeable future, so hopefully people take my objections into consideration.
To quote Thomas Gilovich’s How We Know What Isn’t So:
For desired conclusions, we ask ourselves, “Can I believe this?”, but for unpalatable conclusions, we ask, “Must I believe this?”
I realize that y’all have some momentum behind PEP 660 now and are afraid that it might be derailed by moving to a “virtual wheel” paradigm, but I would appreciate it if y’all would take a moment to “steel man” my suggestion. Come up with the best possible version of “virtual wheels”, and with each objection try and achieve the same thing in the virtual wheel framework. Try to poke holes in PEP 660 and find the advantages of virtual wheels.
I honestly went into this thinking it was much more a matter of trade-offs than my final opinion, because I realized that with minor modifications the idea of “virtual wheels” can achieve everything PEP 660 can do, but with much more freedom for innovation on front-ends.
If anyone wants more information about the details of how a virtual wheel design could work, feel free to shoot me an e-mail.
I’ll write up the counter/competing proposal in the next week or two 
so if anyone can be patient a bit 
3 Likes
I was actually about to create a post here that was set to be a wiki to make it easier for people to coordinate/write together (I’m not an editable install user, so I was doing it just to help focus energy towards an end result). But using a wiki post might alleviate the load if you want help in writing the PEP.
As a long time vocal open source person I think I know exactly how Paul G feels, but I think the argument for PEP 660 is clear. Hopefully no one else is emotionally invested in preventing the mechanism. Thanks for engaging on this issue, so I can finally pip install -e . in my little experimental build system without mocking setup.py.
I’ll add a couple of final points here, just as summary/feedback.
- An editable install by definition means that the user has the ability to change the project source. So it’s perfectly plausible to ask the user to add configuration like
setuptools.editable_style = "pth". topyproject.tomlif they want to select a particular type of editable install. So while I agree that it’s possible users may want control over the style of editable mechanism, I don’t think PEP 660 precludes that. - It should be possible to write a PEP 660 backend that “wraps” an existing backend, adding the
build_wheel_for_editablehook and communicating with the underlying backend in any way it chooses. In particular, it could communicate with the underlying backend using the “virtual wheel” approach. So that implies that anything the “virtual wheel” approach offers is also possible via PEP 660, with the addition of a little bit of integration boilerplate.
I’m wary of characterising either approach as a “strict superset” of the other, but as one of the arguments for “virtual wheels” is that it can do pretty much everything that PEP 660 can, I thought it would be worth making these points for contrast.
PS I’ll also note that I’m only an occasional user of editable installs myself, so I have no vested interest in either solution over the other. My interest comes mainly from wanting to be able to give an answer to pip users who want to know why pip install -e is limited to legacy setuptools projects. So yes, I do have an interest in getting something sorted sooner rather than later - but conversely I’d like to see the solution done right, so we don’t have to keep revisiting this subject as users find edge cases we didn’t consider.
1 Like
But we willing to end it to break all the world?
In my view the most important benefit of PEP 660 compared to virtual wheels is that it allows for easier experimentation and innovation.
It seems quite clear that the community has a wide range of theoretical and practical expectations about editable installs. The reality is that the only one there is wide experience with is path insertion via .pth (i.e. what setup.py develop does). editables is a very promising approach, yet it has been mentioned it does not cover all use cases.
With PEP 660, several approaches are possible without any change to the specification nor to frontends. For example these only require backend variations:
- insert a directory in python path (à la .pth)
- expose specific files (à la
editables) - allow for a different source layout in the project and installed code
All these would require additional wording in a virtual wheel specification, plus specific frontend support, in addition to backend support.
So, in my opinion the additional specification and implementation complexity of virtual wheels is real, and would certainly cause more delay.
While I do understand and appreciate the argument that the choice of “editable style” should be left to the frontend/developer, I think it is not worth the additional complexity, especially knowing that this choice can still be passed to a PEP 660 backend via config settings [1] or another environmental mechanism.
I plan to add wording like this in the rejected ideas section of PEP 660 that refers to virtual wheels. I hope it captures the essence of the discussion.
If there are other ideas that were mentioned elsewhere that people feel should be mentioned in PEP 660, just let me know.
[1] config settings have not been implemented because they lack concrete use cases and no-one has really asked for them so far
4 Likes
Hey folks, I’ve wrote up the competing PEP under TBD: Editable installs by gaborbernat · Pull Request #1977 · python/peps · GitHub (if you’re a core dev and like what it’s proposing please ping me if you wish to sponsor it). Feedback welcome, hope is moderately readable though.
Discussion on proposal moved to Discuss TBD: Editable installs by gaborbernat
I think PEP 609 – Python Packaging Authority (PyPA) Governance | peps.python.org means you don’t need an explicit sponsor since you are already a member of the PyPA. But having a core dev does make things easier for marking someone in CODEOWNERS as being in charge of merging PRs for your PEP.
2 Likes
One option that is not supported is a set of symlinks, the way flit currently implements editable installs. This is because the wheel format doesn’t support symlinks. There has been talk about adding symlink support to wheel, but at this moment, that wouldn’t be an option.
I don’t have an opinion as to whether this is a major issue. Maybe @takluyver could confirm whether flit is OK with changing its editable install approach?
I’ll see to add a mention about symlinks in the text.
Note that flit does support .pth too, and the flit POC linked in the initial post is based on that. It works just fine, especially with the src layout that flit now supports too.
As a side note I do have a requirement to use symlinks in a custom backend and I could implement it with PEP 660 by generating the symlinks on interpreter startup (via an executable .pth that is put in the wheel by the backend). It is a not particularly elegant, but I just wanted to note that it is possible today.
1 Like
I mean, it’s a general purpose computer, so you could get it to do whatever you want. My point is that changing the project configuration is not usually the right way to specify things that change between use cases.
Consider the case of someone who is working on a project in an IDE and also running console scripts in a terminal. Their IDE may want to do an editable install into one virtual environment using a mechanism that allows the IDE to interactively rebuild certain things, or that makes different trade-offs (e.g. the IDE knows when you add files because it’s doing the file adding, so it may want the stricter approach), while for the purposes of the use of the terminal, you’d want to use pip or some other tool that does things a different way. In the “backend config” model, you’d have to toggle the configuration for the whole project depending on which tool you are using, and you are using them simultaneously.
Consider also that most backends will not implement multiple different editable install modes, so it won’t be a simple matter of just changing .editable_style somewhere, it will often be the case that if you want a different editable style you need to switch your build backend, which is a lot of lift for a configuration change.
So yes, usually the person doing the configuration also can change backend configuration, but from a UX perspective it’s best to assume that all changes to backend configuration are project-wide and long term while front-end configuration is per-invocation.
I’m sorry but there really is an asymmetry here because of the information that the standard exposes. In PEP 660, the only information that the front-end is transmitting to the back-end is “This should be an editable install”. The back-end then changes what goes in the wheel in an arbitrary and unspecified way, opaque to the front-end. Theoretically a front-end could be aware of all the mechanisms that every backend uses, introspect the wheel and attempt to reverse engineer how to expose the right files, but I think we’d all be comfortable saying that this approach is not going to be supported.
In the “virtual wheel” proposal (note: the wheel doesn’t have to be virtual, it could be an actual wheel file with a file ↔ file mapping in text or something), the front-end still communicates the same information to the back-end, because it’s still letting the back-end know that it needs to build an editable wheel. The difference is that the back-end now communicates more information to the front-end, namely it’s telling the front-end enough information to expose the right files in the right places, which is something that only the back-end knows, and is basically the raison d’être of a back-end. I don’t think it would be prudent for the back-end to build a completely different thing when build_editable is called, but if there’s any problem that only the backend can solve because of something specific to the project or the nature of the backend, the backend has all the information it needs to solve that problem, and the only difference is that any files it was going to generate and put in the “editable wheel” file needs to be a file that exists locally on disk.
That is why I say that my proposal is a strict superset of the PEP 660 proposal, because the back-end has all the same information it gets in PEP 660, but in my proposal the front-end gets more information and in most cases (basically every case that I can think of), it gets enough information to create an editable install that is as good or better than what we have now.
I’m not clear why you think these are not possible in a virtual wheel specification. None of these require special handling in a “virtual wheel” (though it may be misleading to call it a “virtual wheel” because the fact that there’s no file on disk is not the salient part of the proposal — the important part is that what’s transmitted from the backend to the frontend is a mapping from location on disk to install-time layout).
When building a wheel, a backend needs to take resources on disk and translate them into an install-time layout. Right now it does that by copying them, but if instead of copying them it were to just record the mapping and expose that to the front-end, the front-end can, instead of copying, make those files directly available to the interpreter. The front-end then has the option to:
- Use
.pthfiles to insert all directories containing the relevant files into the Python path (à la.pth). - Generate a wrapper file that exposes only the files listed in the mapping (à la
editables) - Use symlinks to directly implement the mapping
- Something else of its choosing.
And importantly, this choice will be up to the person running the install, and not a per-project setting. Since we’re talking about a mapping (e.g. {'/path/to/source/src/myproject/__init__.py': 'myproject/__init__.py'}), the backend still defines how the input and output structures map onto one another, so it would not be a problem to have a different installed layout from the source layout.
I did not say they are not possible. I’m not going to rehash my argument, please re-read my post.
I have re-read it and I still don’t understand why you think it would be any more complex. What about my description of how to achieve those things makes you think any additional complexity is needed? No special support is required from backends for any of those things, they just implement the spec as normal. The complexity that was on the backend (implementing the editable installs) moves to the front-end, which is more qualified to implement it anyway (since front-ends can specialize to specific platforms, etc, and in general they know more about the system they are installing into).
I genuinely want to know why you think these things are more difficult to achieve or require more complexity in my proposal, because I suspect it means I haven’t adequately explained what I’m proposing or I’m missing something fundamental.
Another note here. PEP 660 says
Build backends may use different techniques to achive the goals of an editable install. This section provides examples and is not normative.
in what to put in the wheel. In theory, a backend could put any old junk in the wheel and claim to be compliant. While I don’t want to open up the question of “do we define what it means to be an editable install?” again, can we at least say something like
Build backends must populate the generated wheel with files that when installed will result in an editable install. Backends may use different techniques…
That way we’re being explicit that it is the backend’s responsibility to achieve the effect of an editable install.
This PEP does not cover at all how would this work for C-extensions. Namely, when within the wheel you’d have pyd/so files, data, scripts or include. The way editables package is today only handles pure python files. What should a backend do when it would generate data/scripts/include files into the wheel?
I’m extremely perplexed by this question, which makes me think I’m missing something fundamental.
If editables exposes /path/to/foo as package foo, and you have built .(pyd|so) files properly within said path, what is the issue?