This may be a difference of perspective. When you are implementing the backend you have to put the wheel somewhere, and the filename should be different to avoid overwriting the regular wheel. In enscons I’m currently adding a tag to the rest of the wheel tags. I think I do like the idea of replacing the Python tag, or possibly the platform tag, with the word ‘editable’ and having the installer accept that in this mode. It solves the unique filename problem, prevents accidental distribution, and prevents accidental installation.
In normal use pip asks the backend to generate the editable wheel, and then pip installs that wheel. pip of course does not generate the wheel by itself.
I think we’re trying to move away from formulating things from the POV of pip (or at least that’s my impression - hence why we don’t have a pip build command). So it’s clearer to express this in terms of build frontend, build backend, installer and a workflows orchestrator. pip might do all these, or some parts of it and delegate the rest to other tools. Doing this distinction helps users understands better what changes are needed in what parts, and who is responsible of doing those changes to support editable installs.
I’m baffled by this comment. There’s nothing stopping someone doing that right now so I don’t see there’s any new issue here.
Personally, I think that attempting to prevent this (whatever “this” actually is…) is likely to cause more harm than good. Python has always worked on the principle that we’re all consenting adults, and I don’t want to see that change.
I think it might be worth adding a note to the PEP, addressing the question of “can this feature be misused”, noting that anything someone could do with this feature, they could just as easily do with a “normal” wheel already, so there’s no new potential for abuse (or security concerns) around the PEP.
This was very poor wording from my part. I was not advocating that we should somehow prevent something like this to be done, but rather that we should prevent it from leaking into the rest of the ecosystem.
At the end of the day Users (with a capital U) will not listen to me, or you, they will search something on Google and look at the first Stackoverflow response. So my point is, it would be very helpful to have the “official” tooling say “Hey you probably don’t want to do that, are you sure?”, instead of validating these actions. If I was a user and the top voted answer on Stackoverflow said that I could build editable wheels, stack them in a directory and pip install them when I want, I would probably take that for granted because they are just normal wheels, unbeknownst to me that unlike other wheels these ones not only depend on the system, but also on the machine state, and I shouldn’t be saving and re-using them.
Anyway, I think I have made my point clear. I don’t think a user should be able to build an editable wheel and manually install it with an end-user wheel installer, they should at least receive a warning.
IMO editable wheels should be marked and installers meant for end-users should refuse to install them, or at the very least give a warning. Probably the best option right now to mark the wheels would be by adding a tag to the wheel spec.
I’m not sure we have this distinction. Many tooling uses pip under the hood, so the only real wheel installer today (wheels) is both meant for end-user and machines.
I’m still baffled, as you can do exactly this right now. There’s absolutely nothing special about an “editable wheel”. The only new thing is that until now, “editable installs” have been a custom, setuptools-only mechanism. We’ve realised that they don’t need to be, and that existing install mechanisms can handle editable installs, the wheel just has to have different content.
Want me to write that stack overflow answer? I can do that right now, and never need any of the machinery in this PEP to do so. I could put it under a question that said “how do I do an editable install in flit?” if you wanted me to be particularly obnoxious about it.
I won’t because it is a stupid thing to do. But it’s not a consequence of this PEP, nor is it something we need this PEP to “protect” against. If anything, the PEP will stop people looking for “creative” ways of doing editable installs because pip install --editable now works for all backends, not just for setuptools.
You can, but there’s no standardized mechanism for it. Implementing this like that in my project won’t be an issue unless lots of users start to use it. Introducing a standard mechanism will bring a lot of users, which is when this can become an issue.
I never said this is a consequence of the PEP, nor that the PEP needs to solve it. I think this would be an improvement to the PEP.
I don’t think we’re that far apart. @FFY00 understands that pip will create and then immediately delete the editable wheel in 99.9% of cases and users will be unaware that it ever existed. Occasionally the editable wheel will escape. It could have some kind of special tagging or already proposed special version number to make an escaped editable wheel more distinct.
We have since the beginning of wheel avoided transmitting random Linux wheels to random other machines, and then manylinux was invented.
I don’t think the two positions are really much different here either. @FFY00 thinks the user should be discouraged more to manually fishing out the wheel and install it elsewhere, and @pf_moore thinks the standard doesn’t need to get into such things. IMO the views aren’t even contradicting.
The way editables are implemented in both editables and setuptools use some sort of redirection logic to point an import to a path, and the only thing that can separate a working editable from a broken one is whether that redirection target exists (an existing path does not mean the redirected module will work, but that’s true for non-editable wheels as well, it’s not meaningful to get into that). PEP 657 already mandates the installer must create a direct_url.json describing where the original project is[1], and the only thing left to ensuring the editable is valid from that is to check that path to the original project actually exists—if it does, the editable installation is valid as far as the installer is concerned.
So IMO whether an editable wheel is valid for installation can be determined entirely if a Python project can be found where it’s expected. If there isn’t, something is very wrong and the installer should abort. If there is, either the entire workflow is working as intended, or the user somehow fished out the editable wheel and reproduced the project it was reditecting to on another machine, which probably means they know what they’re doing and we can just let things work? We can probably add some text to advice installers to do the file path check, and that’s about enough.
[1] But how? The PEP says the editable wheel must look like a regular wheel, and the installer must install it like a regular wheel. Assuming the build frontend is a separate component to the installer (e.g. pypa/build vs pradyunsg/installer), the installer must be able to figure out on its own whether a wheel is editable to do the right thing, does it not? Should the editable wheel contain a flag somewhere to communicate this information to the installer?
I agree, I don’t think @FFY00 and myself are in that much disagreement (at least, if I understand his position correctly).
I’m not clear what you’re saying here. Who is trying to determine if the wheel is valid? If it’s the installer that called build_wheel_for_editable, then it’s valid because it was returned from that call. If it’s another installer that has somehow got hold of that wheel, it has no way of knowing, because there’s nothing in that wheel that tells it where the original project is located (direct_url.json is created by the installer, it’s not part of the wheel, and the mechanism the wheel uses to do the redirection is unspecified, so you can’t introspect anything in the wheel to find out that information…)
I still think that this is not something we should worry about, but I guess if people want the extra cross-checking machinery, we could require backends to add a file to the .dist-info directory in the wheel (maybe editable_target.txt) that (1) signals that this wheel was created via build_wheel_for_editable and (2) contains the target project location. We’d have to include a recommendation that frontends add a check on that file for all non-editable installs, which I think is an unnecessary overhead given that the vast majority of installs are non-editable. And I still maintain that anyone determined enough to manage to get an editable wheel in the first place is entirely capable of bypassing any sanity checks we apply, so we don’t actually gain any practical benefit here. But the option’s there if we want to go down that route.
I think we’re talking at cross purposes here. I’m assuming that the only workflow that involves editable wheels is as follows:
Installer gets told “install this directory in editable mode” (pip install -e .)
Installer calls build_wheel_for_editable
Installer unpacks and installs the returned wheel, and adds the direct_url.json file containing the directory named in the original request.
Build frontends like pypa/build don’t provide a user-facing interface to build_wheel_for_editable, so “editable wheels” never appear in the wild¹.
Installers that only support pre-built wheels (pradyunsg/installer) ignore all of this and just unpack the wheel as per the wheel spec. They should never see editable wheels, so they ignore the possibility.
¹ pypa/build or another library might provide an API for installers that support --editable to use, but that’s not “user-facing” in the sense that I mean.
This is what I meant by “library APIs”. Does installer.install_wheel need a new argument telling it to write a direct_url.json file (which the code above can supply as from the variable project)?
This feels like a footgun for every “orchastrator” implementation.
Well, “orchestrator” implementations will need to write direct_url.json for other cases, so less of a footgun, more of a detail that they need to get right. Look at it another way, only the “orchestrator” knows whether an editable install or a normal install has been requested, so it’s up to that program to get the details right (call the right hook, install the wheel correctly, make sure direct_url.json has been installed). Libraries can include APIs to make that easier, but they can’t take the responsibility away from the front end altogether.
I repeat, I’m not against having something in the wheel, I just don’t know that it’s been demonstrated (yet) that it’s necessary.
That says something about how close the installer and backend need to cooperate to do an editable install. That also says that storing everything needed in the “editable” wheel so that it is possible to install it without additional information may not be trivial.
Let’s enforce that these wheels need their build tag to be “editable”.
pkg-1.0.0.dev0-editable-py3-none-any.whl
That’ll make it clearer what these wheels are for, addressing @FFY00’s concerns without more complexity in any of the other tooling. If folks wanna still use these wheels for weird things, nothing is stopping them, but sane people would likely notice the word editable in the wheel file name and stop.
This would replace the current recommendation:
Build-backends are encouraged to append a distinguishable PEP 440 local version
identifier, such as +editable, so as to make editable installs easier to identify
by users inspecting installed distributions.