To me the PEP read that it was only specifying the interface between build front-ends and back-ends when installing packages as editable, like PEP 517. It only suggests 2 example mechanisms for how those packages are editable. .pth files could be phased out, but that’s completely independent of the interface.
Is there a case when a standard user would even see this ephemeral wheel? If the format exactly matches, I would argue that having a different extension would only introduce more confusion for implementers down the line. The compatibility stuff should already be there from the build_wheel part of PEP 517.
I don’t think this is correct. There have been discussions to disallow arbitrary code execution from pth files. However, the intent is not to stop having this functionality but create a dedicated interface for it. See PEP 648 -- Extensible customizations of the interpreter at startup | Python.org for more details. So it’s more of a case of changing the implementation detail then to remove pth.
If your module is not being imported, it’s not being used; so it’s actually good/doesn’t matter that it’s not updated.
IMHO, maybe sometimes. But generally no. These wheels will be mostly ephemeral and we can easily add some metadata that makes them fail on other machines at install time (they would already fail at load time). I don’t like moving away from the wheel because now install frontends need to support a new binary. With the current design the install frontend never needs to change, only the backend that’s generating the blob. This would make it much easier to adopt it.
I guess this is semantics, but I’d say they are definitely planning on removing the executable functionality of .pth files, it’s just that they are proposing an alternative mechanism to solve the issue (completely outside .pth files).
This is not true, I know several people who are shipping data-only packages. I, myself, am planning to ship data-only packages.
Frontends don’t need to support a new binary, it’s the same binary, the only thing that changes is the name.
Frontends will need to change anyway because nobody is supposed to be pip install(-ing) my-editable-wheel.whl. The way this is intended to be used is by asking frontends to install the project in editable mode, the wheels should be ephemeral. The use case where frontends do not need to change is exactly the use-case that should not be happening, nobody should be manually installing editable wheels – I will not add support for the editable wheel hook in the CLI of pypa/build. The actual use-case that is intended will require frontends to change, as they now will need to use this new mechanism.
I think the argument that frontends do not need to change doesn’t hold almost any value in this situation, so, given that the wheels should not be used directly, but there isn’t really anything enforcing that, IMO we should be preventing issues where try to do just that, it will inevitably happen.
I have a really strong opinion here, for what that is worth.
From the POV of this PEP when that PEP is accepted we can just switch from using pth files to the new functionality. The PEP generally are referring to the latest accepted standards, to be independent of each other. For example, PEP 648 might get rejected, in which case pth files remain, and will remain until we create an alternative for them. We generally don’t drop functionality without offering an alternative.
The way CPython 3 is designed (and evolving) those data files should be loaded with importlib.resources and that goes through the import system also. The design here would not need to change to use that. If you want to use resource (data) files without going through importlib.resources then this might not work for you, but in that case, you might first standardize the alternative usage, and then we can come up with a solution for that.
I think you’re conflating a few things, so let’s try to clear this up. Components here are:
pip here is a combination of things, but you can think of as just an orchestartor. When you say pip install -e ., that roughly is translated by pip to:
provision a build backend to build an editable wheel (acts as a build frontend),
call the backend to build the editable wheel (for this and previous might just delegate the job to the build project),
install the editable wheel to the target environment (for this it might have it’s own installation logic, or again just delegate it to the builder - act as an installer).
So in this scope, if we change the extension the installer has the right to reject to install that as a wheel, per PEP-427. It’s no longer a valid wheel (note though this change affects the installer, not the build frontend). The separation of concerns is important because, for example, tox might decide to not use pip, or have its own implementation of the installer to speed up operations (virtualenv already does this when provisioning seed packages).
Maybe not pip install them, but definitely use installer to do so (as said various development workflow must install these generated wheels somehow). I’d argue though there’s no reason why not to allow pip install too.
As said above, build frontends only build the project. Installing wheels is the responsibility of wheel installers. When the wheel gets passed between the build frontend to the wheel installer the wheel needs to be persisted, it can’t be ephemeral.
We don’t need to be that draconian. There’s no real reason not to be backwards compatible here, and allow editable wheels to work when there’s a valid wheel installer available.
Perhaps this discussion and decision should be made together with the other build maintainers, not solely by you.
Kinda redundant, but yeah, me too. Like stronger than titanium.
@ffy00 do I understand correctly that the core of your concern is that you are looking for a way for frontends to detect that a wheel is an “editable” one ?
If there is no such way, I personally don’t think it would be a big problem in practice:
these wheels will normally not be exposed to users
the PEP recommends adding a local version identifier (such as +editable) so if a user finds one it will be easily recognizable as “editable”
even if the user insists on installing that wheel, not much harm will be done, as the only thing an installer frontend should do differently is generate an adapted direct_url.json
That said, if we conclude the risk is significant nevertheless, or that there would be other benefits, then I would prefer another approach than changing the .whl extension. Indeed it is still a valid wheel. As an alternative, we could, for instance, make the local version identifier containing the editable string a MUST for the backend.
We could also revive the proposal for a “local” platform tag that means “only valid on the machine it was built on” and mandate that editable wheels use that platform tag. But that can be a follow-up PEP, it doesn’t have to be part of the first iteration of the editable install spec. We could wait until we confirmed there’s a problem in practice before taking that route.
Installers are used to installing wheels these days, so if you look at the pip source code you’ll see that installing a not-wheel would be a separate and buggy endeavor. Rather than invent an almost-wheel we chose to use the format.
In the example a .pth file for editable would not need to be executable. It would only mention the source folder.
We don’t think it would be useful to copy these wheels to another computer, but if a clever person finds a way to make that useful in their specific workflow that would be fine.
My issue is not “whell vs not wheel”, I think it being a wheel is definitely the correct solution in this case, I just think these wheels should be differentiated from normal wheels – they serve a different purpose.
Here’s my issue, there’s nothing enforcing that. There is absolutely nothing preventing user i-like-caos from creating a tool that will build editable wheels and let users re-use them.
The issues we have seen with Python packaging over the years have not generally been due to bad design, but rather due to failure to predict how users will use, and abuse, the mechanisms.
User i-like-caos doesn’t see a problem, for them this is purely informational. They will try to use it as any other wheel.
This can cause some obscure errors. IMO the wheel should not be allowed to be installed in the first place, as it will generally successfully install and likely not work, which is bad UX.
The problem is that any wheel installer that does not know about this will gladly install it. We have issues getting users to move to newer pip versions, a lot of users will have access to such unequipped wheel installers.
Updating the wheel spec to add an editable marker would work.
This is a good idea, and something I have considered, however, I don’t see how that can be properly implemented. We would need to add a machine identifier, actually not only machine (hardware), but system. Most modern Linux distros will have this, systemd will handle it for you, on other distros we can use the DBus machine id, but there are systems that will not have such mechanism, or that it will not be very straight forward. And since we need to describe all this in the PEP, because the wheel builder and installer need to calculate the id the same way, it is not optimal.
An alternative, like I mentioned above, would be updating the wheel spec to include a editable marker. This would be essentially the same as having a different extension, I originally wanted to avoid it, but given the feedback here, it might make more sense.
editable would be different than local because it would only server this purpose. With local I would expect to be able to build a normal wheel in build_wheel and it not work on other systems, because the validation would fail.
Other than once installed the import would throw runtime error? Aka there would be no advantage in doing this. That being said you could add a meta-field into the wheel that identifies the machine, and disallow installation on other systems, but that can quickly become an issue with docker images and mounted disks (even with systemic ids and such).
@FFY00 suggests naming the wheel something like package-1.0-editable-none-any.whl and having the installer accept the editable tag only when it knows it is installing an editable wheel. An interesting idea… although my preference is to enable and not prevent behavior.
It’s not as straightforward as that, because editable wheels might system to update the code.
If this is allowed, I can see a build backend trying to be smart when rebuilding native code and guess the files that should be built – let’s say it will try to build all .c files under ~/someproject – which will try to build random files on my machine and will likely error out with obscure errors.
Yeah, as I explained above, getting a system id is tricky.
Could you describe the situations where you actually want this behavior? I am struggling to come up with anything.
In all situations I can think of, the user would be better off having something like pip generate the editable wheel and install it.
I don’t see much of an use for users to generate and then install editable wheels themselves, like I said in my first reply, this just seems like a footgun.
Anyway, sorry for taking over the thread
I will take a step back and come back later.
I wanted to allow this all the time, and actually thought of this exact problem when proposed the import hook solution to expose projects as editable. So for me, this is a feature, not an issue. It’s up to the build backend to make sure the errors it throws in this instance are not obscure; rather than excluding this use case. Also, note the backend doesn’t have to guess, it can be explicitly configured too. Also can restrict binary builds to be only allowed on in-project sources.
This may be a difference of perspective. When you are implementing the backend you have to put the wheel somewhere, and the filename should be different to avoid overwriting the regular wheel. In enscons I’m currently adding a tag to the rest of the wheel tags. I think I do like the idea of replacing the Python tag, or possibly the platform tag, with the word ‘editable’ and having the installer accept that in this mode. It solves the unique filename problem, prevents accidental distribution, and prevents accidental installation.
In normal use pip asks the backend to generate the editable wheel, and then pip installs that wheel. pip of course does not generate the wheel by itself.
I think we’re trying to move away from formulating things from the POV of pip (or at least that’s my impression - hence why we don’t have a pip build command). So it’s clearer to express this in terms of build frontend, build backend, installer and a workflows orchestrator. pip might do all these, or some parts of it and delegate the rest to other tools. Doing this distinction helps users understands better what changes are needed in what parts, and who is responsible of doing those changes to support editable installs.
I’m baffled by this comment. There’s nothing stopping someone doing that right now so I don’t see there’s any new issue here.
Personally, I think that attempting to prevent this (whatever “this” actually is…) is likely to cause more harm than good. Python has always worked on the principle that we’re all consenting adults, and I don’t want to see that change.
I think it might be worth adding a note to the PEP, addressing the question of “can this feature be misused”, noting that anything someone could do with this feature, they could just as easily do with a “normal” wheel already, so there’s no new potential for abuse (or security concerns) around the PEP.
This was very poor wording from my part. I was not advocating that we should somehow prevent something like this to be done, but rather that we should prevent it from leaking into the rest of the ecosystem.
At the end of the day Users (with a capital U) will not listen to me, or you, they will search something on Google and look at the first Stackoverflow response. So my point is, it would be very helpful to have the “official” tooling say “Hey you probably don’t want to do that, are you sure?”, instead of validating these actions. If I was a user and the top voted answer on Stackoverflow said that I could build editable wheels, stack them in a directory and pip install them when I want, I would probably take that for granted because they are just normal wheels, unbeknownst to me that unlike other wheels these ones not only depend on the system, but also on the machine state, and I shouldn’t be saving and re-using them.
Anyway, I think I have made my point clear. I don’t think a user should be able to build an editable wheel and manually install it with an end-user wheel installer, they should at least receive a warning.
IMO editable wheels should be marked and installers meant for end-users should refuse to install them, or at the very least give a warning. Probably the best option right now to mark the wheels would be by adding a tag to the wheel spec.
I’m not sure we have this distinction. Many tooling uses pip under the hood, so the only real wheel installer today (wheels) is both meant for end-user and machines.
