Do we need to say that? Considering we don’t have any alternative supported project formats, can we just drop the pep-517/pyproject.toml based part entirely?
I’ll update PEP-662 soon to also use the wheel to transport the information (though not the files) so will be just as much weelish.
Well, the frontend is unaware of the editable aspect at all, hence why I think is more backend driven.
It’s frontend driven to the point where the frontend will treat editable wheels differently. Though I’d concede PEP-662 is both frontend and backend driven, as either can power the editable aspect.
We could, but IMO it’s relevant because it makes it clear that the proposal is extending the existing “legacy” concept of editable installs to the standardised pyproject.toml based format.
In which case PEP 662 is still being changed in a fairly fundamental manner, and I don’t think we should be trying to get the titles to capture the difference as it’s too like trying to hit a moving target…
Yes but both frontend and backend are involved in PEP 662, so “backend” vs “frontend” is still inaccurate, as @sbidoul said and you agreed.
My apologies if it feels like that. My intention was to point out that there are more pressing concerns with getting PEP 662 ready for pronouncement, and I’d rather you focused your time on those, or we’ll end up in a situation where it doesn’t matter because there’s only one proposal ready.
How about we give up on the whole “make the title express the difference” debate and just go for
PEP 660: Editable installs for pyproject.toml based projects (option 1)
PEP 662: Editable installs for pyproject.toml based projects (option 2)
People can read the abstracts to see the difference. I don’t think we should be catering for people who are going to make a judgement purely on the title, so why worry?
I personally think (wheel based) and (virtual wheel) is fine for differentiating these in the title.
Both of these clearly state that they build off of the wheel standard, while making it clearer that they’re doing different things — reusing the existing spec (wheel based) vs introducing new representations (virtual wheel). And, it’s up to PEP 662 to decide what “virtual wheel” means and how closely it resembles a “real wheel”.
Any reason we’re mandating this? Note this design does not allow a fast path to acquire the project dependencies. To find out what is your project dependencies you must first build the wheel. This is slightly sub-optimal compared to the wheel where the frontend may invoke prepare_metadata_for_build_wheel not to pay the expensive wheel build operation. The main use case where this is not that good is C-extension and unsatisfiable dependency sets.
Imagine a project that defines the following dependencies pytest>5 and pytest<5. Before this wheel is installed, frontends performer a dependency set resolve operation to find out if the dependencies can be satisfied or not. If the dependencies cannot be satisfied, there’s no point in building a wheel, and we might bail early with an error message. However, with this design, the front-end must first build the editable wheel to determine the project dependencies. If this build is compiling some files, this is a significant overhead to fail the install ultimately.
To solve this problem and keep symmetry with the normal wheel build, I recommend adding a prepare_metadata_for_build_editable, which operates similarly with prepare_metadata_for_build_wheel. And make build_editable operates similarly to build_wheel.
IIRC we considered it and dropped it because we felt the performance benefit would be marginal. For regular requirements, the installer needs to download, get metadata to check dependencies and version compatibility and possibly discard to download another version. Editables, on the contrary, are guaranteed to be available locally and the user wants to install that local directory. So if they are built and ultimately the dependency resolution fails, it is probably a rare situation and the local build can be reused when the user retries.
That said, I don’t have a strong opinion on this and I definitely see benefit to have a prepare_metadata for editables to gain complete symmetry with regular builds. It could simplify implementation.
As someone who will have to implement this would appreciate symmetry. It would make it easier to reason about it because build wheel/editable behaviour could be considered equivalent when talking about. Also would offer a performance boost for my use case before.
I’ll probably be the one doing it in pip (if PEP 660 it is) so it will simplify my life too . So if no one objects.
BTW, the hook is currently named build_wheel_for_editable. It’s explicit but a bit verbose. prepare_metadata_for_build_wheel_for_editable is a mouthful, as is get_requires_for_build_wheel_for_editable.
I’d personally be fine to shorten it and go for build_editable, prepare_metadata_for_build_editable and get_requires_for_build_editable. What do people think ?
I felt that you would be slowing yourself down, invoking a new build just for the metadata, and then building the metadata again when you build the entire wheel. It is not practical to re-use the first metadata when building the second wheel. Probably process startup time is the biggest waste since writing a few text files (the .dist-info directory) is fast.
Maybe if you think of just python files and UNIX. But in the world of c extensions the overhead is significant and costly if the resulting metadata is unusable. Also making it behave differently than build wheel makes implementation and reasoning about its behaviour harder.
It can’t because the PEP states that requires parts of the metadata can be different between wheel and editable hooks. And full wheel mandates c extension builds.
I think you’re suggesting we need the metadata by itself during the editable install so that pip can use it to install a build dependency, and then build the full editable install plus the side-effect in place build
No I’m not talking about build dependency, but install dependency, so pip can use it’s resolver to determine if once built the wheel is installable or not (before actually doing the editable wheel build). It also would allow symetry in design with the normal wheels, which would simplify implementations.
Since the metadata hook is optional, the editable build can behave in the same way as if the optional hook was always missing. To me this is better than the symmetry of having to implement a mostly superfluous hook.
Currently pip always calls prepare_metadata_for_build_wheel and relies on the pep517 library to fall back on build_wheel if it is absent. pep517's build_wheel in turn does not rebuild a second time when the frontend calls build_wheel after a prepare_metadata_for_build_wheel fallback.
So except for the subprocess calls this is as efficient as it can get I guess.
If we don’t have prepare_metadata_for_build_editable, the frontends will need a different logic for the editable and non editable paths, effectively doing something similar to what pep517 already does.
While I agree that prepare_metadata_for_build_editable is not strictly necessary in most cases (as I explained above), its presence or absence does not really harm performance, and symmetry simplifies reasoning and frontend implementation.
You’re right that it does not impact performance, however, if and only if the installation can be done. If the installation cannot be done, e.g. because conflicting install requires dependencies, the presence of this hook could speed up significantly the use case.
I’m happy to have this discussion now and to get the PEP correct, but can I please remind you that the PEP has been confirmed as ready for pronouncement. I was hoping to spend some time this weekend reviewing, and I don’t want to have to re-read things after they get updated. So will this discussion be resolved by the weekend (tomorrow midday, UK time, let’s say to be definite) or should I defer reviewing for a week, till the following weekend?
The responsibility is on the PEP authors to ensure that all relevant feedback has been received, and discussion has died down, before submitting the PEP. If you feel that you need more time, I’m absolutely OK with you taking that time, but please let’s not have a cycle of “it’s ready, no it isn’t, …”
The above comments also apply to PEP 662, which has been announced as being ready but is also looking at this question.
so will be just as much weelish.
. So 
if no one objects.