Discuss PEP 662: Editable installs via virtual wheels

Packaging
21

Thanks for writing this draft, Bernát. It confirms my initial impression, though.

The point is not “pip vs other frontends”. But frontends (and pip in particular) must be able to handle anything that backends throw at them, otherwise we’ll end-up with projects specifying their supported frontends, and that will kill interoperability.

And, IMO there are so many edge case to handle with virtual wheels that the only practical way to achieve interoperability if is that someone writes a library that handles all these in a correct way (in addition to mentioning and specifying all the cases in the PEP text, otherwise we are back at an implementation-defined specification).

And to get the promised benefits of user control, that library must also implement editable styles options (assuming such a thing is possible at all).

That sure sounds like a lot of work. As I understand from reading the text, editables is far from covering all the cases allowed by this spec.

22

I think this is the biggest difference between the two camps here. One wants to make all frontends handle all backends by forcing all the lifting onto the build backend; while the other camp is alright with some incompatibility/difference between what some frontends can handle and what some cannot, at the benefit of allowing frontends to innovate individually.

That being said can anyone point a specific case where the current spec might allow one frontend to work and another not? Specifics, rather than broad statements so we can argue the topic better.

23

I thought the whole idea with ‘virtual’ wheels from the other thread, is that the backend would simply provide a mapping of all the files on disk (packages, modules, resources) to their would-be location in the wheel, for the frontend to do with as it pleases. If this PEP does not mandate that the backend actually do this then how is the frontend meant to take advantage of things like exposing files a la carte, at their right locations, universally? We’re back at being beholden to the backend. Furthermore, if the backend is actually allowed to determine the import mechanism (cf. the editables example from the PEP), how is the frontend supposed to sanely provide any kind of configuration for editable installs, which I thought was the aim of this counter-proposal?

24

Yes, but this to do with as it pleases is what is the issue here I think. Some as please mechanism might offer more or less functionality; and some projects might depend on this more functionality (e.g.,
a backend might really on symlink functionality to enforce an exclude list, a frontend using pth files instead can’t enforce this).

This PEP does mandate those files/resources but then it’s up to the frontend to make them available.

The frontend mechanism for which a configuration might be active is should a frontend use a symlink or pth file, is the question AFAIK.

PS. This PEP is my understanding of the virtual wheel proposal, @pganssle might have a differing view on this and is not endorsing this PEP currently.

25

I feel like that’s a mischaracterisation. Or maybe what I want isn’t in either “camp” :slightly_smiling_face: What I want is that users of any front end can successfully install any project in “editable” mode, regardless of which backend is used. Maybe that means “all frontends handle all backends”, but I don’t personally require that we “force all the lifting onto the build backend”. In fact, I don’t want anyone to do “all the lifting”. The point of interoperability standards should surely be that any frontend can operate with any backend.

The contention doesn’t seem to be around interoperability, though, it seems to be about who gets to choose the specific mechanism used for any given install. As far as I’m concerned that’s a new requirement (currently there’s no choice - pip/setuptools = .pth, anything else = doesn’t work) and not one that I have any personal interest in. But I’m not sure everyone is in agreement that it’s a valid requirement, although I believe that if stated in terms of “the user must be able to select which mechanism is used” then the two proposals do indeed place the responsibility for implementing the user’s choice in different places (although both allow the user to choose).

(Edit: To be clear, I was always perfectly happy with the idea that the backend implements one mechanism, probably by calling a library like editables, and that’s it. So I’m definitely of the view that nobody should be having to do any “lifting” here…)

Can you define “work”? One of the big problems here is that people seem to have wildly differing opinions on what constitutes a working editable install. There have been suggestions on the PEP 660 thread that requiring people to reinstall every time a new Python file is added to a project counts as “working”. I disagree, based on what current editable installs do, but maybe that’s OK. Conversely, I can’t see how a .pth based implementation could work at all for the proposal here.

Clearly if someone writes some code that implements a solution, it could be added to every frontend. So obviously the only way that one frontend would work and another not, is if the second frontend for some reason didn’t want to implement that code. An example of that might be an implementation in C, which pip wouldn’t be willing to use. But if you don’t accept “the frontend doesn’t want to implement that mechanism” as a valid objection, then obviously anything can be implemented, and “work”. But the same is true of PEP 660 - all backends can implement the spec, it’s not about whether they can, it’s about whether they will…

26

Um, isn’t that a strong argument in favour of the backend being in control of the implementation method (i.e. PEP 660)? Or am I misunderstanding you?

27

If you believe that the backend should be able to determine the import mechanism then the frontend should not be able to do the same, at least not without there being a way for either one to assume control.

Unless I’m misunderstanding, the PEP doesn’t require that the paths point to the files that would’ve been packaged in the distribution.

28

Coming up with the implementation detail of achieving an editable installation is what I consider here the lifting. Who has to implement the mechanism that makes an import resolve to a source tree file. And with PEP-660 this falls onto every build backend individually (who may share that code via the editables package - but the same way frontends can share this logic in a package).

This is not true the way I understand thigns. Currently, we have no standard solutions. pip/setuptools operates with pth files and flit operates with symlinks (note here flit is both frontend and backend). PEP-660 would make flits solution out of standard and only the pip/setuptools way is sanctioned.

In a world where this PEP is accepted some frontends can use the pth file solution of pip/setuptools, some frontends can choose the symlink path; or any other alternative mechanism they wish to do so. Does some mechanism over benefits over others? Sure. But the frontends may document this, and then the end-user can choose one which they’re comfortable with. Or some frontends might decide to implement multiple mechanisms and let the user choose.

There’s an example of this within the PEP. Can you detail what do you think is a limitation that stops you from doing this?

Not necessarily. Some solutions might be platform-dependent, and some frontends might be happy to target just some platforms. Frontends that intend to be cross-platform compatible are not required to adopt these solutions. Especially in corporate world being platform-dependent is not a big issue.

Yes, but should the backend be in control of choosing between symlink/pth or the frontend. With PEP-660 the backend is in charge, and cannot support a symlink solution. With this PEP the frontend is in charge and can support both mechanism, and provide an API for the end user to choose.

Depends. The user might be willing to settle for the pth path because its platform does not support it and accept that the exclusion list will not work. The idea here is to allow the frontend to have multiple strategies, each with its own trade-offs and put the choice in the end-users end, because the frontend is the one directly communicating the the end-user.

Considering the backend is the one generating the files to install there’s no way to disallow it to take over. However, the backend has that choice in his hands. Can decide to pass files through transparently and not have to deal with the complications of import hooks or not, and then implement some import hook based solution.

There’s no way to mandate this. E.g. when you’re merging files during the build you’re clearly not having a file that you can just point to. The paths for the simple cases should point to the on-disk source file, but in general, you can’t mandate that requirement, because often there’s no source to link in.

29

Agreed. But no-one has offered to work on such a library for frontends yet.

It doesn’t work unless the backend returns precisely that data. So a frontend implementing this fails to work with any backend that doesn’t supply precisely that information. So every time a backend chooses to return a different data structure, the front end (or the library that frontends use) needs to add new code for that?

So the user has to pick a frontend that supports all of the backends that their chosen projects use? That’s the scenario I said before doesn’t sound very user friendly :frowning:

Can you give a specific example here? I’m having a hard time following what you mean. What exclusion list are you talking about? You keep saying that the idea is for front ends to have multiple strategies, but you’ve not offered any reason why front ends would want to offer multiple strategies (at least none that aren’t in order to work around problems caused by hypothetical “projects might only work with one particular strategy” issues). PEP 660 on the other hand does not expect anyone to implement multiple strategies. The backend chooses a strategy and the frontend respects it.

One valid objection that should be made against PEP 660 is that it doesn’t support the symlink strategy, because wheels don’t allow symlinks. I will note that on the PEP 660 thread. If there are backends which want to use symlinks (flit being the obvious case) then they should pick that point up over there.

30

You can mandate in the PEP that it does not take over. A lot of things which are within the realm of possibility are prohibited in practice.

You can have a file that you can point to through an intermediate compilation step. There was talk in the other thread of backend daemons - this would be their purpose.

31

I haven’t had time to respond to this or the other thread and probably won’t for a few days, but when I reviewed this PEP earlier on in the process, it was decidedly not in line with my thinking for the virtual wheel proposal, particularly the complicated import hook mechanism.

It is certainly closer than PEP 660, but I think it makes things far too complicated, and unnecessarily so.

32

I’ve removed that in the latest version, and now only maps the files and provides the .dist-info.

1 Like
33

OK. I fairly carefully re-read the proposal here, and I’m still none the wiser on how I’d write frontend code to implement this. As a result, I came to the conclusion that I’d want any proposal that gets approved to come with a working “proof of concept” implementation.

PEP 660 has this, in the form of the editables project, @sbidoul’s POC implementation for pip, and @ofek’s use in hatch.

Looking back into the history, I note that this wasn’t an original thought with me :slight_smile: @pganssle originally called for a PoC here:

Now admittedly, he was asking for a PoC of the “virtual wheel” proposal, and what ended up being developed was the PEP 660 PoC. So, well, sorry about that :slight_smile:

But the point still stands. We should have proof of concept code, because fundamentally we’ve been going round in circles for a long time when we just talk in the abstract. So whether it’s this proposal, @pganssle’s or some yet-to-be-agreed compromise, I think we need some proof of concept code that:

  1. Implements the mechanism for taking the data returned by the hook, and making it into a working installation the current Python environment. It only needs to implement one mechanism for doing this, but it can implement more if there’s trade-offs that are worth demonstrating, and it should be clear what works, what doesn’t, and what options there are to address those limitations (see how editables handles namespace packages for an example).
  2. Demonstrates how a front end UI would look under the proposal - feel free to implement it for pip, but I’m fine if the example is a different front end (I appreciate that tackling a pip PR is a big ask).
  3. Demonstrates how this would be implemented in setuptools. Or another backend, but as the point of this proposal is that it makes it easier for backends to implement, I’d hope that a setuptools implementation is achievable in a PoC.

My reasoning on points (2) and (3) is that PEP 660 puts a lot of the work on the backend, so it’s fair to build a PoC that does the “easy” frontend task in pip, but the “harder” backend work in any backend, just to demonstrate it’s possible. Conversely, virtual wheel proposals should expect to do the “easy” backend work in setuptools, and the “harder” frontend work in whatever frontend it’s easiest to use to demonstrate that it can be implemented.

If we don’t get a “virtual wheel” PoC, then we’ll need to make a choice:

  1. Accept the proposal that’s delivered working code.
  2. Defer editable installs yet again, and continue with the status quo.
  3. Abandon editable installs altogether as something we can’t unify or standardise. Make the idea of backends providing a custom editable install mechanism (such as setup.py develop and flit install --symlink) the officially supported way of accessing the functionality.

If we did (3), we’d need to decide whether to drop --editable from pip when we remove the legacy setuptools-specific code. I fully expect a huge outcry if we did that, though :slightly_frowning_face: But that’s an internal pip issue, so not really relevant here.

34

Uh, I’ll try but this will take a few weeks :slight_smile: doing changes to pip/setuptools is not easy, and I guess this is the reason why PEP-660 used hatch/flit as POC :blush:

Sure, but why use a daemon process that needs to get notified and use resources when you can just manifest it in memory at impor time?

36

If the question is just “how would one implement this”, then it would make sense to just do a simple PoC using flit for the backend. I imagine it doesn’t matter much what the front-end uses since it’s a fairly new implementation. You could presumably make a custom editable-only installer for it based on installer or the like using pypa/build’s isolated environment. I imagine it will be fairly simple to modify a sufficiently well-organized wheel installer to do editable installs via symlink or via .pth file.

37

Well, because that places the burden on the backend to perform some sort of runtime wizardry to merge the two files. That’s how it’d be tackled in PEP 660 with, say, editables. I suppose what I’m asking is, what sets this PEP apart in that regard?

38

No change in this use case, but does allow the frontend to control if the installation is via pth file generation or symlinks. Being able to symlinks gives a way to do the installation without any additional files compared to what would happen with a normall install (while PEP-660 implies different source files in the platlib than what would be without editable install). So in this sense, this PEP is PEP-660 plus some more. The backend can still perform the wizardry that happens in PEP-660, but doesn’t have to, the frontend can do alternative solutions if the backend gives up on doing themselves all of it.

39

Backend and frontend aren’t the major focus here. It’s getting a working library that does the implementation, handles the API, etc, that matters. For example, the backend can return files in the “headers” directory. How does the implementation process those?

But if a proposal can’t offer an implementation of either the pip end of the protocol or the setuptools end, it feels like it’s going to end up not actually getting implemented, so maybe we shouldn’t approve it (as you said yourself @pganssle, we are in danger of ending up with a backlog of unimplemented standards, so I’m trying to take a somewhat stricter view on making sure we focus on proposals that will actually get implemented…)

40

Symlink it? How would PEP-660 handle it?

41

Whatever, I don’t really care, I’m just saying that your objection was that you don’t understand how it would be implemented. I think it might well be easier to understand that by implementing it de novo rather than implementing it in pip. I don’t really know much about the internals of pip, so maybe it’s already fairly trivial to do such an implementation anyway. :man_shrugging:

If it means that the implementation takes half as long and you get it weeks sooner, I would think a proof of concept not using pip should be fine to give you a better idea of the PEP. I think once the PEP actually passes, there will be a lot of enthusiasm to get it implemented in pip regardless of which one is chosen, considering that apparently multiple backends all implemented proofs of concept for PEP 660 (and a backend implementation for PEP 660 is basically a frontend implementation for this PEP).