PEP 660: Editable installs for PEP-517 style build backends

Practically speaking, in PEP 660 the backend is doing the interesting / meaningful part of the installation. It has to choose how the files are exposed to the system (.pth file, the editables hack, etc), within some constraints offered by the PEP.

The backend is doing the interpreter mangling to make the editable logic work, however it does not handle any addition/removal of the files from within the python prefix. From this POV practically only the frontend is installing files. The backend only alters the interpreter startup logic (via the pth file installed by the frontend), so the way I see it does no installation.

These are not standardized and frankly I think no one knows what to do with them. I’ve never seen them used.

Consider what you would need to do in order to write a front-end that never installs via .pth files because for whatever reason your environment doesn’t work well with them:

  1. Every backend would need to be updated to support the editables mechanism for installation.
  2. Every backend would need the same configuration option in order to switch between the editables and .pth mechanism OR
  3. Your frontend would need to know the configuration options for every backend OR
  4. You’d need to maintain a list of different configuration options that people will have to pass to your front-end in order to do the switch, per-project.

In an alternative world where the backend does some compilation steps, then gives you a list of all the paths it wants to expose, you could write your own front-end that simply has a switch between .pth and editables mode, and it would work automatically for all projects.

Consider another hypothetical: You would like to write a “daemon mode” front-end, that instead of using a .pth or editables hack, watches all the files from an input and does incremental builds of real wheels, installing them in the background when files are edited. In PEP 660, you’d need to either parse the .pth files or the editables code or something to find out what files are actually being exposed, or you’d need to introspect the source project in a non-standard way and do non-standard editable installs. And note that PEP 660 allows the backend to expose the files however it wants, not just via those two mechanisms, so really you’re outside the standard either way you slice it.

In the “pass the front-end a virtual wheel and let it expose the files to the system” mechanism, your “daemon mode” file gets a list of files to watch, and can execute its builds whenever those files change.

This is pure semantics. The point is that the choices you make about how to expose the files to the interpreter are all done in the backend. There are real practical consequences to this and there are not comparable consequences to doing it the other way around. The relevant characteristics of “doing an install” are all in the backend in the PEP 660 proposal.

In this alternative world you’d struggle to build/compile files on the fly, when the user is requesting it (aka, only do it when needed, at import time). It would also mean the backend needs to trigger the reinstallation of the editable project whenever new files are added to the project. In my POV (and also the POV of the PEP as stated currently) this daemon mode functionality is the responsibility of the backend, not the frontend.

I’d say there’s no need for a daemon here though to poll the file system, the backend can just check at first import time if the current time of modification and the last known time of modificiation of the file changed, if yes, do a rebuild; if not use the already built file. In the spirit do the work as late as possible, when absolutely must.

1 Like

What are you talking about? The front-end can do exactly what the backend is doing in the current PEP 660 suggestion. It can make a .pth file or it can take the list of files and install symlinks or it can use the editables proxy mode to expose a proxy module. None of that needs to be done in the backend or re-implemented in every backend. The only difference between my proposal and PEP 660 in this regard is that in PEP 660 the front-end can’t make any of these choices because it doesn’t have enough information to do so.

Is this response separate from what I’m talking about? You are aware that backends are not long-lived processes, generally speaking? You’re suggesting that a library for building packages, rather than an application for installing and managing the installation of packages is the right thing to be installed as a system-wide, long-lived daemon? This does not seem like a good idea, considering that individual packages may rely on specific and possibly conflicting versions of their backends, so you may need many such backend daemons running.

I’m sorry that you don’t like the PEP. There were years available to produce a working alternative, but we came up with this. Other suggestions were not fully realized and may have been difficult to implement with existing backends including setuptools. We will probably standardize on Paul’s editables module.

1 Like

Yeah, that’s why I followed it up with the backend doesn’t need to be a daemon. It can just invoke itself when the import of a module is happening, on-demand. Also I don’t think we said every backend needs to implement this from scratch. That’s the idea of the editables project. To share this logic where we can unify things.

Also coming up with a list of files to expose is not always trivial. The way I remember you’ve tried to do this with setuptools in 2019 and concluded it can’t be easily done, and likely would need a major reorg to achieve it. I’m not aware of anyone actually completing this task.

1 Like

I wasn’t at the 2019 summit, and while I believe a summary was posted somewhere, I don’t know where.

Is the approach you’re talking about written up as a proposal anywhere? I’d be happy to review two competing proposals. But we need to remember that the lack of editable support in PEP 517 is giving users problems right now and has been for a long time. I’m not particularly keen on delaying further when we have a proposal on the table, if all we’re delaying for is the possibility that someone will have the time in the indefinite future to write up something that might turn out to be better.

Also, let’s not set the bar too high here - we’re in severe danger of letting perfection be the enemy of the good. Having something that’s simply no worse than the current setuptools-only mechanism will still be a significant improvement, for pip and for end users, as we’ll be one step closer to removing the current privileged position that setuptools has.

4 Likes

We didn’t exactly try very hard, so I don’t know that it’s not trivial.

For example, right now PEP 621 and PEP 643, both much easier to implement than even the current version of PEP 660, have gone unimplemented in setuptools for a long time. It’s not like there will be much motivation to implement PEP 660, either, particularly when it’s, you know, bad.

We had many long threads in which you asked this question and got thorough answers. I suggest searching for them.

It is probably true that I should have been less stringent about asking that we have a proof of concept for modern editable installs in setuptools before coming up with a formal PEP for it, since it seems the current modus operandi in Python packaging is to design the entire architecture first, accept the PEP, and then move on to the next PEP. In that world we don’t really have to worry about whether something’s feasible, since it is never implemented :wink: .

Yeah, I suppose this is true. If this PEP said, “This is a codification of the current status quo — it’s not intended to be good, it’s intended to be easy to implement. If we ever get to the point of implementing an alternative, it will be deprecated.” then I’d be much more OK with it. Instead, when I suggested that I don’t like this PEP, all the responses I’ve gotten seem to indicate that people think it is well-designed rather than a necessary compromise between expediency and design criteria.

Sorry you don’t like it.

Is it possible for a PEP 660 implementation to generate an install that would meet your standards?

By the way, some of the other newer packaging PEPs also made it into alternative backends before setuptools. The demand is higher in these new systems; they may not have a long-established alternative mechanism.

Well, PEP 660 has a proofs of concept in pip, flit and other backends.

I had a look at adding it myself to setuptools but I’m not familiar enough with the code base and I admit I got lost between wheel and setuptools. But it can’t be that hard to stick a .pth file in a wheel for someone familiar with setuptools. And even if no-one volunteers to do it, pip still supports legacy editables and there is no pressure to remove it.

Opinions… I’m sorry too you don’t like it, but that does not mean the design is objectively bad.

2 Likes

As someone that wrote a backend, this proposal is great actually and I’ve been using the PoC branch of pip in all of my environments :slightly_smiling_face:

2 Likes

I apologize for saying that your PEP is bad, I understand that it is a lot of work to put one together and the area of packaging in general is very contentious, so I recognize the work and courage it requires to put this together.

That said, I think I’ve given strong objective reasons why this is not an ideal design. Absent historical considerations that require additional work to overcome (which are not acknowledged as a motivating factor in the PEP!), I see it as strictly worse than at least one alternative that we’ve spent quite a bit of time discussing and fleshing out. It imposes many design constraints on the solution space of the problem for no additional benefit, so I think it is a bad solution to the problem.

That is not to say that a bad solution is not better than no solution, or to say that it’s not the best bad solution there is, just that I don’t think it’s a good solution. Absent justification for choosing this sub-optimal design, I don’t think it should be accepted.

2 Likes

Where is this coming from? This PEP has proof-of-concept implementations in flit, pip, enscons and more.

It also allows us to innovate on more strict models (like the ones you’re advocating for) purely on the build backend’s side without needing to go through a standardization process. If some build backends want to be stricter, they have the control to do so within this model.

The notes are cross-linked from here: PyCon US Packaging Mini-Summit 2019 - #60 by sumanah

The notes from the editable discussion are editable mode is not supported · Issue #256 · pypa/packaging-problems · GitHub.

As per my memory (and reading of the notes), the design of this PEP is basically what we agreed upon at the summit. The notes also state that you didn’t like this approach at that time either.

I see where @pganssle is coming from, and I agree. The current approach is not optimal, but I think it’s very reasonable IMO. This has been an issue for years and I don’t think the proposed design not being the most optimal solution for all use-cases should block it.

I think we have two options, either someone writes up an alternative PEP and we take both into consideration, or we go with the current one and iterate on it in the future if it makes sense.

I don’t think getting the current hook in as it is proposed should be a blocker to introduce an approach like Paul is describing. We could simply introduce a new hook, which offloads more into the frontend, and have frontends choose which one they prefer.
This is not the most optimal solution, but I think reworking this PEP from scratch and further delaying the standardization of editable installs will result in a smaller net value, especially considering that backends other than setuptools are gaining more popularity each day.

2 Likes

The fact that several packaging PEPs have gone un-implemented in setuptools, and thus are effectively unused in the greater Python packaging ecosystem, because setuptools is really hard to work with since it’s everywhere.

Again, the backend is not the relevant place for this to be “stricter”. It’s not even a question of strictness. All the interesting configuration for this depends on the user environment and preferences and not on the project itself. The same person may develop the same project on platforms with and without good support for symlinks. Having a per-project configuration for such a thing would be annoying at best and useless at worst.

If you want to make the case that this is as good as we can get implemented in a reasonable timeframe, that’s fine, but please take the time to understand and engage with my actual substantive points of disagreement.

You and I sat down together at PyCon 2019 and wrote a PEP based on the “virtual wheel” version, not anything even close to PEP 660. Here is a description of the high level concepts, clearly incompatible with what is being proposed here.

2 Likes

Indeed, and I stand corrected!

I’m certainly in this “boat” at the moment. Unless someone volunteers to pick up the task of writing a PEP draft for this approach and puts it up for discussion in a reasonable timeframe (~2-4 weeks?).

Sorry about that! I was still wading through the context and background, as I wrote my post and should’ve probably made it through it all before saying things. That said, your link to the notes helped remind/bring me up to speed on the sequence of events here. :slight_smile:

The virtual wheel idea has important problems. We believe that it cannot be implemented without refactoring setuptools. Might it also require you to re-run the editable install when you add or remove a .py file? Might it fail for imaginative backends that scatter the source files in a way that does not resemble the installed layout? These are some of the reasons why we got to PEP 660. There is not an implementable alternative proposal that solves the ‘don’t accidentally include setup.py’ problem without causing other serious problems.