PEP 660: Editable installs for PEP-517 style build backends

Packaging
103

These are not standardized and frankly I think no one knows what to do with them. I’ve never seen them used.

Consider what you would need to do in order to write a front-end that never installs via .pth files because for whatever reason your environment doesn’t work well with them:

  1. Every backend would need to be updated to support the editables mechanism for installation.
  2. Every backend would need the same configuration option in order to switch between the editables and .pth mechanism OR
  3. Your frontend would need to know the configuration options for every backend OR
  4. You’d need to maintain a list of different configuration options that people will have to pass to your front-end in order to do the switch, per-project.

In an alternative world where the backend does some compilation steps, then gives you a list of all the paths it wants to expose, you could write your own front-end that simply has a switch between .pth and editables mode, and it would work automatically for all projects.

Consider another hypothetical: You would like to write a “daemon mode” front-end, that instead of using a .pth or editables hack, watches all the files from an input and does incremental builds of real wheels, installing them in the background when files are edited. In PEP 660, you’d need to either parse the .pth files or the editables code or something to find out what files are actually being exposed, or you’d need to introspect the source project in a non-standard way and do non-standard editable installs. And note that PEP 660 allows the backend to expose the files however it wants, not just via those two mechanisms, so really you’re outside the standard either way you slice it.

In the “pass the front-end a virtual wheel and let it expose the files to the system” mechanism, your “daemon mode” file gets a list of files to watch, and can execute its builds whenever those files change.

104

This is pure semantics. The point is that the choices you make about how to expose the files to the interpreter are all done in the backend. There are real practical consequences to this and there are not comparable consequences to doing it the other way around. The relevant characteristics of “doing an install” are all in the backend in the PEP 660 proposal.

105

In this alternative world you’d struggle to build/compile files on the fly, when the user is requesting it (aka, only do it when needed, at import time). It would also mean the backend needs to trigger the reinstallation of the editable project whenever new files are added to the project. In my POV (and also the POV of the PEP as stated currently) this daemon mode functionality is the responsibility of the backend, not the frontend.

I’d say there’s no need for a daemon here though to poll the file system, the backend can just check at first import time if the current time of modification and the last known time of modificiation of the file changed, if yes, do a rebuild; if not use the already built file. In the spirit do the work as late as possible, when absolutely must.

1 Like
106

What are you talking about? The front-end can do exactly what the backend is doing in the current PEP 660 suggestion. It can make a .pth file or it can take the list of files and install symlinks or it can use the editables proxy mode to expose a proxy module. None of that needs to be done in the backend or re-implemented in every backend. The only difference between my proposal and PEP 660 in this regard is that in PEP 660 the front-end can’t make any of these choices because it doesn’t have enough information to do so.

Is this response separate from what I’m talking about? You are aware that backends are not long-lived processes, generally speaking? You’re suggesting that a library for building packages, rather than an application for installing and managing the installation of packages is the right thing to be installed as a system-wide, long-lived daemon? This does not seem like a good idea, considering that individual packages may rely on specific and possibly conflicting versions of their backends, so you may need many such backend daemons running.

107

I’m sorry that you don’t like the PEP. There were years available to produce a working alternative, but we came up with this. Other suggestions were not fully realized and may have been difficult to implement with existing backends including setuptools. We will probably standardize on Paul’s editables module.

1 Like
108

Yeah, that’s why I followed it up with the backend doesn’t need to be a daemon. It can just invoke itself when the import of a module is happening, on-demand. Also I don’t think we said every backend needs to implement this from scratch. That’s the idea of the editables project. To share this logic where we can unify things.

Also coming up with a list of files to expose is not always trivial. The way I remember you’ve tried to do this with setuptools in 2019 and concluded it can’t be easily done, and likely would need a major reorg to achieve it. I’m not aware of anyone actually completing this task.

1 Like
109

I wasn’t at the 2019 summit, and while I believe a summary was posted somewhere, I don’t know where.

Is the approach you’re talking about written up as a proposal anywhere? I’d be happy to review two competing proposals. But we need to remember that the lack of editable support in PEP 517 is giving users problems right now and has been for a long time. I’m not particularly keen on delaying further when we have a proposal on the table, if all we’re delaying for is the possibility that someone will have the time in the indefinite future to write up something that might turn out to be better.

Also, let’s not set the bar too high here - we’re in severe danger of letting perfection be the enemy of the good. Having something that’s simply no worse than the current setuptools-only mechanism will still be a significant improvement, for pip and for end users, as we’ll be one step closer to removing the current privileged position that setuptools has.

4 Likes
110

We didn’t exactly try very hard, so I don’t know that it’s not trivial.

For example, right now PEP 621 and PEP 643, both much easier to implement than even the current version of PEP 660, have gone unimplemented in setuptools for a long time. It’s not like there will be much motivation to implement PEP 660, either, particularly when it’s, you know, bad.

111

We had many long threads in which you asked this question and got thorough answers. I suggest searching for them.

It is probably true that I should have been less stringent about asking that we have a proof of concept for modern editable installs in setuptools before coming up with a formal PEP for it, since it seems the current modus operandi in Python packaging is to design the entire architecture first, accept the PEP, and then move on to the next PEP. In that world we don’t really have to worry about whether something’s feasible, since it is never implemented :wink: .

Yeah, I suppose this is true. If this PEP said, “This is a codification of the current status quo — it’s not intended to be good, it’s intended to be easy to implement. If we ever get to the point of implementing an alternative, it will be deprecated.” then I’d be much more OK with it. Instead, when I suggested that I don’t like this PEP, all the responses I’ve gotten seem to indicate that people think it is well-designed rather than a necessary compromise between expediency and design criteria.

112

Sorry you don’t like it.

Is it possible for a PEP 660 implementation to generate an install that would meet your standards?

By the way, some of the other newer packaging PEPs also made it into alternative backends before setuptools. The demand is higher in these new systems; they may not have a long-established alternative mechanism.

113

Well, PEP 660 has a proofs of concept in pip, flit and other backends.

I had a look at adding it myself to setuptools but I’m not familiar enough with the code base and I admit I got lost between wheel and setuptools. But it can’t be that hard to stick a .pth file in a wheel for someone familiar with setuptools. And even if no-one volunteers to do it, pip still supports legacy editables and there is no pressure to remove it.

Opinions… I’m sorry too you don’t like it, but that does not mean the design is objectively bad.

2 Likes
114

As someone that wrote a backend, this proposal is great actually and I’ve been using the PoC branch of pip in all of my environments :slightly_smiling_face:

2 Likes
115

I apologize for saying that your PEP is bad, I understand that it is a lot of work to put one together and the area of packaging in general is very contentious, so I recognize the work and courage it requires to put this together.

That said, I think I’ve given strong objective reasons why this is not an ideal design. Absent historical considerations that require additional work to overcome (which are not acknowledged as a motivating factor in the PEP!), I see it as strictly worse than at least one alternative that we’ve spent quite a bit of time discussing and fleshing out. It imposes many design constraints on the solution space of the problem for no additional benefit, so I think it is a bad solution to the problem.

That is not to say that a bad solution is not better than no solution, or to say that it’s not the best bad solution there is, just that I don’t think it’s a good solution. Absent justification for choosing this sub-optimal design, I don’t think it should be accepted.

2 Likes
116

Where is this coming from? This PEP has proof-of-concept implementations in flit, pip, enscons and more.

It also allows us to innovate on more strict models (like the ones you’re advocating for) purely on the build backend’s side without needing to go through a standardization process. If some build backends want to be stricter, they have the control to do so within this model.

The notes are cross-linked from here: PyCon US Packaging Mini-Summit 2019 - #60 by sumanah

The notes from the editable discussion are https://github.com/pypa/packaging-problems/issues/256#issuecomment-504790004.

As per my memory (and reading of the notes), the design of this PEP is basically what we agreed upon at the summit. The notes also state that you didn’t like this approach at that time either.

117

I see where @pganssle is coming from, and I agree. The current approach is not optimal, but I think it’s very reasonable IMO. This has been an issue for years and I don’t think the proposed design not being the most optimal solution for all use-cases should block it.

I think we have two options, either someone writes up an alternative PEP and we take both into consideration, or we go with the current one and iterate on it in the future if it makes sense.

I don’t think getting the current hook in as it is proposed should be a blocker to introduce an approach like Paul is describing. We could simply introduce a new hook, which offloads more into the frontend, and have frontends choose which one they prefer.
This is not the most optimal solution, but I think reworking this PEP from scratch and further delaying the standardization of editable installs will result in a smaller net value, especially considering that backends other than setuptools are gaining more popularity each day.

2 Likes
118

The fact that several packaging PEPs have gone un-implemented in setuptools, and thus are effectively unused in the greater Python packaging ecosystem, because setuptools is really hard to work with since it’s everywhere.

Again, the backend is not the relevant place for this to be “stricter”. It’s not even a question of strictness. All the interesting configuration for this depends on the user environment and preferences and not on the project itself. The same person may develop the same project on platforms with and without good support for symlinks. Having a per-project configuration for such a thing would be annoying at best and useless at worst.

If you want to make the case that this is as good as we can get implemented in a reasonable timeframe, that’s fine, but please take the time to understand and engage with my actual substantive points of disagreement.

You and I sat down together at PyCon 2019 and wrote a PEP based on the “virtual wheel” version, not anything even close to PEP 660. Here is a description of the high level concepts, clearly incompatible with what is being proposed here.

2 Likes
119

Indeed, and I stand corrected!

I’m certainly in this “boat” at the moment. Unless someone volunteers to pick up the task of writing a PEP draft for this approach and puts it up for discussion in a reasonable timeframe (~2-4 weeks?).

Sorry about that! I was still wading through the context and background, as I wrote my post and should’ve probably made it through it all before saying things. That said, your link to the notes helped remind/bring me up to speed on the sequence of events here. :slight_smile:

120

The virtual wheel idea has important problems. We believe that it cannot be implemented without refactoring setuptools. Might it also require you to re-run the editable install when you add or remove a .py file? Might it fail for imaginative backends that scatter the source files in a way that does not resemble the installed layout? These are some of the reasons why we got to PEP 660. There is not an implementable alternative proposal that solves the ‘don’t accidentally include setup.py’ problem without causing other serious problems.

121

Sorry. I’m aware we’ve had these threads, and I know there have been a lot of points made. My understanding was that the synthesis of all the discussion was PEP 660. So what I was asking was how your view of what was “proposed and agreed on” differs from PEP 660.

Looking at the 2019 summit summary (thanks for the link @pradyunsg!) I see the following:

  • Frontend says ‘give editable’ Declares its needs
  • Backend gives a dist, like a wheel, and provide where the files/list to install
  • The front end will decide how to expose this
    • Tbd. pth, something else?
  • Argument made (P Ganssle?) @pganssle that the B.E. shouldn’t have to know about the surrounding system

That sounds remarkably like PEP 660 to me, in particular “backend gives a dist, like a wheel”. So I remain unclear on what specifically you believe was agreed at the 2019 summit that isn’t captured by this. In particular, what difference between the 2019 agreement and PEP 660 locks us into managing install schemes per project the way you suggest.

Or we can accept that there’s been a lot of discussion since 2019, and just focus on what we have now - I didn’t mean to drag in a whole load of history that isn’t going to help us move forward :slightly_smiling_face:

I will freely admit that I haven’t re-read everything yet to make sure PEP 660 has addressed all the points made in the subsequent discussion. I intend to do that as part of reviewing the PEP for pronouncement, but the PEP hasn’t reached that point yet.

As PEP sponsor, I should probably remind the PEP authors that the PEP should explicitly reference any objections and/or concerns that have been made over the (long!) history of this discussion, and either include links to where consensus was achieved, or make a clear statement of why the PEP decided to reject a concern. And I’d encourage people who have raised concerns to flag where such links are missing. At the moment, the PEP is very light on such information, and I’d advise the authors not to submit it for approval before fixing that :slightly_smiling_face:

See this section of PEP 1, in particular:

The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion.

This is a good point. A PEP is supposed to make choices, and they won’t always be popular. But to be accepted the PEP should respond to concerns that are raised. In particular, people wanting to know why the design is the way it is should be able to read the PEP and see why certain ideas were rejected. That does mean that the onus is on people with concerns to be specific, though.

Re-reading @pganssle’s original comment, I think the PEP needs to address the comment:

That’s a specific concern, and deserves a response. Even if it’s simply “the PEP authors do not agree that this level of flexibility is needed, and no-one in the subsequent discussion supported the suggestion that it was”. (Which I believe is a fair summary of what’s been posted here so far).

There may be other specific points in Paul’s comments that are worth addressing too, but I’ll leave it to the PEP authors to extract them, with Paul’s help.

I don’t think that’s the case, and I certainly don’t want it to be the case. But I can see how it feels like that, we’ve had a number of PEPs that have been accepted which haven’t yet been implemented fully (in particular, by setuptools). I’ve not been pushing projects to implement approved PEPs, as I don’t see that as my role.

In the case of this specific PEP, people have done a lot of work implementing machinery, front end interfaces, and backend implementations. Sure, no-one contributed a setuptools implementation, but I don’t think it’s fair to dismiss all that has been done as “design the architecture and move on”.

I think this is something that warrants a separate discussion, and I’d welcome someone opening a conversation on it. I don’t have the time to do so myself, but IMO it’s not something that needs me to lead it - on the contrary, it needs community consensus, and in particular it needs backend developers to agree (most of the pending PEPs are for backend changes, I think?) It may even be that all that is needed is for people to pitch in and offer help to setuptools - I’m not sure if other backends are struggling to keep up with the new PEPs in the same way as setuptools is.

1 Like
122

The reason I think (only half-jokingly) that “setuptools or it’s not real” is because working on setuptools is both really hard and also the most important part of most of these things, because stuff that doesn’t work on setuptools, right now, may as well not be standards. de novo implementations of stuff are really easy compared to getting them implemented in setuptools in a way that won’t make people furious about some minor change in whatever change you’ve made.

If I ever have time to work on setuptools again (I have had vanishingly little time for OSS lately, and particularly less for setuptools, which was getting harder to work with reasonably last time I made any contributions), I’ll probably work on an implementation of PEP 643, which has the potential to actually make significant, almost instantaneous improvements to the whole ecosystem (especially with PEP 658 on the horizon). Maybe PEP 621 would also be next on the list so I can stop writing .ini files and to help confuse people about the pyproject.toml messaging in a different (but probably better) way than they’re being confused now.

I’m just saying that I’m not optimistic about PEP 660 being implemented any time soon in setuptools. I imagine that PEP 660 will be much easier to implement in setuptools, but I also think that virtual wheels are almost trivial to implement in flit or poetry or any number of other backends, so “we’ve implemented this version in a bunch of backends” is not exactly saying much compared to the other proposal.

I agree, and I suspect that the whole ecosystem is being held back by the lack of resources spent in making setuptools more usable and maintainable. If we had funding to hire someone for 6-12 months to work on refactoring and actively deprecating (and communicating deprecations) for setuptools, I think we could have made real progress on this and many other features. As it stands, I don’t think any active maintainers of setuptools even participate in these decisions that are allegedly binding on them!

I’m not like, less guilty of this in general — I was BDFL delegate on PEP 643 and approved it without making any sort of reference implementation, and even probably intimated that I would implement it.