Pronouncement on PEPs 660 and 662 (Editable installs)

After reviewing the two PEPs on editable installs, and the (extensive) discussion on the subject, I have made my decision.

PEP 660 is hereby accepted, and consequently, PEP 662 is rejected.

Thanks to the authors of both PEPs for the amount of work they put into developing the PEPs, handling the discussions and writing proof of concept code, and to everyone else involved in the discussions for useful feedback and comments.

As usual, I’ll provide a summary of the reasoning behind my decision below. Feel free to ignore it if you’re not interested :slightly_smiling_face:

First of all, I would like to put the proposals into context. If we had been designing an “editable installation” mechanism from scratch, then IMO the discussion would have been very different, and to be honest I suspect that ultimately we’d have ended up rejecting the whole idea. However, that’s not the situation we’re in - we have editable installations, implemented by setuptools since the very early days, and in use by many, many projects. So the goal here is not to design a feature, but to work out how to standardise and extend that feature (both to support other backends, and if possible to address problems with the existing implementation).

My approval of PEP 660 is based very much on the fact that it achieves the goal of standardising the existing functionality, while adding some support for improvement. It has its limitations (in particular, it doesn’t handle symlink-based approaches, and it concentrates mainly on sys.path entries) but those are acknowledged in the PEP and are in line with the existing .pth based setup.py develop mechanism, for better or worse.

The arguments made for more flexibility in letting the user control how the editable mechanism is chosen were strong, to the extent that they were probably the main concern I had with deciding in favour of PEP 660. However, PEP 662 in my view failed to address the major point raised by all reviewers, that the interface between the frontend and the backend was unclear, and badly specified. The point was made repeatedly, but ultimately the PEP’s only response was “that’s for the tools to decide”, which I don’t believe addressed the questions raised. I could have sent PEP 662 back for revision before making a decision, but the PEP author clearly stated that not specifying was a deliberate choice, so it didn’t seem like that was going to make any significant difference. As a result, I did not consider PEP 662 to have adequately defined the mechanism it was proposing, and I therefore did not feel that I could reasonably approve it.

7 Likes

As a follow-up point, I should note that I don’t believe that accepting PEP 660 closes the door on a “front end controlled” approach totally. In a very practical sense, I believe that PEP 662 can be implemented by taking PEP 660 as a starting point, and supplementing it with a wheel specification change. In fact, that’s basically what the final PEP 662 iteration does, just without formally standardising the editable.json file as part of the wheel spec. The “file mapping” mechanism from PEP 662 could be added to the wheel standard, with the rules as to how such a mapping gets realised by the front end being part of the wheel spec, rather than part of an “editable install” spec. There would be a lot of details to clarify (most notably, whether and how to prevent “general” wheels using the mechanism), but I’m comfortable that if the issues that PEP 662 was trying to address turn out to be significant in practice, we can handle them “on top of” PEP 660, rather than needing to abandon PEP 660 and start again from scratch.

I will be interested to see whether the supporters of the “front end controlled” approach want to explore this option further. My personal expectation is that end users will be more than happy with the simpler .pth based approach that PEP 660 supports, and it will be useful to have the option to defer consideration of more complex approaches until more concrete use cases have been identified.

1 Like

This is harsh, so I’ll post here one final time because I feel like I’ve already written it down way too many times. The interface between the backend and the frontend was purely the editables.json. Which has a JSON schema and is clearly specified for every entry in it. There’s no other interface in between them. I asked multiple times to clarify what’s underspecified, and I never really got a clear answer… and when people asked with specific I’ve answered.

Guess the use cases I’ve presented in my PEP under the example use cases have not been identified somehow. Sure most users are going to be happy with the sub-optimal current state of affairs, but that doesn’t mean we could not have used this opportunity to open the door for supporting it.

No thanks, with such harsh push-back I can better spend my time elsewhere. PEP-662 is dead, long live PEP-660. Was a bloody hard uphill battle all throughout.

5 Likes

I personally do not see how frontend editables can be built on top of PEP 660 without being faced with the same issue that led to PEP 662’s rejection but I might be lacking imagination :wink: I think PEP 660 will serve us well however and the .pth-based approach will indeed suffice. Thanks to everybody involved (on all sides!) for making editable installs happen. (And if there is any residual interest in frontend editables, the frontend-editables project is still there.)

5 Likes

It won’t - sorry, I should have been clearer. It will allow anyone who wants to pursue the “front end decides” approach to have another go at addressing the division of responsibility issue. But it won’t make that problem go away.

1 Like

For people reading this, I just want to clarify that PEP 660 is not “the .pth-based approach”. It is “the backend decides” approach.

PEP 660 being accepted means that your project only provides editable installs if your selected build backend provides them, and in whatever format your build backend provides.

If we keep referring to this as “the .pth-based approach” then we’ll forever associate the two, and it will cause even more confusion in the future.

5 Likes

I’m not sure how you got that PEP 660 is the .pth-based approach from that. I was quoting:

… and was agreeing that .pth files, which can be provided by a PEP 660 implementation, will do just fine.

1 Like

On a completely different process-y note, I feel like we should mark PEP 660 as Provisional (instead of Accepted) right now. That’s the approach we took with PEP 517 as well and I think it applies for similar reasons – to allow ourselves to make changes to the interface based on feedback from actually rolling this out to our users.

/cc @pf_moore since this is basically poking at the decision in OP (accept vs provisionally accept). :slight_smile:

2 Likes

I agree, that makes sense to me (and is something I should have thought of).

@sbidoul, could you make that change, please?

1 Like

I’ll not be at a PC for a while, so if there is any urgency please someone else do it.

That said there isn’t much to vary in that spec.

3 Likes

Yea, there’s no urgency here.

1 Like

Consider it a pre-emptive clarification :wink: (It’s also hard to make Discourse not direct the reply at the person you quote, because it seems to ignore you when you tell it not to make it a reply…)

One thing we’ve definitely seen in packaging is that the way we explain things matters. A recent example is that a lot of people think that pyproject.toml is PEP 517, because of the way the PEP number was used. That one is fairly innocuous, but if people start off thinking PEP 660 refers to one specific implementation detail, we could inadvertently cut off the innovation that we want to promote.

So let’s tell people that “PEP 660 enables your build backend to give you editable installs”, rather than something like “puts a .pth file in your environment” (which PEP 662 could also have done).

(Edit: In case it’s not clear enough, I know @layday didn’t do it explicitly/deliberately in the original post, but they came close. Other people in other threads have done it more directly, including Paul, who wrote the main implementation of it two different ways without using .pth files :wink: . I just wanted to call out the general concern here before it becomes more common.)

7 Likes

Thanks to everyone for taking an interest in this feature, and especially for building prototypes.

"PEP 660 helps to make Python distributions importable during development, for arbitrary build backends. This is called an ‘editable’ installation."

Join me in six months for the sequel, where we add symlinks to wheel‽

3 Likes

I have not followed all the discussions on editable installs. But after reading PEP 660 and PEP 662 this morning, my understanding is that PEP 660 specifies a concrete editable wheel approach (i.e. a back-end-driven implementation of the source–installation mapping), while PEP 662 specifies an abstract editable wheel approach (i.e. a specification editable.json for a front-end-driven implementation of the source–installation mapping).

I like the simplicity of the concrete editable wheel approach (it keeps the responsibilities encapsulated: the back-end is for building the files and the front-end is for installing the files). I am glad that the community has finally reached a consensus. Thanks to all the people involved in this feature.

2 Likes

I can’t quite tell if this PEP has implications for how extensions are rebuilt in an editable install. Previously in my project, prior to using PEP-517 compliant build environment, I had a Makefile that I ran whenever I wanted to rebuild C extensions. This was fast because it only rebuilt what had changed since the last build. Now when a PEP-517 environment is used the entire project is rebuilt every time I run pip install -e . because the files all get touched when moved by the build backend from the temporary build directory into the source tree. The workaround is to use python setup.py build_ext --inplace but that pins the project to use setuptools instead of whatever build backend the developer desires to use.

Are there plans to define a standard for smart extension rebuilding in this PEP somehow? Perhaps this is more of an issue for the build backends to deal with than the PEP…? I could imagine for example a build frontend having a “watch” mode where it asks the backend to rebuild any extension files it sees have changed.

1 Like

No. In-place and incremental builds were discussed extensively during the original PEP 517 discussions, and the conclusion was mostly “leave it to the backend”.

Pip currently (for non-editable builds) copies the source tree to a temporary location, so any incremental building a backend might do is likely to be ineffective, but we’re switching to “in-tree builds” in the next few releases, so that should change. For editable builds, I don’t know the precise behaviour of the PEP 660 support PR for pip, but I imagine it will be in-tree. So when setuptools gets a PEP 660 implementation (or whatever other backend you use gets one) whether and how it handles incremental builds will be the backend’s responsibility.

1 Like

Great, thanks for the info. If you’re right about pip eventually providing in-tree editable builds then my current Makefile approach to rebuild extensions incrementally should work again.

1 Like

Since the editable install exposes the source checkout, an in-place rebuild of the extensions should also work. pip install -e would be needed if the metadata has changed or if a tool needs to rebuild in a standard way. If you’re developing you can rerun your makefile or whichever build system you are using.

1 Like

Indeed, my Makefile workaround for incremental building does work, though each developer needs to ensure they have the build dependencies installed in their development environment so incremental building works (via python setup.py build_ext --inplace in the Makefile) as well as specified in pyproject.toml for isolated building of wheels etc. And they need to make sure they have the exact same versions of the build dependencies in their environment as those in pyproject.toml, especially for numpy where ABI incompatibilities can crop up if runtime and build versions mismatch. To make this the case we pin exact, identical dependency versions in both pyproject.toml and setup.cfg as an extras_require “dev” key and tell new developers to run pip install -e .[dev]. It’s not nice to duplicate things, so I hope eventually there is a nicer solution to incremental building with PEP 517 compliant projects. Sounds like it’s going to have to come from the individual build backends though.

1 Like

You’re correct that neither PEP 517 or this PEP have attempted to address that problem.

2 Likes