# PEP 660: Editable installs for PEP-517 style build backends **URL:** \ **Category:** Packaging\ **Created:** [May 2, 2021, 10:38am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510 "2021-05-02T10:38:07Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![gbdlin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/gbdlin/32/4038_2.png) [@gbdlin](https://discuss.python.org/u/gbdlin)\ **Post date:** [May 7, 2021, 1:08pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/62 "2021-05-07T13:08:21Z")
I think I need some clarification. Currently the tool “stack” consists of: - Build backend - responsible for building the package. Chosen by the package maintainer. - Build frontend - responsible for calling the build backend and spitting out a ready to be installed `.whl` file. Chosen by the package end-user. - Installer - responsible for installing a `.whl` file created by build backend and build frontend working together. Chosen by the package end-user. We can consider 2 different scenarios regarding the end-user part (build frontend and installer): - two separate tools, one being a build frontend, and one working as an installer (e.g. `pypa/build` and `pradyunsg/installer` pair). - one tool working both as a build fronend and an installer (e.g. `pip`) Am I right so far? Given that, which part of the stack is responsible for creating a `direct_url.json`? If it’s the installer, how can it know when it’s installing an editable wheel? How can it know what exactly should be inside a `direct_url.json` file? Of course if the installer is consolidated in one tool with the build fronted, this information can be just passed internally, but it won’t be possible when using 2 separate tools. If that’s the case, I think we should either: - move the responsibility to the build frontend or build backend, - provide an interface to pass this information to the installer, - restrict the editable builds feature to tools that are working both as a build frontend and an installer (which I think should be avoided). ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 1:30pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/63 "2021-05-07T13:30:39Z")
Personally, I consider the PEP 517 workflow to be 1. Build backend - responsible for implementing the PEP 517 hooks and doing the actual build - creating wheels, etc. 2. Build frontend - any tool that calls the PEP 517 hooks. Build frontends can be any of a number of types of application: installers like pip, tools like build that manage creation of wheels/sdists, or just adhoc scripts. Some frontends, notably installers, and in particular pip, want to be able to do an “editable install”, where the installed project references the “live” source code of the project. This PEP adds a new backend hook that says “give me the necessary file structure to make an installation that works as an editable version of this project”. The hook is defined to work by returning a wheel format file, because front ends already know how to process such files (that’s frankly the _only_ reason). The `pradyunsg/installer` project doesn’t really fit here at all. It’s not a PEP 517 build frontend at all. In this context it’s simply a library that handles the job of unpacking wheels, which an installer application _might_ use (but doesn’t have to, any more than it has to use any given HTTP library). The editable install PEP also makes some requirements on tools that install (unpack) wheels returned from `build_wheel_for_editable`, specifically around writing a `direct_url.json` file. So as a PEP 517 frontend supporting the various “database of installed packages” specifications, pip would be responsible for writing that file, just like it’s responsible for writing the `RECORD` and `METADATA` files. `pradyunsg/installer`, as a library for unpacking wheels, might choose to help by writing `direct_url.json`. But it doesn’t have to - and it doesn’t for any of the other uses of `direct_url.json` PEP 610 applies to, so there’s no obvious reason why it would in this special case. Hope that helps - yes, it is complicated 🙁 ---
**Author:** ![xafer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xafer/32/248_2.png) [@xafer](https://discuss.python.org/u/xafer)\ **Post date:** [May 7, 2021, 1:34pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/64 "2021-05-07T13:34:49Z")
Thanks for the clarification on `direct_url.jon` @sbidoul, I clearly skimmed over PEP-610 too quickly 😖 Your second point on `scheme` brings an other question. The PEP currently only says: > `scheme`: a dictionary of installation categories `{ 'purelib': '/home/py/.../site-packages', 'platlib': '...'}`. This makes it possible to > use relative paths to the source code, which might help the interpreter find > the package after the root path changes with `chroot` or similar. But it is still unclear for me why `scheme` is actually needed for building an editable wheel (and/or why it isn’t needed for “classical” wheels and the `build_wheel` hook). Finally, I’m uncomfortable with the recommendation to add a `+editable` local identifier to the version. It seems like a hacky solution to an unclear problem. What should happen if the version already has a local version identifier ? I would expect `pip install .` and `pip install -e .` to install the exact same version. The explanation of “_so as to make editable installs easier to identify by users inspecting installed distributions_” seems weak to me since the `direct_url.jon` file should already contain this information. If we want to make editable wheels identifiable, I’d be more comfortable with a new key in `{distribution}-{version}.dist-info/WHEEL`. And in fact, on a slightly off-topic note, I think that it might make sense to also store there the backend (and its version) used to produce the wheel. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 1:41pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/65 "2021-05-07T13:41:01Z")
> [@xafer](#): > > But it is still unclear for me why `scheme` is actually needed for building an editable wheel (and/or why it isn’t needed for “classical” wheels and the `build_wheel` hook). Agreed. I’m not clear why this is needed, either. The `editables` project doesn’t need that information, so I’m not sure what the backend would do with it if it’s using `editables`. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [May 7, 2021, 1:41pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/66 "2021-05-07T13:41:20Z")
We are calling the wheel editable but it’s the project that is editable. The wheel is intended to be generated then immediately installed and deleted. The scheme allows the backend to produce relative paths to the project. Normal wheels do not care about the relationship between the source code and the installed path. The `dist-info/WHEEL` file already stores the name and version of the backend. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 1:45pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/67 "2021-05-07T13:45:09Z")
> [@dholth](#): > > The scheme allows the backend to produce relative paths to the project. Given that moving virtual environments isn’t supported, is this needed? ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [May 7, 2021, 1:48pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/68 "2021-05-07T13:48:18Z")
Is the editable wheel always installed into a virtual environment, and do we want relocating to be even less supported? ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [May 7, 2021, 1:59pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/69 "2021-05-07T13:59:55Z")
> [@dholth](#): > > Is the editable wheel always installed into a virtual environment, I don’t think so. E.g. within a docker image you can use the system Python. Relocating is probably out of scope, at least from my POV. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 2:03pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/70 "2021-05-07T14:03:09Z")
No, and yes 🙂 To be more explicit, what is the actual use case that having the schemes (and having relative paths in the wheel) is intended to support? Is it acceptable for backends to ignore the scheme and always use absolute paths? If so, what functionality will the users lose? Currently setuptools appears to use absolute paths in editable installs, so there’s no precedent for relative editables that I can see. Is there any reason the PEP is trying to add support for something that no existing tools allow? ---
**Author:** ![xafer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xafer/32/248_2.png) [@xafer](https://discuss.python.org/u/xafer)\ **Post date:** [May 7, 2021, 2:25pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/71 "2021-05-07T14:25:42Z")
> [@dholth](#): > > The `dist-info/WHEEL` file already stores the name and version of the backend. Right, I somehow also missed that (PEP skimming is clearly not my forte 😅 ) This could then maybe be expanded to also include the used hook and when relevant the `config_settings` as JSON ? We currently have: ```auto Wheel-Version: 1.0 Generator: bdist_wheel (0.36.2) Root-Is-Purelib: true Tag: py3-none-any ``` which might become ```auto Wheel-Version: 1.1 Generator: bdist_wheel (0.36.2) Generator-Hook: build_wheel_for_editable Generator-Config-Settings: {"CC": "gcc"} Root-Is-Purelib: true Tag: py3-none-any ``` ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [May 7, 2021, 3:09pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/72 "2021-05-07T15:09:35Z")
Doesn’t the PEP only exist to support things that no existing tools allow? A generic way to link source repositories to Python environments? I thought it might be useful to support relative paths. We could allow `scheme` to be `None`, and then both the backend and the frontend could ignore the feature if they choose. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 3:29pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/73 "2021-05-07T15:29:00Z")
Personal opinion: The PEP should be defining a standard way for tools to implement useful functionality. The existing setuptools `setup.py develop` mechanism has proven usefulness, and people have explicitly asked for that functionality in a form that isn’t tied to setuptools. So my feeling is that “support the features setuptools offers across all backends” is the baseline here. I don’t actually think we need anything more than that - the setuptools mechanism appears to have been perfectly successful in spite of its flaws and limitations, so while I personally prefer a more precise and detailed spec, I’m fine with _not_ trying to pin down details that setuptools has survived without. It’s perfectly OK for the PEP to offer extra features that setuptools doesn’t have, but as a front end maintainer I’d want to limit the amount of complexity I have to support, and I’d be against extra complication just to provide something that “might be useful”. If there’s a good justification, then fine, but otherwise let’s keep things simple. In this case, having to know the install scheme when we request the editable wheel, and not just when we install that wheel, seems like unnecessary extra coupling that might make it more difficult for an installer. At the very least, it adds a risk of bugs if what gets passed to the hook doesn’t match what’s used to install. And currently the PEP doesn’t say what should happen then, so there’s no guidance on how to fix such bugs. Furthermore, it’s a _huge_ warning sign to me that I’d expect the setuptools implementation of this new hook to completely ignore the scheme argument. PEP Delegate opinion: If the PEP contains a feature, I’d expect it to include an explanation of why that feature is needed. “It might be useful” isn’t sufficient, and I’d reject the PEP with a request to provide better use cases if it were submitted for approval in that case. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [May 7, 2021, 3:38pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/74 "2021-05-07T15:38:36Z")
That’s no fun at all. Hasn’t setuptools been deprecated since 2013 when wheel came out? 🙂 I have noticed that with Docker & containers I am looking at my code through a different root than the one it will run under more often than not, through a bind mount. That’s the main reason I think relative paths will be increasingly useful. ---
**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\ **Post date:** [May 7, 2021, 5:21pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/75 "2021-05-07T17:21:16Z")
> [@xafer](#): > > Finally, I’m uncomfortable with the recommendation to add a `+editable` local identifier to the version. > It seems like a hacky solution to an unclear problem. It is true that direct\_url.json provides the information that the install is editable, and might suffice for some use cases. Nevertheless I proposed this for the following reasons (which I will clarify in the PEP unless we conclude it’s a bad idea): 1. a casual look at the list of installed versions, possibly with tools that do not support PEP 610 yet, would reveal the editable-ness by looking at the version 2. the version found in installed .dist-info metadata may be out-of-date compared to the actual version of the project and the +editable local version identifier can serve as a warning for that situation for users looking at installed distributions 3. public index servers reject wheels with local version identifiers, so if one escapes and someone tries to upload it, it will be rejected 4. also in case the wheel escapes, it clearly shows in the file name that it is not a regular wheel 1 and 2 can be covered by tools that enhance their UI using PEP 610, but tools that are not updated for that would readily benefit from the editable local version identifier. 3 and 4 would need alternatives if we remove that from the PEP. > [@xafer](#): > > What should happen if the version already has a local version identifier ? If there is already a local version identifier, I’d recommend appending `.editable` to it. IMO, re-reading PEP 440, that usage is not an abuse of local version identifiers. Your proposal to update the WHEEL metadata sounds interesting to me. Complementary I’d say ? ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 7, 2021, 6:21pm UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/76 "2021-05-07T18:21:52Z")
> [@sbidoul](#): > > IMO, re-reading PEP 440, that usage is not an abuse of local version identifiers. Hmm, this makes me think. How would this work with version specifiers? If foo `1.0+mylocal.editable` is installed, as a result of installing `1.0+mylocal` in editable mode, that wouldn’t satisfy a requirement `foo==1.0+mylocal`: > If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison. Yes, it’s an obscure corner case, but PEP 440 goes into a lot of detail about corner cases like this, and I don’t think we want to have to deal with people complaining that installing a local version in editable mode breaks their dependency data… Because of this, I’m inclined to say that editable installs must have identical metadata to the equivalent non-editable install. That seems far simpler, and ensures that there’s no possibility of differing behaviour (it’s also what setuptools currently does, so we know it doesn’t cause people issues in practice). The point about public indexes is a valid one, but I don’t think it’s worth the risk of compatibility problems. So my counter-proposal would be: 1. Backends MUST ensure that wheels generated by `build_wheel_for_editable` have exactly the same metadata as if they had been built using `build_wheel`. 2. Tools SHOULD properly support PEP 610 and `direct_url.json`, so that they can correctly report editable installs. 3. Formally add the `local` platform tag, noting that it is intended to reflect “the machine the wheel was built on only” (even though in practical terms, tools won’t be expected to try to enforce that) and that installers MUST always include `local` in their list of supported platforms. Indexes MUST NOT allow uploading of wheels with the `local` platform tag. Installers should prioritise `local` wheels over more generic wheels like `manylinux` or `any`. 4. `build_wheel_for_editable` MUST return a wheel with the `local` platform tag. As indexes don’t allow uploading `local` wheels, this protects against editable wheels getting accidentally published. Yes, it’s more work, and it requires us to change more of the ecosystem to bring in the `local` tag, but it’s much more in line with existing standards, and (to me, at least) it feels like a much cleaner design with fewer edge cases to worry about. The `local` platform tag also has a history of people requesting something like it, so it already has a good justification. The biggest problem might be that I think some people argued that `local` wheels should have a _low_ priority. If that becomes a sticking point, we could change the tag to `editable` and say it’s only for editable wheels, if that avoids an extended debate. ---
**Author:** ![FFY00](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ffy00/32/4007_2.png) [@FFY00](https://discuss.python.org/u/FFY00)\ **Post date:** [May 8, 2021, 12:24am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/77 "2021-05-08T00:24:56Z")
Requiring the same metadata prevents us from adding extra dependencies, which we will likely need if the package requires any generation/compilation step. ---
**Author:** ![nhatkhai](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nhatkhai/32/3950_2.png) [@nhatkhai](https://discuss.python.org/u/nhatkhai)\ **Post date:** [May 8, 2021, 1:29am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/78 "2021-05-08T01:29:49Z")
This is exactly the architecture which creat it own problems. All of these lengthy discussion cycling back to the same point. Wheel would never cover all the user cases event if one try as hard as they could. It is by design. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 8, 2021, 8:41am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/79 "2021-05-08T08:41:49Z")
> [@FFY00](#): > > Requiring the same metadata prevents us from adding extra dependencies Well spotted, thanks! Editables adds a dependency so I should have thought of that 🙂 ---
**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\ **Post date:** [May 8, 2021, 9:55am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/80 "2021-05-08T09:55:23Z")
@pf_moore pinning on a local version identifier _and_ installing it as editable is indeed an edge of the edge case I had not considered. Well spotted. I’m going to remove that part of the text and move it to rejected ideas. To make the wheel generated by `build_wheel_for_editable` recognizable as such in case it escapes, shall we combine an editable build tag and the `local` platform tag ? So: `pkg-1.0.0.dev0-editable-py3-none-local.whl` ? I’m not sure which spec needs updating to introduce the `local` platform tag, nor what the text would look like. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 8, 2021, 10:28am UTC](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510/81 "2021-05-08T10:28:45Z")
> [@sbidoul](#): > > shall we combine an editable build tag and the `local` platform tag The wheel spec says build tag must start with a number. It looks like it’s more intended to be a build _number_ than a textual tag (it sorts by initial numeric part then remaining text), so I fear this may be an abuse of the field, and could cause problems with someone’s weird build pipeline that we don’t know about 🙁 > [@sbidoul](#): > > I’m not sure which spec needs updating to introduce the `local` platform tag The [platform compatibility tags](https://packaging.python.org/specifications/platform-compatibility-tags/) spec. Which is derived from [PEP 425](https://www.python.org/dev/peps/pep-0425/) which is unfortunately pretty vague, so I think writing a bunch of standalone text and bunging it in as a standalone section in the compatibility tags spec would be sufficient. My suggestion: > ## Local platform tag > > Wheels may use `local` as a platform tag. This tag should be interpreted as meaning that the wheel is only valid on the exact system it was built on. Using a wheel with this tag on any other system is unsupported. > The `local` tag should be treated as the highest priority of platform tag by installers, to be chosen in preference to any other platform tag. > The `local` tag should _not_ be combined with other platform tags in a “compressed tag set” - the interpretation of such a combination is not defined by this spec and installers are allowed to interpret such a combination in an implementation-defined manner (including raising an error and refusing to install such wheels). Arguably, adding this should be a separate PEP, but I’m personally comfortable with making it part of the editable PEP, for convenience. If anyone wants to insist that it’s a PEP in its own right, please speak up. One thing I’d note about adding a `local` platform tag is that it adds a whole bunch of extra options to the list of “Compatible tags” for a system (what `packaging.tags.sys_tags()` returns). On my Windows PC, it adds 21 new tags to the existing 33 (a `local` version of every `win_amd64` tag), getting the list up to 54 entries. On Ubuntu (wsl) I see 87 at the moment, with 19 more being added. While that’s not a disaster, I do think that the “list every combination you accept” mechanism for tags is starting to show signs that it might not scale indefinitely 🙁 [Previous page](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510.md?page=3) [Next page](https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510.md?page=5)