# Standardising editable mode installs (runtime layout - not hooks!) **URL:** \ **Category:** Packaging\ **Created:** [May 5, 2020, 9:38am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098 "2020-05-05T09:38:35Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 21, 2020, 2:18pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/61 "2020-12-21T14:18:47Z")
I was initially against using wheel but it turns out that the build backend knows how to make them. Mine puts extra files in the build directory that don’t make into the archive so it would be extra work to produce a clean directory. The ‘editable’ wheel will be tiny. It includes the metadata and a reference to top level modules. Most incremental builds will not change the ‘editable’ wheel. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 21, 2020, 6:29pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/62 "2020-12-21T18:29:08Z")
> [@uranusjr](#): > > This would make both writing and reading the built result easier (plain filesystem access instead of ZipFile), I’m not sure how much performance benefit we can have when reading a few kilobytes of file. Considering this wheel is likely to be just a few small shim files that expose the actual source files the difference probably is negligible. > [@uranusjr](#): > > There’s an additional benefit that it’d be easier to implement incremental updates to the built result, which is another area where local development would benefit vastly from. Build backends already have a local cache folder they can use to support incremental builds. The only benefit you would be saving is the performance hit for zipping and then unzipping those few small shim files. The drawback would be though that you’d need an installer that can understand directories as an editable install target (this time copy it please, don’t use the build wheel/sdist and install we do with them today), which would complicate the installer code. I think in this case using the wheel makes the logic easier for installers, and the price for it is a very small performance hit of zipping those few shim files that need to be installed (just to spell it out: an editable install will not install the package itself - which could be big - it only needs to install a few python+pth files that are loaded at interpreter startup and makes the actual source tree discoverable within the interpreter - IMHO the easiest is to just inject some importer hooks as @pf_moore editables project does). ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 22, 2020, 1:34pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/63 "2020-12-22T13:34:35Z")
Do we need `metadata_directory` here? Has anyone written a build backend that uses that parameter? I expect that for the `editable` wheel the metadata is likely to be slightly different especially if you count `RECORD` ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [December 23, 2020, 6:15am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/64 "2020-12-23T06:15:30Z")
`metadata_directory` is only meaningful if the frontend calls `prepare_metadata_for_wheel` first (and use this argument to pass the result back into the backend). pip has always used this two-step method to build wheels. The main benefit to this is that it can avoid building the entire wheel when only metadata is needed. It would be nice if `build_wheel_for_editable` would have the same mechanism, since it makes it easier to implement in pip 😉 But the performance benefit would be minimal here, since editables are always specified by direct URL or local path, and a wheel is almost always needed in the end. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 23, 2020, 2:44pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/65 "2020-12-23T14:44:41Z")
pip used to call setup.py egg\_info first to install dependencies before building and installing the package. Now it can call prepare\_metadata. The backend probably generates metadata twice which is fine because it’s very fast to write a few text files. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 23, 2020, 3:17pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/66 "2020-12-23T15:17:18Z")
[https://www.python.org/dev/peps/pep-0517/#build-wheel](https://www.python.org/dev/peps/pep-0517/#build-wheel) does have an optional metadata directory argument, so the backend can check if the content of that is up to date and not generate it again. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 23, 2020, 5:23pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/67 "2020-12-23T17:23:22Z")
Do any existing backends use that parameter? It seems like it would take more time to check than to rebuild the metadata. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [December 23, 2020, 5:30pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/68 "2020-12-23T17:30:28Z")
At least Setuptools does. And the backend does not need to check—PEP 517 mandates the frontend _must_ pass in a directory identical the backend had previously built with `prepare_metadata_for_build_wheel`, so it should be faster. And a backend is not allowed to ignore the parameter either, if it implements `prepare_metadata_for_build_wheel`. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 23, 2020, 6:27pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/69 "2020-12-23T18:27:37Z")
> [@dholth](#): > > It seems like it would take more time to check than to rebuild the metadata. Not really, for example, setuptools can calculate the hash of the `setup.py` and `setup.cfg` and if that stays constant do not regenerate the metadata. And that means it can avoid evaluating runtime code in `setup.py`, not? ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 23, 2020, 6:30pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/70 "2020-12-23T18:30:22Z")
Here’s setup tools build wheel not passing metadata directory down to the builder: [https://github.com/pypa/setuptools/blob/b6bbe236ed0689f50b5148f1172510b975687e62/setuptools/build\_meta.py#L214](https://github.com/pypa/setuptools/blob/b6bbe236ed0689f50b5148f1172510b975687e62/setuptools/build_meta.py#L214) ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 23, 2020, 6:32pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/71 "2020-12-23T18:32:08Z")
And as such there’s a bug ticket for it [https://github.com/pypa/setuptools/issues/1825](https://github.com/pypa/setuptools/issues/1825). ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 23, 2020, 7:46pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/72 "2020-12-23T19:46:59Z")
I don’t mind the directory parameter. Seems like as long as we can convince Paul that “scheme” is okay we can proceed to add this to pep517, or the packaging documentation or (hopefully not) a new pep. @takluyver @njs ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [December 23, 2020, 9:04pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/73 "2020-12-23T21:04:06Z")
> [@dholth](#): > > Seems like as long as we can convince Paul that “scheme” is okay I don’t think you need to convince me. I was only speaking on a personal level, so you can ignore me if you like 🙂 On a procedural note, and speaking with my packaging interop BDFL-delegate hat on, I think this is more than we should just do as an edit to PEP 517. I think this deserves a new PEP defining editable mode properly. So the next step is for someone to put together such a PEP, start a discussion on it and (hopefully fairly quickly) confirm consensus. I’m happy to sponsor the PEP if you need a sponsor. ---
**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\ **Post date:** [March 20, 2021, 2:33pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/74 "2021-03-20T14:33:25Z")
I rebased my [pip proof of concept](https://github.com/pypa/pip/pull/8212) and made a quick [poc in flit](https://github.com/takluyver/flit/pull/400). @dholth since there does not seem to be any serious objection to your proposal above and @pfmoore is kindly proposing to sponsor, I’m considering to convert your text into a PEP. Do you agree ? I’m inclined to drop the `scheme` argument in a first version for the sake of simplicity in the specification and implementation, knowing it could still be added later as an evolution. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [March 20, 2021, 4:25pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/75 "2021-03-20T16:25:33Z")
Sounds like a plan 🙂 ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [March 20, 2021, 10:10pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/76 "2021-03-20T22:10:54Z")
Thanks Stéphane. I’ve been thinking it was time to complete this and I’m very glad you’re willing to do the pep… if it really can’t be just a packaging standard. Hopefully this thread and the github discussion get us most of the way there and I’ll be happy to help edit. ---
**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\ **Post date:** [March 29, 2021, 3:54pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/77 "2021-03-29T15:54:06Z")
> [@sbidoul](#): > > I’m inclined to drop the `scheme` argument in a first version for the sake of simplicity in the specification and implementation, knowing it could still be added later as an evolution. Good plan 👍 ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [March 29, 2021, 4:40pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/78 "2021-03-29T16:40:45Z")
It’s not realistic to expect that it will be added later as an evolution. Otherwise the plan is okay. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [March 30, 2021, 6:14pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/79 "2021-03-30T18:14:27Z")
One thing that came up in [this issue](https://github.com/pfmoore/editables/issues/3) is that the implementation technique in `editables` doesn’t support exposing single Python modules as editable on `sys.path`, only directories. Personally, I think that’s fine, and backends should be permitted to only expose packages, and not single-file modules when creating an editable-mode wheel. But I do think the PEP should be explicit about this - and should probably require backends to at least give a warning if they don’t produce an editable install that exposes the same things on `sys.path` that a normal install would. It’s worth noting that setuptools’ `setup.py develop` doesn’t handle single-file modules properly either - it installs them, but it does so by exposing the whole directory they are in on `sys.path` (to demonstrate this, try doing `import setup`, which runs `setup.py`). ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [March 30, 2021, 7:09pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/80 "2021-03-30T19:09:39Z")
I think the PEP has no right to tell backends what they must do, but it should describe the history of editable installs starting with PYTHONPATH, the mechanism of creating a special wheel, and the current ‘gold standard’ of stub modules that expose no more and no less than what would be installed. [Previous page](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098.md?page=3) [Next page](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098.md?page=5)