# Standardising editable mode installs (runtime layout - not hooks!) **URL:** \ **Category:** Packaging\ **Created:** [May 5, 2020, 9:38am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098 "2020-05-05T09:38:35Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [May 6, 2020, 7:08pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/41 "2020-05-06T19:08:49Z")
If you mean “the front end doesn’t need to build a wheel” then sure. I wouldn’t actually build a wheel then unpack it. But the _backend_ can build a wheel, and provide it to the frontend via the existing `build_wheel` hook. Then there’s no change needed in the front end at all (other than to tell the backend to build an editable wheel rather than a normal one). ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [May 6, 2020, 7:13pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/42 "2020-05-06T19:13:46Z")
Thanks. We can produce the next POC with minimal complexity in both ends. ---
**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\ **Post date:** [May 12, 2020, 1:03pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/43 "2020-05-12T13:03:35Z")
I really like the “editable mode wheel” idea, where the package init module takes care of making the rest of the package contents findable. It not only avoids any dependency on the arbitrary code execution loophole in pth files, it also means there aren’t any side effects on the global sys.path needed, even for applications that import that package. Edited to add: for single file projects where there aren’t any submodules, and for any other projects with a non-empty ` __init__.py`, reading that code in and running it with `exec` should mostly do the right thing, as the module metadata variables will have been populated based on the location of the editable wheel install, not the original source code. Two key constraints on that would be: - ` __file__ ` would need to be updated to refer to the actual source file so relative path lookups work as expected - in the package case, any ` __path__ ` manipulation would need to occur before the real package init code was executed ---
**Author:** ![pradyunsg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pradyunsg/32/206_2.png) [@pradyunsg](https://discuss.python.org/u/pradyunsg)\ **Post date:** [December 9, 2020, 7:19pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/44 "2020-12-09T19:19:57Z")
What’s needed next to move this forward? ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 9, 2020, 7:33pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/45 "2020-12-09T19:33:49Z")
From what I remember we need some volunteers to create a POC for some backends by using editables package to validate the issue. Mainly on setuptools side, but ideally, also flit and poetry. Once we have 1-2 backends proving our approach works, we can write up a PEP and move on to accept it. @pf_moore am I right? ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [December 9, 2020, 8:59pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/46 "2020-12-09T20:59:45Z")
I believe so. Looking back in the thread, it was @pganssle who was keen on having POC implementations, so I’d be interested in his view. It would also be very good to have the `editables` package getting some actual use. At the moment, it’s not been battle tested at all. ---
**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\ **Post date:** [December 11, 2020, 2:12pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/47 "2020-12-11T14:12:58Z")
My proof of concept on pip side is [here](https://github.com/pypa/pip/pull/8212), showing how the frontend/backend interface could look like. I’m happy to work with anyone willing to prototype the backend side to move this forward. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 13, 2020, 12:53am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/48 "2020-12-13T00:53:11Z")
I wrote rudimentary editables support for enscons. It seems to work. It should be possible to automatically add the `editable` target to builds but at the moment it is boilerplate that would need to be added to every `enscons` project’s `SConstruct`. [https://github.com/dholth/enscons/pull/9/files](https://github.com/dholth/enscons/pull/9/files) ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 13, 2020, 3:36pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/49 "2020-12-13T15:36:26Z")
Figured it out. Today’s `enscons` release adds a `scons editable` target by pointing Paul’s `editables` package at `src_root`. ---
**Author:** ![daniele](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@daniele](https://discuss.python.org/u/daniele)\ **Post date:** [December 13, 2020, 4:00pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/50 "2020-12-13T16:00:41Z")
@dholth, the link to the `enscons` homepage from PyPI points to a defunct Bitbucket repo. I think it would be worth fixing it. When I looked a `enscons` a while back this gave me the impression that `enscons` is a dead project. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 13, 2020, 4:48pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/51 "2020-12-13T16:48:10Z")
Thanks for your concern. `enscons` was caught in the bitbucket Mercurial purge. I’ve automatically migrated everything over to github. Today’s release fixes those links. I also fixed the project URL, thanks. ---
**Author:** ![daniele](https://avatars.discourse-cdn.com/v4/letter/d/dfb087/32.png) [@daniele](https://discuss.python.org/u/daniele)\ **Post date:** [December 13, 2020, 5:09pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/52 "2020-12-13T17:09:35Z")
The “Project links: Homepage” link still points to Bitbucket. This is probably due to the fact that the project metadata still contains `url="https://bitbucket.org/dholth/enscons"`. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 13, 2020, 11:26pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/53 "2020-12-13T23:26:40Z")
I’m pleased with @sbidoul’s proof of concept and recommend it be standardized. The only thing that’s missing is a pointer to the source distribution in `pip list` a-la `pip 20.0.2 /home/dholth/opt/py38/lib/python3.8/site-packages` so that you can tell which installs are `editable` or not. ---
**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\ **Post date:** [December 13, 2020, 11:33pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/54 "2020-12-13T23:33:26Z")
> [@dholth](#): > > the only thing that’s missing is a pointer to the source distribution in `pip list` Indeed. The information is present in the installed PEP 610 `direct_url.json` with an editable flag and the file:// URL of the project directory, so it’s just a matter of displaying it. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [December 21, 2020, 3:38am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/55 "2020-12-21T03:38:10Z")
# Optional Hooks build\_wheel\_for\_editable * * * :: build\_wheel\_for\_editable(wheel\_directory, scheme=scheme, metadata\_directory=metadata\_directory, config\_settings=None) … Editable distributions are supported by building a wheel that, when installed, allows that distribution to be imported from its source folder. This was traditionally done by having `setup.py` install a `.pth` file into `site-packages`. It can be done more precisely by producing a proxy module that replaces itself with the target module on import. `scheme`: a dictionary of installation categories `{ 'purelib': '/home/py/.../site-packages', 'platlib': '...'}`. This makes it possible to use relative paths to the source code. Must build a .whl file, and place it in the specified `wheel_directory`. It must return the basename (not the full path) of the `.whl` file it creates, as a unicode string. May do an in-place build of the distribution as a side effect so that any extension modules etc. are ready to be used. The filename for the “editable” wheel need not contain the same tags as `build_wheel` but must be tagged as compatible with the system. If the build frontend has previously called `prepare_metadata_for_build_wheel` and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created `.dist-info` directory as the `metadata_directory` argument. \*\* do we care about matching metadata for ‘editable’ wheels? Backends which do not provide the `prepare_metadata_for_build_wheel` hook may either silently ignore the `metadata_directory` parameter to `build_wheel_for_editable`, or else raise an exception when it is set to anything other than `None`. An ‘editable’ wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [December 21, 2020, 9:37am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/56 "2020-12-21T09:37:11Z")
> [@dholth](#): > > An ‘editable’ wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly. Worth adding here that these wheels are not intended to be cached or redistributed. You’ve implied it, for those who understand the implications, but may as well explicitly mention these two points for complete clarity. From the POV of someone who’s implement this interface, this definition seems sufficient to me. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [December 21, 2020, 11:03am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/57 "2020-12-21T11:03:04Z")
I feel it’d be better to build the content directly into the directory, instead of a wheel. The only usage of an editable wheel is to be extracted shortly on the same machine for installation, so the zip format is only adding needless overhead. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [December 21, 2020, 11:21am UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/58 "2020-12-21T11:21:34Z")
> [@uranusjr](#): > > feel it’d be better to build the content directly into the directory, instead of a wheel By using wheel we guarantee that existing wheel installers (e.g. pip) can handle it even if they’re and older version. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [December 21, 2020, 1:10pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/59 "2020-12-21T13:10:57Z")
The whole point here is that we don’t want the backend to be installing directly, the backend is responsible for building the distribution but _not_ for installing it, installation is the responsibility of the frontend. And wheel is the only standard format for passing a “thing to be installed” around. For example, we don’t want to require backends to know how to write executable wrappers for console entry points. Also, IMO the argument for the “scheme” parameter seems weak. It’s supposed to be only to allow backends to use relative filenames for the install, but how much of a benefit that is, I’m not clear (after all, virtual environments aren’t relocatable, nor are wrappers). And having the scheme parameter is a strong temptation for backends to think it’s OK to install directly, so I’m concerned it will be an “attractive nuisance” in practice. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [December 21, 2020, 2:10pm UTC](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/60 "2020-12-21T14:10:16Z")
> [@pf\_moore](#): > > The whole point here is that we don’t want the backend to be installing directly, the backend is responsible for building the distribution but _not_ for installing it, installation is the responsibility of the frontend. And wheel is the only standard format for passing a “thing to be installed” around. I was not proposing for backends to install the editable directly into the target location (`site-packages`), but that backends can provide a directory to the frontend, instead of an archive file. This would make both writing and reading the built result easier (plain filesystem access instead of ZipFile), and likely more efficient since the built result is not likely to be moved around (unlike a real wheel, which is likely to be distributed elsewhere). There’s an additional benefit that it’d be easier to implement incremental updates to the built result, which is another area where local development would benefit vastly from. To be clear, I don’t really hold a strong opinion that a wheel is a bad idea. Bernat’s reasoning (the editable wheel would be usable by older installers) is compelling. It’s also not impossible to use a wheel for incremental builds either; zip entries can be updated in-place, and have last-modified timestamps. Updating `RECORD` entries might be a pain, but that can probably be abstracted away by some common library used by backends (the reverse of `installer`) so authors don’t need to worry about it too much. But it’s important to remember that the main advantage of using an archive format is the ease of distribution, and editables don’t really need to be archives because nobody would (should) distribute them. [Previous page](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098.md?page=2) [Next page](https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098.md?page=4)