# Third try on editable installs **URL:** \ **Category:** Packaging\ **Created:** [April 23, 2020, 8:15pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986 "2020-04-23T20:15:28Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 23, 2020, 8:15pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/1 "2020-04-23T20:15:28Z")
So I took a look at flit. It has two kinds of editable installs. One uses symlinks to link the install into site-packages. The other adds a .pth entry, more like how setuptools does it and more compatible with Windows. I don’t think it would be practical to do the symlink option with enscons. It’s more like a set of tools to make it easy to build wheels than a fixed system. The files that wind up inside the wheel could come from all over, or be generated directly into the wheel and come from nowhere at all. It does have the concept of src\_root, where you put your code, usually “.” or “src”. For enscons a decent installer-driven develop() would be required to emit a dist-info directory in src\_root, and it would return src\_root as an absolute path, or as a relative path from pyproject.toml. The installer would maintain the .pth file or whatever mechanism it used to add us to PYTHONPATH. A develop() hook could return different keys in a dict depending on what kind of editable install the back end was able to support. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 8:16am UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/2 "2020-04-24T08:16:09Z")
I think we agreed on how editable install should work at last PyCon on the packaging summit. Someone needs to implement a POC of it, give it a test run and then we can formulate a PEP on is. Where things are blocked now is packaging tools need to implement it, give it a dry run: - someone needs to do the pip side (@pf_moore, @pradyunsg) - @pganssle knows more if setuptools made any headway on it or not, - @sdispater knows about poetry, - @uranusjr about flit (and probably pep517 too). ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [April 24, 2020, 8:35am UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/3 "2020-04-24T08:35:21Z")
I know there’s some very broad strokes, but I likely missed the MO if there’s actionable direction on what exactly to implement. I think I voiced this in another thread somewhere as well. I don’t know what to do, I hear people saying they know what to do, but never anyone explaining what exactly they want to do. (Edit: except @dholth, of course, and every attempt he made thus far was shut down with someone saying “we know what to do” without explaining what that is.) ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 8:50am UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/4 "2020-04-24T08:50:48Z")
@pganssle knows the exact details, but AFAIK he got held up with trying to detangle setuptools and distutils. Basically someone needs to: - propose a PEP-517 extension point for editable installs (we have a summary from last pycon summit with the expectation of this hook) via a PEP, - implement it inside a builder (setuptools, poetry, flit) - implement the usage of this within pip. Given these three items we’re good to go. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [April 24, 2020, 9:11am UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/5 "2020-04-24T09:11:25Z")
Digging up the notes from the packaging summit so others don’t need to (the link should take you directly to the comment containing the notes; you don’t need to read the thread from the beginning) > > > Recently, tests that were previously passing have started failing, possibly due to pypa/pip#6331. The error message is: > Obtaining file:///home/travis/build/jaraco/pip-run > ERROR: Error installing 'file:///home/travis/build/jaraco/pip-run':... Quasi-summary with thoughts for people to act on directly: - I think we should use symlinks if possible, `.pth` otherwise. Everyone likes to hate `.pth`, but that’s the best thing we have and standardising something else for this purpose is in yak-shaving territory IMO. We can discuss replacing it _after_ editable is done. - Regarding symlinks, Windows does support it somehow, and [virtualenv contains logic to detect it](https://github.com/pypa/virtualenv/blob/8be186a1155dc353f872b9cd227ab0edf58978b9/src/virtualenv/info.py). Maybe it’d be a good idea to split this into a support lib so we can share implementation? I’ll just tag @bernatgabor since you’re in this thread already 😛 - Both code and metadata should be linked in. File exclusion (have files in the source tree but not included in the installation) is not worth the effort, let’s not consider it (i.e. everything in the source tree will be editable-installed, ignoring metadata definition). - Similar applies to inclusion update, it’s unreasonable to magically add new links when metadata changes. Accept this and require the user to re-install. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [April 24, 2020, 11:54am UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/6 "2020-04-24T11:54:05Z")
> [@bernatgabor](#): > > propose a PEP-517 extension point for editable installs (we have a summary from last pycon summit with the expectation of this hook) via a PEP This is the bit I keep getting stuck on, because I can’t find the details. Can someone confirm, as a standalone post in this thread, not a link back to the previous discussion, exactly what the proposal is that needs to be written up as a PEP? I’m not bothered about a full PEP write-up, just a spec that’s clear and explicit enough to be implementable. If there isn’t a spec to that level of detail yet, then isn’t that where we’re stalled, rather than at the “someone needs to implement a POC” stage? ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 12:26pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/7 "2020-04-24T12:26:45Z")
@sumanah do you have the meeting notes from the packaging summit? ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 24, 2020, 1:00pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/8 "2020-04-24T13:00:09Z")
I’ve remembered that pkg\_resources requires an `.egg-link` file to look for metadata, develop installs add one of those and add a path to the editable source code to `easy-install.pth`. So instead of putting `.dist-info` in the editable distribution, Use the existing pep517 `prepare_metadata_for_build_wheel` to get `.dist-info` Have `develop()` hook do the in-place build, return where a .pth should point to (or a tree of symlinks, but do .pth first, it is easier). Return a dictionary with a `{ "src_root" : }` key. Install as if the new `.dist-info` and `.pth` were in a wheel. No special uninstall logic required. Suppose you create a tree of symlinks in `site-packages` and you want an editable install for `spam/ham` and a second package `spam/jam`. What would the installer do? Sorry I couldn’t make it to the packaging summit. Was at the conference but couldn’t make it to that bit. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 1:24pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/9 "2020-04-24T13:24:23Z")
The way I remember we agreed on an API that’s very declarative, does not contain any of the implementation details you’ve been proposing here. Basically the interface would specify explicitly which (and all) files would need to be made available within the site packages folder. Was initially proposed the way I remember by @ncoghlan and then acceppted; @pganssle probably know better the details. ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [April 24, 2020, 2:21pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/10 "2020-04-24T14:21:05Z")
> [@bernatgabor](#): > > @pganssle probably know better the details. The details are certainly available in the editable installs thread. @pradyunsg also has some notes I believe. I have not been particularly motivated to try and recap anything because every time I read one of these threads, the hostile tone puts me off and I end up closing the window. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 24, 2020, 2:47pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/11 "2020-04-24T14:47:12Z")
Lately I’ve found this forum and open source in general to be less stressful than dealing with COVID-19. Seriously. It’s a good distraction. I don’t think the recap is necessary. The problem should be better understood after another year of experience. ---
**Author:** ![sumanah](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sumanah/32/1722_2.png) [@sumanah](https://discuss.python.org/u/sumanah)\ **Post date:** [April 24, 2020, 3:38pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/12 "2020-04-24T15:38:05Z")
> [@bernatgabor](#): > > @sumanah do you have the meeting notes from the packaging summit? @bernatgabor It’s as @uranusjr [mentioned](https://discuss.python.org/t/third-try-on-editable-installs/3986/5). [Here are the notes on the future of editable installs from the Packaging Summit at PyCon North America 2019.](https://github.com/pypa/packaging-problems/issues/256#issuecomment-504790004) They were previously [in this Google Doc section](https://docs.google.com/document/d/1Wz2-ECkicJgAmQDxMFivWmU2ZunKvPZ2UfQ59zDGj7g/edit#heading=h.v7q9p2i36xvt) and I moved them to that packaging-problems issue comment. There were also discussions in [Pip 19.1 and installing in editable mode with pyproject.toml](https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553) (mostly before the summit) and in [Specification of editable installation - #40 by pganssle](https://discuss.python.org/t/specification-of-editable-installation/1564/40) – that is a direct link to a comment where @pganssle indicates why he thinks there ought to be a proof of concept before we work to standardize stuff with a PEP. Am also pinging @techalchemy since he was planning to be part of the effort as well, and @takluyver since Daniel mentioned Flit. Currently @uranusjr @pradyunsg and @pf_moore are committed to spending a fair amount of time on the pip resolver project, which I am the project manager on. Tzu-Ping and Paul will be done with that commitment in probably June, but Pradyun will still have some committed time on this project till the end of December (not going to go into complicated “we have 2 funders and different timelines for different parts of the project” details now). So that’s going to affect their ability to volunteer to work on editable installs. If there’s a specific deadline someone is trying to meet to ship better editable installs, then it would be good to know that. If anyone knows a company who would like to sponsor this work, please speak up and talk with them (see [the entry about editable installs on the Fundable Packaging Projects page](https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Provide_more_standardized_editable_installations)). And if anyone would like to help write some grant proposals to get funding through the Packaging Working Group for a proper push on this work, I’d love that! Message me - I can help you get started. In about two months, for instance, [the Chan Zuckerberg Initiative’s next Essential Open Source Software for Science funding opens for applications](https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/). And applications for [Mozilla Open Source Support funding](https://www.mozilla.org/en-US/moss/) can be submitted at any time. ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [April 24, 2020, 4:27pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/13 "2020-04-24T16:27:48Z")
Here is the current specification that I am suggesting we build a PoC for: [Specification of editable installation](https://discuss.python.org/t/specification-of-editable-installation/1564/21) Presumably with Bernat’s additional suggestion of adding a version number. It’s pretty dead-simple, and I don’t know that it’s worth going into much more detail until we know what’s feasible and what problems could come up, but I think it’s the minimal specification that meets the requirements of: 1. Back-ends don’t need to know anything about how to install files. 2. Front-ends handle entry point generation and installation. 3. Front-ends have the option of “installing” _only_ the files that will be visible in a non-editable install. There will certainly need to be more details fleshed out, but I still think that the next step is a proof of concept in `pip` and `setuptools`, since those are the most widely used tools and the tools most likely to run into backwards compatibility problems that will need to be addressed in the PEP. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [April 24, 2020, 4:43pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/14 "2020-04-24T16:43:15Z")
Thanks @pganssle. I have a somewhat-theoretical1 interest in this subject, and might be willing to work on a pip POC. However, as @sumanah pointed out, I am committed to other work for a while yet, so it would have to wait until that’s done. However, from my experience implementing PEP 517, I can say with some certainty that work like this goes far better if there’s a good backend implementation available before starting work on a frontend. So I think the initial priority would be a proof of concept in either setuptools, or another backend that’s simple enough to make adding tests into pip relatively straightforward. 1 i.e., I don’t use editables much, but I find the mechanisms for implementing them interesting in the abstract. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 24, 2020, 5:26pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/15 "2020-04-24T17:26:23Z")
The older thread concludes it is not convenient to implement the new feature “#3 Front-ends have the option of “installing” _only_ the files that will be visible in a non-editable install.” While features #1 and #2 are less likely to restrict how build systems work, just moving some of what `setup.py develop` already does into pip. I should be able to produce my own POC. > When I did take a crack at it before, the major issue was that `distutils` (and by extension `setuptools` ) doesn’t have a clean separation between “figure out all the stuff that needs to go into the package” and “put the stuff into the package”, it just sort of assembles the package as it goes. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 5:28pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/16 "2020-04-24T17:28:46Z")
> [@dholth](#): > > The older thread concludes it is not convenient to implement the new feature “#3 Front-ends have the option of “installing” _only_ the files that will be visible in a non-editable install Do you have a link to this? > [@dholth](#): > > While features #1 and #2 are less likely to restrict how build systems work Can you detail on these? ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 24, 2020, 5:33pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/17 "2020-04-24T17:33:05Z")
> [@Specification of editable installation](https://discuss.python.org/t/specification-of-editable-installation/1564/40): > > No change in status so far. Currently we have a generally good “big picture” idea of what this should look like: backend produces a a pseudo-wheel consisting of wheel metadata plus a mapping between the locations of the files as they should exist in the package and the locations that they exist on disk front-end uses whatever mechanism it deems proper to expose those files to Python in such a way that they will update as the files change on disk We need a proof of concept for that for both se… Earlier in this thread > 1. Back-ends don’t need to know anything about how to install files. > 2. Front-ends handle entry point generation and installation. > 3. Front-ends have the option of “installing” _only_ the files that will be visible in a non-editable install. ---
**Author:** ![bernatgabor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bernatgabor/32/3003_2.png) [@bernatgabor](https://discuss.python.org/u/bernatgabor)\ **Post date:** [April 24, 2020, 5:45pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/18 "2020-04-24T17:45:09Z")
I meant why you think those statements apply to those points 😄 ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [April 24, 2020, 5:49pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/19 "2020-04-24T17:49:51Z")
> [@dholth](#): > > The older thread concludes it is not convenient to implement the new feature “#3 Front-ends have the option of “installing” _only_ the files that will be visible in a non-editable install.” I mean, it requires a refactoring, but one that I think is important _anyway_, because it would allow for the deprecation and removal of `setup.py install`. AFAIK the way wheel works is that it will just run `setup.py install` into a workspace location and then zip up the results. If we refactor this now into a “build package manifest” step and an “install package from manifest” step, then both `bdist_wheel` and `develop` can invoke the “build package manifest” step. In any case, I think in this situation it’s more important to get the hook _right_ than it is to get it _right now_. Paul Moore and I are both busy working on higher priority things right now, but if you look in the thread, I’ve been suggesting a timeline that involves me trying to take a crack at it after the Python 3.9 feature freeze for months now. > [@dholth](#): > > I should be able to produce my own POC. You mean a PoC for `setuptools` or a PoC for something else? A `setuptools` PoC would, I think, be very useful. Creating a PoC for something like `flit` would be pretty easy and would make a `pip` PoC easier to do and so might be worth it, but I think we definitely _do_ need a `setuptools` PoC before we can move forward here. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [April 24, 2020, 6:12pm UTC](https://discuss.python.org/t/third-try-on-editable-installs/3986/20 "2020-04-24T18:12:34Z")
I’ll try to do a setuptools + pip protoype of a “simple” proposal in the next week. [Next page](https://discuss.python.org/t/third-try-on-editable-installs/3986.md?page=2)