So I took a look at flit. It has two kinds of editable installs. One uses symlinks to link the install into site-packages. The other adds a .pth entry, more like how setuptools does it and more compatible with Windows.
I don’t think it would be practical to do the symlink option with enscons. It’s more like a set of tools to make it easy to build wheels than a fixed system. The files that wind up inside the wheel could come from all over, or be generated directly into the wheel and come from nowhere at all. It does have the concept of src_root, where you put your code, usually “.” or “src”.
For enscons a decent installer-driven develop() would be required to emit a dist-info directory in src_root, and it would return src_root as an absolute path, or as a relative path from pyproject.toml. The installer would maintain the .pth file or whatever mechanism it used to add us to PYTHONPATH.
A develop() hook could return different keys in a dict depending on what kind of editable install the back end was able to support.
I think we agreed on how editable install should work at last PyCon on the packaging summit. Someone needs to implement a POC of it, give it a test run and then we can formulate a PEP on is.
Where things are blocked now is packaging tools need to implement it, give it a dry run:
I know there’s some very broad strokes, but I likely missed the MO if there’s actionable direction on what exactly to implement. I think I voiced this in another thread somewhere as well. I don’t know what to do, I hear people saying they know what to do, but never anyone explaining what exactly they want to do. (Edit: except @dholth, of course, and every attempt he made thus far was shut down with someone saying “we know what to do” without explaining what that is.)
Digging up the notes from the packaging summit so others don’t need to (the link should take you directly to the comment containing the notes; you don’t need to read the thread from the beginning)
Quasi-summary with thoughts for people to act on directly:
I think we should use symlinks if possible, .pth otherwise. Everyone likes to hate .pth, but that’s the best thing we have and standardising something else for this purpose is in yak-shaving territory IMO. We can discuss replacing it after editable is done.
Regarding symlinks, Windows does support it somehow, and virtualenv contains logic to detect it. Maybe it’d be a good idea to split this into a support lib so we can share implementation? I’ll just tag @bernatgabor since you’re in this thread already
Both code and metadata should be linked in. File exclusion (have files in the source tree but not included in the installation) is not worth the effort, let’s not consider it (i.e. everything in the source tree will be editable-installed, ignoring metadata definition).
Similar applies to inclusion update, it’s unreasonable to magically add new links when metadata changes. Accept this and require the user to re-install.
This is the bit I keep getting stuck on, because I can’t find the details.
Can someone confirm, as a standalone post in this thread, not a link back to the previous discussion, exactly what the proposal is that needs to be written up as a PEP? I’m not bothered about a full PEP write-up, just a spec that’s clear and explicit enough to be implementable.
If there isn’t a spec to that level of detail yet, then isn’t that where we’re stalled, rather than at the “someone needs to implement a POC” stage?
I’ve remembered that pkg_resources requires an .egg-link file to look for metadata, develop installs add one of those and add a path to the editable source code to easy-install.pth. So instead of putting .dist-info in the editable distribution,
Use the existing pep517 prepare_metadata_for_build_wheel to get .dist-info
Have develop() hook do the in-place build, return where a .pth should point to (or a tree of symlinks, but do .pth first, it is easier). Return a dictionary with a { "src_root" : } key.
Install as if the new .dist-info and .pth were in a wheel. No special uninstall logic required.
Suppose you create a tree of symlinks in site-packages and you want an editable install for spam/ham and a second package spam/jam. What would the installer do?
Sorry I couldn’t make it to the packaging summit. Was at the conference but couldn’t make it to that bit.
The way I remember we agreed on an API that’s very declarative, does not contain any of the implementation details you’ve been proposing here. Basically the interface would specify explicitly which (and all) files would need to be made available within the site packages folder. Was initially proposed the way I remember by @ncoghlan and then acceppted; @pganssle probably know better the details.
The details are certainly available in the editable installs thread. @pradyunsg also has some notes I believe. I have not been particularly motivated to try and recap anything because every time I read one of these threads, the hostile tone puts me off and I end up closing the window.
Am also pinging @techalchemy since he was planning to be part of the effort as well, and @takluyver since Daniel mentioned Flit.
Currently @uranusjr@pradyunsg and @pf_moore are committed to spending a fair amount of time on the pip resolver project, which I am the project manager on. Tzu-Ping and Paul will be done with that commitment in probably June, but Pradyun will still have some committed time on this project till the end of December (not going to go into complicated “we have 2 funders and different timelines for different parts of the project” details now). So that’s going to affect their ability to volunteer to work on editable installs. If there’s a specific deadline someone is trying to meet to ship better editable installs, then it would be good to know that.
Presumably with Bernat’s additional suggestion of adding a version number.
It’s pretty dead-simple, and I don’t know that it’s worth going into much more detail until we know what’s feasible and what problems could come up, but I think it’s the minimal specification that meets the requirements of:
Back-ends don’t need to know anything about how to install files.
Front-ends handle entry point generation and installation.
Front-ends have the option of “installing” only the files that will be visible in a non-editable install.
There will certainly need to be more details fleshed out, but I still think that the next step is a proof of concept in pip and setuptools, since those are the most widely used tools and the tools most likely to run into backwards compatibility problems that will need to be addressed in the PEP.
Thanks @pganssle. I have a somewhat-theoretical1 interest in this subject, and might be willing to work on a pip POC. However, as @sumanah pointed out, I am committed to other work for a while yet, so it would have to wait until that’s done.
However, from my experience implementing PEP 517, I can say with some certainty that work like this goes far better if there’s a good backend implementation available before starting work on a frontend. So I think the initial priority would be a proof of concept in either setuptools, or another backend that’s simple enough to make adding tests into pip relatively straightforward.
1 i.e., I don’t use editables much, but I find the mechanisms for implementing them interesting in the abstract.
The older thread concludes it is not convenient to implement the new feature “#3 Front-ends have the option of “installing” only the files that will be visible in a non-editable install.” While features #1 and #2 are less likely to restrict how build systems work, just moving some of what setup.py develop already does into pip.
I should be able to produce my own POC.
When I did take a crack at it before, the major issue was that distutils (and by extension setuptools ) doesn’t have a clean separation between “figure out all the stuff that needs to go into the package” and “put the stuff into the package”, it just sort of assembles the package as it goes.
I mean, it requires a refactoring, but one that I think is important anyway, because it would allow for the deprecation and removal of setup.py install. AFAIK the way wheel works is that it will just run setup.py install into a workspace location and then zip up the results.
If we refactor this now into a “build package manifest” step and an “install package from manifest” step, then both bdist_wheel and develop can invoke the “build package manifest” step.
In any case, I think in this situation it’s more important to get the hook right than it is to get it right now. Paul Moore and I are both busy working on higher priority things right now, but if you look in the thread, I’ve been suggesting a timeline that involves me trying to take a crack at it after the Python 3.9 feature freeze for months now.
You mean a PoC for setuptools or a PoC for something else? A setuptools PoC would, I think, be very useful. Creating a PoC for something like flit would be pretty easy and would make a pip PoC easier to do and so might be worth it, but I think we definitely do need a setuptools PoC before we can move forward here.
