Yeah, work in progress, but we’ll get there in a few weeks 
I do quite like my symlinking approach - it feels like the obvious, straightforward way to achieve what I want from an editable install, and it doesn’t have the same unwanted effect as .pth files of making adjacent files importable. I really only implemented the .pth option as a fallback for Windows. But it seems like I’m an outlier on this, as both this PEP and the proposal in the previous round of discussions I remember don’t allow for symlinking the package into site-packages.
I’m expecting to maintain and use the symlinking method in Flit even if some other way of doing editable installs gets standardised. This has limited my enthusiasm for getting involved with the discussions (I doubt I’ll find time to properly catch up on this thread - sorry).
2 Likes
My PEP, see https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat, would allow that.
2 Likes
I don’t think @takluyver would benefit from reading this whole thread. 
This PEP exposes a somewhat improved status quo to build systems. You’re either looking for that or not. We also know this PEP is easy to implement.
Will we be able to provide all things from a single PEP?
I think it’s great if an alternative build system can offer an improved editable experience through its own command. You might choose the alternative for that reason.
Here is another possible approach to work with symlinks in PEP 660.
build_wheel_for_editable can generate a symlink structure under $PROJECT_ROOT/.editable/$COMPATIBILITY_TAGS/ with symlinks that point to the actual files that need to be exposed. In the wheel that is communicated to the frontend, it can then add a .pth file that point to $PROJECT_ROOT/.editable/$COMPATIBILITY_TAGS/.
This way the backend can do a mapping from editable to source that is as simple or sophisticated it wants, while keeping simplicity and predictability on what the frontend will do.
2 Likes
I’d be happy to accept a PR to editables that implements something like this. I don’t really want to try to implement it myself, though, as I don’t have sufficient experience with symlinks to know how to robustly confirm if they work or how to fall back if needed, etc. But if no-one else picks up on it I can try.
As a starting point, maybe log it as a feature request for editables and then it won’t get forgotten.
Edit: Or maybe it doesn’t need anything in editables - if the backend is OK building the link farm itself, then just asking editables to supply a .pth file is possible with the existing API.
Also, just as a matter of curiosity, do they need to be symlinks if they are local to the project directory? In the absence of symlink support the backend could fall back to hard links (which are available on NTFS on Windows since forever) - there’s no risk of needing cross-FS support in that situation.
Some old time Unix editors break hard links (create a copy) when you edit a file, like a copy on write filesystem.
I’ve been wondering if vscode also does this.
1 Like
I’m not sure that solves data files and include files support, though granted PEP-660 purposefully only wants to handle purelib/platlib (and even there no pyd/so files, no?),
I’m not sure what the requirements are for that, actually.
How so ? pyd/so is well supported with PEP 660.
Edit: or rather does not need to have explicit support as a rebuild/reinstall is expected when it changes.
While I’m throwing ideas, here is another one regarding the choice of editable style.
In a first approach I’m saying the frontend can communicate the editable style to the backend via config settings. I think we don’t need to standardize that immediately.
But if it is deemed important, we can imagine a mechanism where the backend has a hook to expose the list of supported styles, and then let the front end ask the user to pick one of them and communicate it to the backend when building the wheel.
2 Likes
TBH my personal experience with symlinks in debuggers and IDEs has often been less than optimal (with breakpoint not being hit because they are set in the source file and not the symlinked path, the same symlinked file opened twice in the editor, etc). Not sure if that’d be better or worse with hard links. Or if it’s just me not using the tools correctly 
1 Like
Do we have “the front end controls the linking method” in the rejected ideas section of PEP 660 yet? We think the PEP 660 approach is the only way to get a predictable result.
The alternative proposal mostly argues “you need several ways to do it” and not “you must have symlinks”.
Others might appreciate symlinks in this backend-driven system.
Doesn’t a src dir, or a proxy src dir with a single symlink to the one package you are trying to export, solve most of the accidental ‘expose setup.py’ problems (which are themselves only occasionally a significant problem)?
In an earlier version of this idea we had the backend create an actual unpacked wheel without going so far as to zip it up. This would allow symlinks if we know the rules for copying and uninstalling the symlinks. Pip used to unpack a wheel to a temporary directory before copying it. But I fear we took that step out of pip and it may no longer know how to copy files from the filesystem.
I don’t think the backend has any business creating symlinks in the project tree. IMO, if this is desired it should be done through an extension to the wheel standard, or by making an exception for editable wheels.
Why not ? The backend builds and, doing this, creates various artifacts in the project tree. That’s pretty common behaviour for build systems.
Creating a runtime dependency at some obscure location in the project tree, outside of the install prefix, would be a fairly unusual thing for a build system to do.
Setuptools always copies the whole project into build/ to archive it.
Many modern build systems (meson, ninja?) don’t even support in tree 
builds which setup.py develop mode expects to expose extension modules.
Well, yes, but the built distribution does not depend on the build folder’s contents - the build folder is ephemeral. We’re trying to encode editable installs and what you’re suggesting is to layer another, undefined editable install on top of the one initiated by the editable wheel.
Yes, can we please explicitly leave out symlinking for now?
2 Likes
@dholth I think this referred to use cases for the scheme argument. Have you had a chance to think about it ? We need your input to decide if the scheme argument must be kept or not. At this stage, the trend I gather from the feedback received is to drop it.
On my side I’ll handle the comments you made on the PR.
I suppose we could drop it. Although I think it’s a cool idea I’m not motivated to produce a prototype. It was always optional that the frontend pass the scheme, or not, and optional that the backend use the scheme.