PEP 660: Editable installs for PEP-517 style build backends

The problem would be adding new scripts, but that could be solved by adding a communication back channel to trigger the frontend to generate them.

Windows 10 from the last few years does support symlinks, you should try to use it helps a lot efficiency/performance in virtualenv for example :smiley:

I think based on discussion above we can conclude that we can’t agree and we’ll have to let the packaging BDFL or the steering council pick a solution. Many people do want editable installations, but both PEP provide this functionality so there’s no this reasoning can’t be used by either option.

We do still have to work out exactly what the other proposal would do before it can be claimed to be equivalent.

Yeah, work in progress, but we’ll get there in a few weeks :+1:

I do quite like my symlinking approach - it feels like the obvious, straightforward way to achieve what I want from an editable install, and it doesn’t have the same unwanted effect as .pth files of making adjacent files importable. I really only implemented the .pth option as a fallback for Windows. But it seems like I’m an outlier on this, as both this PEP and the proposal in the previous round of discussions I remember don’t allow for symlinking the package into site-packages.

I’m expecting to maintain and use the symlinking method in Flit even if some other way of doing editable installs gets standardised. This has limited my enthusiasm for getting involved with the discussions (I doubt I’ll find time to properly catch up on this thread - sorry).

2 Likes

My PEP, see https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat, would allow that.

2 Likes

I don’t think @takluyver would benefit from reading this whole thread. :crying_cat_face:

This PEP exposes a somewhat improved status quo to build systems. You’re either looking for that or not. We also know this PEP is easy to implement.

Will we be able to provide all things from a single PEP?

I think it’s great if an alternative build system can offer an improved editable experience through its own command. You might choose the alternative for that reason.

Here is another possible approach to work with symlinks in PEP 660.

build_wheel_for_editable can generate a symlink structure under $PROJECT_ROOT/.editable/$COMPATIBILITY_TAGS/ with symlinks that point to the actual files that need to be exposed. In the wheel that is communicated to the frontend, it can then add a .pth file that point to $PROJECT_ROOT/.editable/$COMPATIBILITY_TAGS/.

This way the backend can do a mapping from editable to source that is as simple or sophisticated it wants, while keeping simplicity and predictability on what the frontend will do.

2 Likes

I’d be happy to accept a PR to editables that implements something like this. I don’t really want to try to implement it myself, though, as I don’t have sufficient experience with symlinks to know how to robustly confirm if they work or how to fall back if needed, etc. But if no-one else picks up on it I can try.

As a starting point, maybe log it as a feature request for editables and then it won’t get forgotten.

Edit: Or maybe it doesn’t need anything in editables - if the backend is OK building the link farm itself, then just asking editables to supply a .pth file is possible with the existing API.

Also, just as a matter of curiosity, do they need to be symlinks if they are local to the project directory? In the absence of symlink support the backend could fall back to hard links (which are available on NTFS on Windows since forever) - there’s no risk of needing cross-FS support in that situation.

Some old time Unix editors break hard links (create a copy) when you edit a file, like a copy on write filesystem.

I’ve been wondering if vscode also does this.

1 Like

I’m not sure that solves data files and include files support, though granted PEP-660 purposefully only wants to handle purelib/platlib (and even there no pyd/so files, no?),

I’m not sure what the requirements are for that, actually.

How so ? pyd/so is well supported with PEP 660.
Edit: or rather does not need to have explicit support as a rebuild/reinstall is expected when it changes.

While I’m throwing ideas, here is another one regarding the choice of editable style.

In a first approach I’m saying the frontend can communicate the editable style to the backend via config settings. I think we don’t need to standardize that immediately.

But if it is deemed important, we can imagine a mechanism where the backend has a hook to expose the list of supported styles, and then let the front end ask the user to pick one of them and communicate it to the backend when building the wheel.

2 Likes

TBH my personal experience with symlinks in debuggers and IDEs has often been less than optimal (with breakpoint not being hit because they are set in the source file and not the symlinked path, the same symlinked file opened twice in the editor, etc). Not sure if that’d be better or worse with hard links. Or if it’s just me not using the tools correctly :slight_smile:

1 Like

Do we have “the front end controls the linking method” in the rejected ideas section of PEP 660 yet? We think the PEP 660 approach is the only way to get a predictable result.

The alternative proposal mostly argues “you need several ways to do it” and not “you must have symlinks”.

Others might appreciate symlinks in this backend-driven system.

Doesn’t a src dir, or a proxy src dir with a single symlink to the one package you are trying to export, solve most of the accidental ‘expose setup.py’ problems (which are themselves only occasionally a significant problem)?

In an earlier version of this idea we had the backend create an actual unpacked wheel without going so far as to zip it up. This would allow symlinks if we know the rules for copying and uninstalling the symlinks. Pip used to unpack a wheel to a temporary directory before copying it. But I fear we took that step out of pip and it may no longer know how to copy files from the filesystem.

I don’t think the backend has any business creating symlinks in the project tree. IMO, if this is desired it should be done through an extension to the wheel standard, or by making an exception for editable wheels.

Why not ? The backend builds and, doing this, creates various artifacts in the project tree. That’s pretty common behaviour for build systems.

Creating a runtime dependency at some obscure location in the project tree, outside of the install prefix, would be a fairly unusual thing for a build system to do.

Setuptools always copies the whole project into build/ to archive it.

Many modern build systems (meson, ninja?) don’t even support in tree :christmas_tree: builds which setup.py develop mode expects to expose extension modules.

:dark_sunglasses:

Well, yes, but the built distribution does not depend on the build folder’s contents - the build folder is ephemeral. We’re trying to encode editable installs and what you’re suggesting is to layer another, undefined editable install on top of the one initiated by the editable wheel.