Once again, the discussion on editable mode seems to have started going in circles. We now have at least the following threads on the subject:
- Offer a “dumb” PEP 517 develop hook
- Pip 19.1 and installing in editable mode with pyproject.toml
- Third try on editable installs
I’ve reached the point where I’m convinced that the problem is not about designing hooks to communicate between backends and frontends, but rather the more fundamental question of what “develop mode” or “editable mode” actually is. So I’m going to reboot things yet again (yes, I know, sorry!) to address that question first.
What is “editable mode” right now?
The first, and most key IMO, point is that users are not asking for a new feature here. What they want is for PEP 517 to support the same functionality that they had in the “legacy” mode.
In terms of implementation, that consists of a
.pth file to expose the user’s source directory to
sys.path, plus some extra machinery to expose metadata and to put built C extensions somewhere that is importable.
What is wrong with that approach?
Honestly, it seems like not much. Users seem to like it. However, in the process of debating the implementation, the following issue was pointed out:
That came from here - discourse doesn’t seem to be linking it properly.
It appears from what @pganssle said here that this wasn’t something that was reported by users, but was spotted as part of the discussion. So while he’s describing it as “a pretty serious bug”, it doesn’t appear to be bothering users particularly. It’s also specific to the setuptools
exclude= feature - I don’t know if other backends have anything similar. And conversely the current behaviour means that people can add files to an editable install without needing a reinstall, which seems like something that people probably do.
What would a standardised runtime layout for editable mode look like?
Two basic approaches have been suggested:
.pthfile based approach, essentially like the current one.
- Using symlinks to reference the user’s development tree from
In both cases, we seem to be clear that the
.egg-link mechanism for exposing metadata isn’t needed, and we should just put metadata in
site-packages, the same as we do for non-editable installs.
.pth based approach needs a bit of fleshing out (naming convention,
project-version.pth seems reasonable) and someone needs to clarify how C extensions get exposed (they could be installed to
site-packages as they need a rebuild/reinstall, or they could be exposed via the
.pth file, I guess). I don’t have enough expertise with C extension development to comment, TBH. It also needs to discuss how it handles (or doesn’t handle) the “excluded files” issue above.
The symlink approach needs to describe what happens when a system doesn’t support symlinks (Windows 10 without developer mode enabled, or earlier Windows systems, and Unix on some filesystems). It also needs to discuss changes in semantics, like the fact that you need to reinstall if you add or remove files.
There’s no reason a standard couldn’t allow both mechanisms (or even additional ones). But because of the semantic differences, users would need a way to determine which approach is in use for a project (this could simply be a metadata value, though - we don’t need to over-engineer it).
How does this help?
In my opinion, getting agreement on how we want editable installs to be implemented at runtime is the only way we’ll move forward with implementing editable mode support under PEP 517. What we’re seeing at the moment is a classic issue of implementation-defined behaviour, where the mechanism for editable installs isn’t standardised, and so trying to standardise things on top of it becomes impossible.
Having a standard runtime layout would stop people stressing about who implements the functionality1, as it ensures that no matter whoever implements it will do so in the standard manner.
Why not just finish standardising the hooks?
Because we’re stalled (again). And the discussions keep focusing on implementation, not on interface. But hooks are interface, so we get no conclusions by discussing implementation.
And even if we did standardise the hooks, we’d still have to sort out implementation. And leaving that as a front end implementation choice just ignores the issue - pip may be the only front end right now, but we want to implement a standardised approach. On the other hand, if people want it to be a front end choice, then we can handle that by just having everyone else say here that they are OK with the pip devs writing the standard!2
1 Except for the more basic reasons like separation of concerns and maintainability.
2 But don’t then scream if we choose an implementation with semantics that you don’t like