A thread to discuss TBD: Editable installs by gaborbernat · Pull Request #1977 · python/peps · GitHub
@bernatgabor suppose I wanted to do the equivalent to what setuptools is doing now. Would I return
{
"metadata_for_build_editable": "/home/me/project/src/myproject-1.0.dist-info",
"purelib": {"/home/me/project/src": ""}
}
Suppose I wanted to test the installer’s edge cases.
{
...
"purelib": {
"project/src/a.py" : "mymodule/a.py",
"project/src/b.py" : "othermodule/b.py"
}
}
What happens if I return an entire directory, and also some files inside that directory?
(It looks like part of the document uses purelib_paths and the other part uses bare purelib)
Why not allow all prefixes { "paths": {"purelib": {...}, "scripts": {...}, ...}?
Yes.
It’s a mapping, so I assume it would depend on the value side of this return value.
Can you comment on the PR these instances so I know better where this is?
Scripts are part of the console entry points within the dist-info so no need to declare it again. I could only think of uses for the purelib and platlib, but if you guys can tell me good use cases for others we can add them for sure 
Why not "purelib": {"/home/me/project/src": "project"}?
You’re right that probably would be the correct way 
Using custom importers
For a more robust and more dynamic collaboration between the build backend and the target interpreter, we can take advantage of the import system allowing the registration of custom importers. See PEP-302 for more details and editables as an example of this. The backend can generate a new importer during the editable build (or install it as an additional dependency) and register it at interpreter startup by adding a pth file.
{
"metadata_for_build_editable": "<dir to dist-info>",
"purelib": {
"<project dir>/.editable/_register_importer.pth": "<project dir>/_register_importer.pth".
"<project dir>/.editable/_editable_importer.py": "<project dir>/_editable_importer.py"
}
}
Having users generate source controlled files, especially in the case of editables, seems extremely sub-optimal (I would never). Can you add a way to inject raw content mapped to target paths? Or, Proposal: Adding a persistent cache directory to PEP 517 hooks - #61 by bernatgabor
It’s not user-generated, but instead, build backend generated. That being said the backend could also solve the issue by not generating those files but instead pulling in install dependency containing that code. As mentioned at the start of that segment these examples are there to illustrate how one could solve that problem given this PEP, the backend/frontend is free to use different solutions though.
I’d not use editables as is today, directly. I’ve linked that more as an example of how to register import hooks that manifest the module load dynamically at import time. I should make this clearer.
That PEP only wants to give the option to the frontend to generate cache files in a different folder than the source directory, but would the frontend not request this, the default would still be within the project source directory.
I would expect "purelib": {"/home/me/project/src": ""} to add the entire src/ directory to PYTHONPATH. The src directory contains project and possibly also project2 and module.py. If you specified {"/home/me/project/src": "project"} then I would expect import project to import src/ not src/project.
This is for arguments’ sake and not necessarily what we would normally expect you to send to the hook.
1 Like
What about the scripts, data and include sysconfig paths my package may need to use?
No… Console entrypoints are scripts, but scripts are not console entrypoints. A normal wheel can have custom scripts, and I know packages that take advantage of that mechanism.
Make sense, we can extend it 
to allow all those five.
The scripts have special handling in a wheel. On installing the frontend will replace #!python shebangs with the correct interpreter. We will need to do the same here, I think what makes most sense is to pass the interpreter path in the hook and have the backend do that replacement.
What about keep the frontend doing it. The backend can populate whatever he’d like, but then the frontend generate a fresh script with the right interpreter which calls the script provided by the backend?
A reason PEP 660 abandoned the virtual wheel strategy is because virtual wheels add a lot of complexity to the installer. The hook can send mappings to the installer that do not make sense compared to the layout in the source folder. Think of as many of those as you can and then figure out what the installer is supposed to do about it.
After calling the hook the installer, which has not been required to build wheels in the past, might build an actual (proxy) wheel from the virtual wheel, and install the actual wheel.
If I am a package author, how do I avoid fielding bugs that editable installs of my package only work under certain installers, with certain proxy implementations?
I agree that enscons would be able to provide the necessary information for this version of the hook. In enscons you supply src_dir separately from the normal wheel build and then editables discovers the list of packages to expose. This is totally separate from the list of files going into the normal wheel build. The same could be passed to this hook. It would be up to the packager to make sure they matched.
1 Like
How would you implement that? The scripts can change, so they need to be updated.
Can you clarify a bit a concrete scripts use case, and how would it work with your proposal? Thanks!
Docutils uses the scripts directory. See here. Scripts like rst2html.py get the shebang rewritten.
This part is an implementation detail for the frontend, not? For example, if the frontend wants to use symlinks instead, then building a wheel becomes out of question.
Can you list some of these? I’d imagine the frontend is free to raise an error when the backend returns something that it cannot work with.
This shouldn’t be your problem. You can close this issue and forward them to the backend/frontend issue tracker. You should though document in your contributors’ files what frontends do work, as recommendation for contributors to use.
What frontends are we talking about here, other than pip? If someone says “pip install -e foo” doesn’t work, and it’s because pip chose a particular strategy for installing in editable mode, what installer would the developers of foo recommend instead?
The alternative is to suggest that pip has to offer every installation strategy, and add command line options to let users pick on a per-package basis which one to use. That would need to be recognised on the command line, in requirement files, etc. I don’t think we’re going to be doing that, unless someone develops a library we can just drop in to get the support.
This sounds like a lousy user experience to me. “All I want to do is pip install -e foo, but the authors of foo say it needs the editable install strategy, which means nothing to me, and that it’s pip’s fault that they don’t support that. But pip say that they won’t support that strategy because no-one expect foo needs it. What do I do now?”
conda, poetry, flit, or tox? Could also link to an issue/discussion about why pip decided to not support it. Also would be nice to know exactly what case would not work.
Ultimately it’s up to the frontend to offer less or more. However, I’d imagine the core logic for some methods would be shared the same way we imagined editables to be shared by backends in PEP-660.
What’s wrong with using poetry or tox instead of pip here for the end-user? Why should we make pip the only valid frontend to use that must support everything? I thought the whole point of PEP-517 and co was to increase diversity. Saying all I want to do is pip install -e foo to get some advanced feature, is like someone would complain of a project packaged with flit that all I want to do is python setup.py bdist_wheel.
Well, conda has their own mechanisms, I don’t know if they even have the concept of an editable install. But yes, that’s an option (I know some people have had issues using pip and conda in combination, but I assume they interoperate OK in general).
Does poetry do all of its own installs? I thought it used pip internally. But again, I don’t know it very well so yes, maybe that’s an option.
Flit is a backend, as far as I know you can’t (for example) do “flit install tox”, it only works for projects that are built with flit. So that’s not really a replacement. Unless you mean that flit has its own “flit install” mechanism, but that’s not really the point here - setuptools has setup.py develop but people still expect to use pip rather than use that directly.
And I’ve no idea how I’d use tox to do an editable install of my project into my working virtual environment. Maybe it’s a feature I’m not aware of, but I feel that’s not what you mean when you say “use tox”.
So thanks, those are ideas I hadn’t thought of - but I don’t think they are necessarily what people mean when they think of replacements for pip. And I’m a bit worried about the implications here - PEP 517 is supposed to be an interoperability standard, allowing users to combine tools according to their needs, and we seem to be saying more or less the opposite here, that front ends should implement multiple editable strategies to match different projects’ requirements, or that users will be forced to use specific tools to install specific projects.
Hopefully, yes. Maybe what we need here is for someone to step up and offer a proof of concept implementation for this proposal, a bit like we did with PEP 660, so we’re discussing something a bit more specific.
My feeling is that in reality, neither front ends nor back ends have much appetite for doing anything fancy here. The .pth file approach used by setuptools sucks, but it’s “good enough” for most people, so I think that in a PEP 660 world setuptools would probably simply pass back a wheel containing a .pth file, and in a “virtual wheel” world (is that the right term for your proposal?), pip would just implement a .pth file approach somehow (I’ve not looked at the proposal details yet, so I don’t know whether that means we’d reject any backends that sent individual files, or whether we’d just add directories for each file to the .pth file, but that’s just implementation details). Alternative approaches like symlinks would probably only be used by newer backends (in PEP 660) or frontends (in this proposal). So I’m looking at the two proposals very much from a “how good are they at letting us achieve a baseline of the same functionality as we currently have”, and only secondarily from a point of view of what additional features will they enable (with “being able to do as much as .pth files do, but for other backends” being the key extra feature to start with).