Consider this example project GitHub - gaborbernat/pi-approx How would an editable install work with PEP-660? There’s no source file to link, just a binary module. Would you expose the project root? Can editables handle this today? Can we ensure that we only expose the pyd file and nothing else?
Yes, and yes via the add_to_path method. This is how namespaced packages are supported btw
Unsure, but why not put it in a package?
If you add the root folder to the path wouldn’t that make setup.py import-able?
Because then it would not be an editable install. Generally editable installs link the artefacts inline, no? So that if I recompile the pyd|so I can use it without needing to reinstall the editable wheel?
Note this use case could be solved via a simple symlink between the source trees pyd file and the target interpreters platlib folder. Not sure how this work though in a PEP-660 world.
I meant put it in src/foo and expose src or put it directly in src
There’s no src layout here though 
So you have to expose the root folder, which exposes the setup.py and otehr python files in there too. Unless you install some import hook to defend against those, but I’m not sure editables can do that today.
Yes my point is why wouldn’t one just do that then
edit: Basically, I think it’s valid to encourage real packages rather than dumping code in the same directory as config, licenses, etc.
This would imply that editable installs do not offer support for non-src layout projects. This is a hard stance to take because many people don’t like the src layout. IMHO we should support both the src layout and non-src layout.
editables supports non-src layouts w/o a pth file, I’m saying your example of a non-package single-file module requires a src layout
Only with what PEP-660 does today. In an alternative world, you could use a simple symlink between the pyd|so file and the platlib folder. I’d be interested to see from the PEP authors how PEP-660 can support this use case, and not require changes on the source tree layout side.
There are some contents of a normal wheel that this PEP seems to not address. I think we should clarify what should happen with each of these categories:
- header/include files (setuptools allows to add this via
headerskeyword) - scripts - see docutils/setup.py at master · docutils-mirror/docutils · GitHub
- data files - see docutils/setup.py at master · docutils-mirror/docutils · GitHub
How does the editables project handle these today?
Another topic is interpreter compatibility. Can the frontend use the Python 3.8 interpreters to build an editable wheel for python 3.6? How does the build backend know what python version it needs to target?
In a tox/nox world if I want to install the same project in 3 different python interpreters do I need to call build editable three times? What if I target three interpreters that have the same version? When can I share editable wheels across different interpreters?
From a performance point of view can we avoid needing to call the build backend in multiple interpreters multiple times? How can the backend communicate if an editable wheel is supporting multiple python versions?
Setuptools requires an .egg-info and/or .pth per Python environment, so you do have to call setup.py develop per Python. I’m not sure how it handles the other things.
editables would normally be expected to be called per target Python environment. I suppose headers, scripts etc. would need to be copied, or you’d not worry about them, or the package would have a different search path while in development. The hook would always be expected to run under the targeted interpreter.
If the ephemeral wheel was not interpreter specific, you were trying to re-use the wheel and it used absolute paths then I suppose you could extract it into multiple environments. But, if I were you I would extract the editable wheel into a new, empty directory and add that directory to multiple interpreter search paths.
Would make those files not editable, wouldn’t make more sense to symlink them where the platform supports? Also the front-end should likely display some warning on other platforms, if we cannot come up with a way to make those editable too. Scripts requires special thought with the interpreter shebang rewrite, I’d argue you’d likely want some proxy scripts to account for those.
I think we are finding the edge of the implementation.
PEP 660 basically expects to add your source code to PYTHONPATH plus the .dist-info metadata. This matches the long-held expectations of setup.py develop users without having the build backend directly modify site-packages. We’ve innovated with proxy modules instead of always using a .pth file, so we can expose just your module even if you don’t put it in a src/ directory. Of course if you’re typing python in the checkout, you might still find that the other parts of your checkout are importable.
Once you’ve exposed your module, you can continue to add files there and they will be picked up. Though more complicated rules are possible with proxy modules.
A draft of the alternative proposal suggested even suggested a proxy directory tree. An ambitious build backend could create e.g. an .editable/ tree of symlinks or proxy modules suitable to be referenced by a .pth file in site-packages.
And then that’s pretty much the end of the story. The user doesn’t need to think that their package is 100% installed. It doesn’t pretend to be an almost-faithful normal installation; it’s just editable.
If handling other installation locations requires you to do something other than to simply install the wheel build_wheel_for_editable would produce, then I should think that it doesn’t belong in PEP 660, or if the wheel spec doesn’t support it, then it doesn’t belong in PEP 660 either. I would be in favor of clearly stating that, under this scheme, users should expect that changes will be reflected only if they are made to {pure,plat}lib files which existed during installation. If you need to update your metadata, or if you need to update your data files, or if your project structure changes, then you need to do a reinstall. To me, that sounds like an acceptable trade-off - it strikes a balance between convenience, and simplicity of design and implementation.
These are all things which are also true of the competing proposal, with the exception that it opens the door to symlinking data files; the matadata continues to be static and making substantial changes means (should mean) reinstalling.
@layday I think what you mention is already addressed in the Terminology and goals section of PEP 660.
I’m proposing a few updates to PEP 660 (Updates to PEP 660 by sbidoul · Pull Request #1978 · python/peps · GitHub):
- mention symlinks (spoiler: it is possible to use them with PEP 660)
- mention the competing proposal in the rejected ideas section
To clarify, the complexity I’m referring to is not the implementation complexity, but rather the rollout complexity.
I think we can agree that we don’t have much practical experience about alternative editable installs mechanisms other than what setuptools does.
With PEP 660, we can gain experience by experimenting at the backend level. There is no latitude for frontends, so interop with different frontends will be pretty much guaranteed, and PEP 660 need not be specific about any details.
With your proposal, many variations will require spec clarification to ensure interoperability and help project work reliably with different frontends, so the spec that is already longer than PEP 660 will continue growing as we identify use cases and edge cases. Clarifying the spec will be necessary because if we don’t, backend and project authors will never be sure about what works and what not with different frontends. This process would surely take a long time during which I expect there will be quite a lot of confusion.
So let me repeat that I appreciate your proposal but I think the world is not ready for it, out of lack of practical experience.
A .pth pointing to the project root seems the easiest.
Since you can remove setup.py, nothing else will be importable.
Other approaches are possible, including an import hook or executable .pth that generates a symlink on first use.
An implementation should not end before it’s able to solve all valid use cases.
I guess this is about setting goals. Do we want something slightly better then what the user today can do with pth files, or do we want to solve editable installs properly. If the former, then yeah PEP-660 is good enough. For the latter though seems it’s purposefully deciding to not solve some use cases. You can build your logic on top of what wheels do today, but that implies you’re bound to the limitation of wheels. This is the reason this PEP decides to not do so, to allow further freedom to the frontend on solving these edges you’re referring to.
I do disagree with this. At least with the part where you draw the line in the sand on this. I think we should allow ambitious backends+frontends to be almost faithful because that’s what a normal user (one that hasn’t been indoctrinated with today’s state of affairs) would expect. Sure, we should not mandate this for all frontend/backends but we should at least allow it.
I agree with you, and that’s why I think PEP-660 is a subset of what this PEP offers.
I think headers and data files should be first-class citizens with Python files, and handled without reinstallation, if possible to offer so by the platform - e…g symlinks). These are part of the wheel spec and IMHO we should support it. The same is true for scripts, note at the moment the PEP mandates a copy of these at beast, which IMHO makes those files not editable. A script files is a python file, so in essence you’re proposing to not support some python files…
Yeah, it does solve headers and data files. Will amend the proposal to correctly handle scripts too.
I’d say it’s possible, but triggering a symlink at import time seems like a hack to me.
@sbidoul I’d like the proposal to address headers, data files and scripts; which seems to not pronounce on each of these explicitly.
A more restrictive spec will always be shorter. But not being able to solve some problems makes the spec less competitive in my mind.
No, you cannot. Defining the c-extensions requires setup.py. And even if you could move it to setup.cfg would not help with having helper scripts in that folder that you don’t want to expose.
@sbidoul another thing I’d like to see addressed in PEP-660 is how would the user control the type of editable install the backend produces (pth file, import hooks, symlinks, etc). Would these be passed in within config_settings? Are these settings backend-specific or you plan to standardize them?
Also would expect some wording on each of these topics I’ve raised last night: