I would create a standalone package setuptools_pep660 that imports the existing PEP 517 driver plus our new hook. The package would provide a setuptools command to generate the “editable” wheel, with the appropriate entry point so it gets picked up by setuptools.
The new command depends on the existing dist_info command, strategies taken from the existing develop command, and any other necessary metadata from the Distribution class to write the editable wheel’s contents to a temporary directory. A tiny wheel generator like sdl2_lib/wscript at master · dholth/sdl2_lib · GitHub does the actual zipping & RECORD writing. The editable wheel is always tagged ed.py3-none-any.
If you want to use the prototype, set setuptools_pep660 as the build backend instead of using the standard one.
Coincidentally, I started looking at this yesterday. The following code gets the data needed from setuptools, all that really remains is to build the wheel from it.
from tempfile import TemporaryDirectory
from pathlib import Path
import sys
import os
from setuptools.build_meta import _BuildMetaBackend, no_install_setup_requires
class DevelopBackend(_BuildMetaBackend):
def get_develop_files(self):
with TemporaryDirectory() as tmp:
site_packages = Path(tmp) / "lib" / "site-packages"
site_packages.mkdir(parents=True)
# Hack to convince the "are .pth files processed" check that
# things are OK.
os.environ["PYTHONPATH"] = str(site_packages)
sys.argv = [
sys.argv[0],
"develop",
'--prefix',
tmp
]
with no_install_setup_requires():
self.run_setup()
paths = []
files = []
for f in site_packages.iterdir():
content = f.read_text(encoding="utf-8")
if f.suffix == ".pth":
paths.extend(content.splitlines())
else:
files.append((f.name, content))
return paths, files
def wheel_content():
return DevelopBackend().get_develop_files()
if __name__ == "__main__":
paths, files = wheel_content()
print(paths, files)
I chose to extract the paths from the .pth file because setuptools re-uses the name easy-install.pth.
And an in-place build. Probably easy enough given what the existing develop command does.
Did you notice that setup.py develop is not in-place when 2to3 is used? Thorough of them to handle that case. 2to3 was a compiler to transform Python 2 code into Python 3 code, important during the early part of the Python 3 transition.
Does anyone still maintain a handy database of all the arguments passed to all the setup.py setup() calls of the top n pypi packages? It would be useful to see what gets passed to setup(package_dir=..., packages=..., py_modules=...) in a large sampling of distributions.
IMO this compares very favorably to ‘setup.py develop’ (it is shorter). This version adds no features, just pointing a .pth where setuptools would, but it’s simpler than ‘setup.py develop’ since it only builds a wheel rather than perform the install/uninstall itself.
@pf_moore what can you tell us about the strategy editables==0.2 uses, compared to the first version or individual .py stubs that use the import system to replace themselves with the target source code?
I’m not quite sure what you mean. editables offers two approaches (which can be used individually or together). You can expose one or more directories on sys.path (which has no dependencies and just writes a .pth file) or you can map import names to individual files (foo maps to .../src/foo.py, or bar maps to .../src/bar/) which is done using an import hook and adds a runtime dependency on editables which to the resulting wheel.
Which strategy is used is up to the backend calling editables. Obviously, if you’re just using a .pth file you’d be within your rights to just do that manually without using editables, it’s not as if it’s hard (unless core Python decides to deprecate .pth files for some other mechanism at some point…).
I thought the original version wrote self-contained .py files that loaded the target source code, and then assigned the loaded code to sys.modules['x']. So that when you import site-packages/x.py it replaces itself with $TARGET/src/x.py
The 0.2 version uses a tiny import hook and .pth files.
I dropped the “self-replacing .py stubs” approach because it was harder to reason about, was harder to debug, and didn’t offer any real benefits over the import hook mechanism. But the mechanism used is basically an implementation detail anyway, what should matter to consumers is whether editables provides the functionality they want. And yes, the docs are lacking at the moment, but the relevant functionality is
project.map(name, target) - make import name map to importing a given source file (target).
project.add_to_path(dirname) - add a directory (dirname) to sys.path.
If a backend wants to implement its own strategy, it basically doesn’t need editables, but conversely it takes on responsibility for supporting that strategy itself.
Correct. The 0.1 version of map used self-reloading files, the 0.2 version uses the import hook. As I say, the implementation shouldn’t matter here, what should matter is that map makes import xxx work as if the filename you gave was loaded. A symlink would be a third viable import strategy here, except that wheels don’t support symlinks
Edit: Are you aware of a situation where the 0.2 implementation doesn’t work correctly, but the 0.1 implementation does? I think the import hook is strictly more capable, but maybe there are edge cases where that’s not true. If there are, I reserve the right to document them out of existence by saying that case isn’t supported, though I want editables to be a way of implementing editable functionality, not to be a box of strategies that the backend has to choose between. If you want different trade-offs, you are still free to write your own strategy (or use a different library).
I think the library works, but I had implemented v0.1 for enscons and have to adapt it to 0.2. Will be appending the requirements to METADATA instead of asking the build system to regenerate the file with all deps + editables
Other than the API changes that happened between 0.1 and 0.2 (inevitable, as this is still very much an in-development library) what did you have to adapt? If specifically the change in strategy is a problem, please raise a bug (with self-contained reproducer and/or an explanation of the issue) on the editables tracker. That would be a better place than here.
I may fix the issue by changing the strategy yet again, of course
I had to rearrange it a little bit, and the editable wheel gets its own build directory so that it could have different dependencies. The older version used editables.build_editable(src_root) while the newer version also has to fetch the package name. enscons doesn’t know anything about what you’re packaging but makes it easy to put your files in the appropriate wheel paths. It has src_root metadata that is only used for editable and not for generating the package. It could add a list of top-level modules for the same reason.
I happen to like the simple .pth-to-src-dir strategy but it would be neat to see the slightly stricter exported-top-level-packages-only strategy in action.
I want