Standardising editable mode installs (runtime layout - not hooks!)

If you mean “the front end doesn’t need to build a wheel” then sure. I wouldn’t actually build a wheel then unpack it. But the backend can build a wheel, and provide it to the frontend via the existing build_wheel hook. Then there’s no change needed in the front end at all (other than to tell the backend to build an editable wheel rather than a normal one).

Thanks. We can produce the next POC with minimal complexity in both ends.

1 Like

I really like the “editable mode wheel” idea, where the package init module takes care of making the rest of the package contents findable.

It not only avoids any dependency on the arbitrary code execution loophole in pth files, it also means there aren’t any side effects on the global sys.path needed, even for applications that import that package.

Edited to add: for single file projects where there aren’t any submodules, and for any other projects with a non-empty __init__.py, reading that code in and running it with exec should mostly do the right thing, as the module metadata variables will have been populated based on the location of the editable wheel install, not the original source code. Two key constraints on that would be:

  • __file__ would need to be updated to refer to the actual source file so relative path lookups work as expected
  • in the package case, any __path__ manipulation would need to occur before the real package init code was executed
6 Likes

What’s needed next to move this forward?

From what I remember we need some volunteers to create a POC for some backends by using editables package to validate the issue. Mainly on setuptools side, but ideally, also flit and poetry. Once we have 1-2 backends proving our approach works, we can write up a PEP and move on to accept it. @pf_moore am I right?

I believe so. Looking back in the thread, it was @pganssle who was keen on having POC implementations, so I’d be interested in his view.

It would also be very good to have the editables package getting some actual use. At the moment, it’s not been battle tested at all.

1 Like

My proof of concept on pip side is here, showing how the frontend/backend interface could look like. I’m happy to work with anyone willing to prototype the backend side to move this forward.

2 Likes

I wrote rudimentary editables support for enscons. It seems to work.

It should be possible to automatically add the editable target to builds but at the moment it is boilerplate that would need to be added to every enscons project’s SConstruct. https://github.com/dholth/enscons/pull/9/files

2 Likes

Figured it out. Today’s enscons release adds a scons editable target by pointing Paul’s editables package at src_root.

1 Like

@dholth, the link to the enscons homepage from PyPI points to a defunct Bitbucket repo. I think it would be worth fixing it. When I looked a enscons a while back this gave me the impression that enscons is a dead project.

1 Like

Thanks for your concern. enscons was caught in the bitbucket Mercurial purge. I’ve automatically migrated everything over to github. Today’s release fixes those links. I also fixed the project URL, thanks.

1 Like

The “Project links: Homepage” link still points to Bitbucket. This is probably due to the fact that the project metadata still contains url="https://bitbucket.org/dholth/enscons".

1 Like

I’m pleased with @sbidoul’s proof of concept and recommend it be standardized. The only thing that’s missing is a pointer to the source distribution in pip list a-la pip 20.0.2 /home/dholth/opt/py38/lib/python3.8/site-packages so that you can tell which installs are editable or not.

1 Like

Indeed. The information is present in the installed PEP 610 direct_url.json with an editable flag and the file:// URL of the project directory, so it’s just a matter of displaying it.

1 Like

Optional Hooks

build_wheel_for_editable


::

build_wheel_for_editable(wheel_directory, scheme=scheme, metadata_directory=metadata_directory, config_settings=None)

Editable distributions are supported by building a wheel that, when installed,
allows that distribution to be imported from its source folder. This was traditionally done by having setup.py install a .pth file into site-packages. It can be done more precisely by producing a proxy module that replaces itself with the target module on import.

scheme: a dictionary of installation categories { 'purelib': '/home/py/.../site-packages', 'platlib': '...'}. This makes it possible to use relative paths to the source code.

Must build a .whl file, and place it in the specified wheel_directory. It
must return the basename (not the full path) of the .whl file it creates,
as a unicode string.

May do an in-place build of the distribution as a side effect so that any extension
modules etc. are ready to be used.

The filename for the “editable” wheel need not contain the same tags as build_wheel but must be tagged as compatible with the system.

If the build frontend has previously called prepare_metadata_for_build_wheel
and depends on the wheel resulting from this call to have metadata
matching this earlier call, then it should provide the path to the created
.dist-info directory as the metadata_directory argument. ** do we care about matching metadata for ‘editable’ wheels?

Backends which do not provide the prepare_metadata_for_build_wheel hook may
either silently ignore the metadata_directory parameter to build_wheel_for_editable,
or else raise an exception when it is set to anything other than None.

An ‘editable’ wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly.

3 Likes

Worth adding here that these wheels are not intended to be cached or redistributed.

You’ve implied it, for those who understand the implications, but may as well explicitly mention these two points for complete clarity.

From the POV of someone who’s implement this interface, this definition seems sufficient to me.

1 Like

I feel it’d be better to build the content directly into the directory, instead of a wheel. The only usage of an editable wheel is to be extracted shortly on the same machine for installation, so the zip format is only adding needless overhead.

2 Likes

By using wheel we guarantee that existing wheel installers (e.g. pip) can handle it even if they’re and older version.

2 Likes

The whole point here is that we don’t want the backend to be installing directly, the backend is responsible for building the distribution but not for installing it, installation is the responsibility of the frontend. And wheel is the only standard format for passing a “thing to be installed” around.

For example, we don’t want to require backends to know how to write executable wrappers for console entry points.

Also, IMO the argument for the “scheme” parameter seems weak. It’s supposed to be only to allow backends to use relative filenames for the install, but how much of a benefit that is, I’m not clear (after all, virtual environments aren’t relocatable, nor are wrappers). And having the scheme parameter is a strong temptation for backends to think it’s OK to install directly, so I’m concerned it will be an “attractive nuisance” in practice.

1 Like

I was not proposing for backends to install the editable directly into the target location (site-packages), but that backends can provide a directory to the frontend, instead of an archive file. This would make both writing and reading the built result easier (plain filesystem access instead of ZipFile), and likely more efficient since the built result is not likely to be moved around (unlike a real wheel, which is likely to be distributed elsewhere). There’s an additional benefit that it’d be easier to implement incremental updates to the built result, which is another area where local development would benefit vastly from.

To be clear, I don’t really hold a strong opinion that a wheel is a bad idea. Bernat’s reasoning (the editable wheel would be usable by older installers) is compelling. It’s also not impossible to use a wheel for incremental builds either; zip entries can be updated in-place, and have last-modified timestamps. Updating RECORD entries might be a pain, but that can probably be abstracted away by some common library used by backends (the reverse of installer) so authors don’t need to worry about it too much. But it’s important to remember that the main advantage of using an archive format is the ease of distribution, and editables don’t really need to be archives because nobody would (should) distribute them.