Standardising editable mode installs (runtime layout - not hooks!)

The uninstall case makes sense :+1:

I just realised the implications of what you’re saying here. (It came to me when I thought “why am I looking at writing code for the frontend to build a wheel?”) Sorry if this was the point you were trying to make, and I was just unable to see it.

If I’m reading this right, we don’t need an “editable hook protocol” at all. If a backend can build a wheel which, when installed, will make a project available as editable, then all we actually need is a way for the frontend to say “build in editable mode”. And at a pinch, the PEP 517 config_settings argument can do that (in practice, we’d want to add some sort of official “editable mode” flag, so that there’s a common convention for all backends).

I’m sure we’ll get some pushback from backend developers about “why do we all have to reinvent editable mode?” But that’s what this topic can do - we can define a standard “editable mode layout”, provide a library to support building it, and backends can just reuse that, knowing they are following the standard approach and won’t be left maintaining a bunch of custom code.

3 Likes

I’ve abandoned about three replies clarifying this, as I’m not really having the best week and am trying to minimise posting things on the internet right now :slight_smile:

It wasn’t the main point, but it’s an implication, for sure. The challenge is enabling those backends that really want to use symlinks, as there’s no universal way to put those into a wheel (but then, it doesn’t have to be universal here anyway).

2 Likes

Don’t build a wheel, just build a folder with a *.dist-info and any other files you need copied to site-packages i.e. purelib or platypuslib

Remember ninja mode module initialization?

def _init_module():
   global a_name
   a_name = ...
   del _init_module

_init_module()

I love how creative some of the editable proposals are, even if it’s not stuff we’re used to doing a lot of like import hooks.

1 Like

If you mean “the front end doesn’t need to build a wheel” then sure. I wouldn’t actually build a wheel then unpack it. But the backend can build a wheel, and provide it to the frontend via the existing build_wheel hook. Then there’s no change needed in the front end at all (other than to tell the backend to build an editable wheel rather than a normal one).

Thanks. We can produce the next POC with minimal complexity in both ends.

1 Like

I really like the “editable mode wheel” idea, where the package init module takes care of making the rest of the package contents findable.

It not only avoids any dependency on the arbitrary code execution loophole in pth files, it also means there aren’t any side effects on the global sys.path needed, even for applications that import that package.

Edited to add: for single file projects where there aren’t any submodules, and for any other projects with a non-empty __init__.py, reading that code in and running it with exec should mostly do the right thing, as the module metadata variables will have been populated based on the location of the editable wheel install, not the original source code. Two key constraints on that would be:

  • __file__ would need to be updated to refer to the actual source file so relative path lookups work as expected
  • in the package case, any __path__ manipulation would need to occur before the real package init code was executed
6 Likes

What’s needed next to move this forward?

From what I remember we need some volunteers to create a POC for some backends by using editables package to validate the issue. Mainly on setuptools side, but ideally, also flit and poetry. Once we have 1-2 backends proving our approach works, we can write up a PEP and move on to accept it. @pf_moore am I right?

I believe so. Looking back in the thread, it was @pganssle who was keen on having POC implementations, so I’d be interested in his view.

It would also be very good to have the editables package getting some actual use. At the moment, it’s not been battle tested at all.

1 Like

My proof of concept on pip side is here, showing how the frontend/backend interface could look like. I’m happy to work with anyone willing to prototype the backend side to move this forward.

2 Likes

I wrote rudimentary editables support for enscons. It seems to work.

It should be possible to automatically add the editable target to builds but at the moment it is boilerplate that would need to be added to every enscons project’s SConstruct. https://github.com/dholth/enscons/pull/9/files

2 Likes

Figured it out. Today’s enscons release adds a scons editable target by pointing Paul’s editables package at src_root.

1 Like

@dholth, the link to the enscons homepage from PyPI points to a defunct Bitbucket repo. I think it would be worth fixing it. When I looked a enscons a while back this gave me the impression that enscons is a dead project.

1 Like

Thanks for your concern. enscons was caught in the bitbucket Mercurial purge. I’ve automatically migrated everything over to github. Today’s release fixes those links. I also fixed the project URL, thanks.

1 Like

The “Project links: Homepage” link still points to Bitbucket. This is probably due to the fact that the project metadata still contains url="https://bitbucket.org/dholth/enscons".

1 Like

I’m pleased with @sbidoul’s proof of concept and recommend it be standardized. The only thing that’s missing is a pointer to the source distribution in pip list a-la pip 20.0.2 /home/dholth/opt/py38/lib/python3.8/site-packages so that you can tell which installs are editable or not.

1 Like

Indeed. The information is present in the installed PEP 610 direct_url.json with an editable flag and the file:// URL of the project directory, so it’s just a matter of displaying it.

1 Like

Optional Hooks

build_wheel_for_editable


::

build_wheel_for_editable(wheel_directory, scheme=scheme, metadata_directory=metadata_directory, config_settings=None)

Editable distributions are supported by building a wheel that, when installed,
allows that distribution to be imported from its source folder. This was traditionally done by having setup.py install a .pth file into site-packages. It can be done more precisely by producing a proxy module that replaces itself with the target module on import.

scheme: a dictionary of installation categories { 'purelib': '/home/py/.../site-packages', 'platlib': '...'}. This makes it possible to use relative paths to the source code.

Must build a .whl file, and place it in the specified wheel_directory. It
must return the basename (not the full path) of the .whl file it creates,
as a unicode string.

May do an in-place build of the distribution as a side effect so that any extension
modules etc. are ready to be used.

The filename for the “editable” wheel need not contain the same tags as build_wheel but must be tagged as compatible with the system.

If the build frontend has previously called prepare_metadata_for_build_wheel
and depends on the wheel resulting from this call to have metadata
matching this earlier call, then it should provide the path to the created
.dist-info directory as the metadata_directory argument. ** do we care about matching metadata for ‘editable’ wheels?

Backends which do not provide the prepare_metadata_for_build_wheel hook may
either silently ignore the metadata_directory parameter to build_wheel_for_editable,
or else raise an exception when it is set to anything other than None.

An ‘editable’ wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly.

3 Likes

Worth adding here that these wheels are not intended to be cached or redistributed.

You’ve implied it, for those who understand the implications, but may as well explicitly mention these two points for complete clarity.

From the POV of someone who’s implement this interface, this definition seems sufficient to me.

1 Like