Next steps for editable / develop proof of concept

My proof of concept will proceed like so. The editable hook is passed the {metadata} directory already built by its prepare_metadata_for_build_wheel hook (a directory that contains dist-info, not dist-info itself), and the target site-packages directory to figure out relative paths. It will put something in {metadata} that will effect the editable install when copied into site-packages.

I’ll need a good way to generate the something. Like a tool that generates stub modules that replace themselves with code from the installed package when imported.

This isn’t polished or anything (it’s from my work in progress experiments), but it works:

import os
import sys
import importlib.util


name = __name__
location_of_replacement = r"E:\Work\Projects\editable_poc\data"
target = os.path.join(location_of_replacement, name)
init = os.path.join(target, "__init__.py")
spec = importlib.util.spec_from_file_location(name, init)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)

You’d need to add @bernatgabor’s code for excluding submodules. Also, people should be aware that this doesn’t help with the case where someone omits a critical file from package_data - in that case, an editable install and a proper install will still behave differently. I haven’t been able to think of any way to avoid this other than some form of link farm.

2 Likes

Why do we need this?

@pf_moore what are you trying to achieve there, I don’t think I follow. From the looks of it import a single module from a hardcoded path; but why?

I don’t think this is necessarily true. The real question at this point becomes how is that file accessed. Is it via pkg_util or importlib_resources? If so I think the backend can install hooks for that and validate that the requested file is within its build configuration and if not raise, or just silently fix by adding it in the background. On the other hand when users are accessing it by using the __file__ of the module (which is not recommended) then we can just adjust the __file__ to point to a non-existing folder to raise an error again when that happens (for editable installs only).

I think he missed the __path__ update, but once that’s there the importer will resolve submodules from the source directory, and the exec dance is because there’s no other way to get it to import this own module again. So it’s only needed once.

Update Ah no, the whole module is replacing itself. Even better! That’ll come with the right search directory for free

Ah, so the idea here is to install a small shim for the root module of the project into platlib that materialzies the module from the source tree. So when the user enters import foo the importer will load
foo.py from platlib, but that then will replace itself with the module found at he the source tree location.

this stuff looks way better than what i had to do back in gumby elf in https://github.com/RonnyPfannschmidt/gumby_elf/blob/master/src/bootstrap_template.py.txt

That’s right. Sorry I didn’t explain very well, I was still trying to work through the magic incantations myself - I’d confirmed it worked, but sill wasn’t 100% sure why :slight_smile:

Note that this is actually the bare bones of something that could be made into a standalone tool, that built a wheel to expose arbitrary parts of a source tree. Usage could then be something like

$ build_editable . --expose foo --hide foo.bar
$ pip install ./foo_editable-1.0-py3-none-any.whl

Not as convenient as pip install -e ., certainly, but could be implemented now and we could iterate on the design without needing changes to packaging tools until we’re ready.

It could also be written as a pure backend option, triggered by a config_settings value (editable=True?)

1 Like

Yeah, would be essentially a library that backends can use to delegate implementation of editable installs.

That’s where I was going with it.

I’d be glad of some help working out a good API for such a library. My basic idea would be:

build_editable(location, expose=None, hide=None):
    """Generate files that can be added to a wheel to expose packages from a directory.

    By default, every package (directory with __init__.py) in the supplied
    location will be exposed on sys.path by the generated wheel.

    Optional arguments:

    expose: A list of packages to include in the generated wheel
            (overrides the default behaviour).
    hide: A list of sub-packages of exposed packages that will be
          invisible in the generated wheel.

    Returns: a list of (name, content) pairs, specifying files that should
    be added to the generated wheel. Callers are responsible for building a
    valid wheel containing these files.
    """

I’d be interested in any use cases that wouldn’t be covered by this API.

As noted previously, this mechanism doesn’t have any way of managing which data files are visible from the exposed packages, so don’t bother asking for that :slight_smile:

I’d not bother for now working out the API. Instead let’s make the tool. And let’s try to implement various features of setuptools/flit/poetry. Once we handle a core set of it we can see where the library lands and make up the API then. For now we know way to little to handle things. Want to create a repository for this library or shall I?

I can create a repo.

What would that mean in regards to PEP 420? Additionally, I don’t know if package data is in scope for this topic but, I believe it’s possible to package directories that contain strictly no .py Python files (pure data package), would or should such directories be exposed in editable mode?

First version of the library code is at https://github.com/pfmoore/editables.

It would mean that case hasn’t been considered yet.

This only covers Python code. I’ve already mentioned, but it bears repeating, that the approach being used here (importlib and path hooks) “probably” doesn’t support package data transparently1. The approach used by setuptools at the moment (.pth files) is also limited - it has no way of only including “declared” package data, or of handling directories with no Python code (at least to my knowledge).

This is very early proof of concept code. If you have a genuine use case for this (and please remember it’s targeted at replacing the existing “editable mode”, so if current editables can’t do it, we’d be open to supporting it but not doing so is not a showstopper) please feel free to test it out and suggest improvements.

1 By “probably” I mean “I haven’t tried it, but I’m pretty sure there will be issues”. I also mean “I’m not sure those issues will be solveable with this approach”.

Indeed, I missed that sentence, my bad.

OK.

It depends on how they’re exposed.

Using importlib.resources should be fine, and anything relative to __file__, but if you’re looking them up relative to anything from the sys module (path, prefix, etc) then they’ll be absent. Similarly if they’re generated on build, unless the backend generates them in-tree here, which my projects would be fine with (same for extensions).

as far as i can tell this is only concerned with editable wheels and has no metadata handling yet
would it be advisable to put in general wheel building , in memory wheel building and metadata building as well (i have all that in gumby elf and wouldnt mind to pass it over into a general wheel-maker library

Building a wheel can be layered on top of this. As backend already know how to build a wheel I’d rather not try to duplicate that in the editable support library.

1 Like

@bernatgabor relative paths are useful. Sooner or later someone’s going to run the code in a chroot or a container and the location of / will change.

Will there be a .pth file? The beauty of the current proposal is that the backend decides. No additional standardization necessary.

Sure, if the backend wants. This is just one mechanism a backend could use. The beauty of the “backend supplies a wheel that when installed does the right thing” proposal is that the backend has full control. But by offering a library for “how to implement editable mode” all we’re saying is that backends that don’t want to bother with the details, can just use the standard library.