Discuss PEP 662: Editable installs via virtual wheels

The vast majority of use cases I’m interested in look something like: 'src/foo represents the entire foo package, so symlink that directory as .../site-packages/foo'.

Namespace packages allow different projects to install in the same package, so foo.bar could come from one project and foo.baz from another. This is OK if foo itself is the namespace - you just make .../site-packages/foo and create symlinks in there. But if foo is a regular package containing your app and foo.plugins is a namespace package, then you have to symlink all the other files under src/foo, rather than symlinking the whole directory. It’s not impossible, just an extra complication necessitated by a feature I’ve never liked.

2 Likes

No, to be clear it is always the frontend’s responsibility to choose between a wider install and a narrower install. The backend just says, “Here’s where all the files are supposed to be mapped from and to” and my SHOULD language was that they shouldn’t be trying to control how it gets installed. If you want the behavior where all the directories are exposed or some variation on that, that’s up to the front-end (which could be configuration options in a single front-end or many different front-ends specializing in different kind of editable installs).

To be concrete, say we have a backend whose internal rule is "install everything in /a/b/myproj/src/myproj as myproj/. If src/myproj contains __init__.py and foo.py, it will pass the following to the front-end (simplified, not in the structure of Bernat’s PEP):

{
    "myproj/__init__.py": "/a/b/myproj/src/myproj/__init__.py",
    "myproj/foo.py": "/a/b/myproj/src/myproj/foo.py"
}

The front-end is free to use a mechanism that detects that adding /a/b/myproj/src/ to sys.path is sufficient to expose myproj and do that, even though there may be other stuff in /a/b/myproj/src/. It’s also free to do something like generate a .pth file that creates an import hook that detects any import myproj imports and tries to import the relevant symbols from /a/b/myproj/src/myproj/ (whether or not it’s in the mapping). Or of symlinking the directory $PYTHON_ROOT/site-packages/myproj to /a/b/myproj/src/. All valid modes of installation that different front-ends (or the same front-end with different configuration options) can choose from, with trade-offs evaluated by the end user.

It also has the option to symlink all the files and require re-installation to add new files, or to do something like generate a .pth file to install an import hook that bakes in the mapping, detects if a file would be in the project root and trigger a custom error if someone has added myproj/src/myproj/bar.py, like:

ImportError: The module myproj.bar was not installed, but it is present in
the editable install directory: you may need to re-install the package to
expose new files, and/or update your project manifest.

Lots of options here, but at the core the responsibilities are very clear — the front end decides how strictly the installed layout matches the mapping.

And as I mentioned in an earlier reply, if we feel that a mapping of just files is not sufficient for front-ends to use to build robust heuristics that allow the behavior that users want in common use cases, we can take on a bit more complexity and design an optional, limited “here’s where I look for new files to go in the package” mapping. In which case the back-end would expose something like this:

{
    "purelib": {
        "myproj/__init__.py": "/a/b/myproj/src/myproj/__init__.py",
        "myproj/foo.py": "/a/b/myproj/src/myproj/foo.py"
    },
    "content_expansion_hint": {
        "myproj": "myproj.*"
    }

(or even something slightly more complicated that says, "Everything under this directory except things that start with tests/*). In that case it would still always be the front-end’s responsbility to decide what to expose (as long as it always exposes at least what’s specified in the mapping). The back-end isn’t required to provide hints, but it may, and the front-end isn’t required to use the hints, but if it does use them it can assume that they match the mapping.

Edit @layday’s suggestion of structuring the mapping may also work as a simpler version of “content expansion hints”. I’m not clear exactly how such a thing would work, but it would basically just be encoding information about the semantics that went into the mapping into the structure of the mapping, which I think is not a bad idea if done right.

Well, I doubt “setup.py stub” will be a supported mode forever, but my whole opposition to PEP 660 was that you don’t need implementations to support automatically detecting new or renamed sub-packages or files or whatever particular edge case you care about if you give this responsibility to the front-end, because you just need to have at least one front-end that supports your use case.

It seems that a lot of people seem to be arguing as if the question is whether editable installs should be able to detect new files or not, whereas I think it’s clear that end users disagree on this, and so the best thing to do is to aim for a situation where each individual (not each project) is deciding what heuristics and trade-offs to make in different situations. In my world, even if pip and every other major front-end decided to go with strict installations, you could write your own custom single-purpose tool that just does the looser install you’d like, and it should work for every project out there (probably most of the tools you’d need to build such a thing will be off the shelf, too, like pypa/build for creating isolated build environments).

OK. Thanks for clarifying. With my pip developer hat on, I await with interest the availability of the various support libraries so pip can decide which one (or more than one) to use to implement this logic. And of course I’m fine if only one reference implementation of the functionality is ever developed, we’ll just use that.

I don’t imagine pip ever implementing the logic itself - it’s clearly (to me) something that needs to be in a library, so that we don’t get trapped in the whole “implementation defined functionality” situation again (if someone did offer a PR to pip to implement this in-place I’d ask that it be split out as a reusable library).

That’s why I developed the editables library, to provide similar “off the shelf” functionality for backends in a PEP 660 world. But I won’t be doing the same for the “virtual wheel” proposal, because I don’t feel confident I know how to implement the logic. I’ll leave that to someone else who does feel comfortable taking on that task.

1 Like

A project will work under the editable strategy its developer happens to use. The other strategy or strategies will be untested by the author and may not work. We don’t know whether a ‘strict’ or a ‘loose’ editable install would wind up being more popular, but we know the strategy setup.py develop uses. Under this proposal the more popular editable install strategy would be more likely to work given a randomly chosen project.

Suppose I’m distributing a package on github only and I like to use a ‘loose’ editable install. Other people are doing ‘strict’ editable installs off the main branch of my repository. I work on my package, then make sure it also works under a ‘strict’ editable install every time I commit to the main branch.

If you want a strict install, and you don’t mind typing ‘pip install’ every time you add or remove files, that is an ordinary install. You have to test the real wheel before you can expect your package to work off pypi.

Suppose we figure out a set of globs etc. for a precise ‘loose’ install. What we should then do to make this proposal work is remove the build_wheel hook and use the virtual wheel structure only. So pip can guarantee the rules used to make the ‘editable’ install match the distributed wheel.

Haven’t read all the rest of the replies since this one, because I’d like to ask you to stop expecting developers on Windows to be able to enable symlinks. That is simply not a viable assumption to make at the level we’re working at.

Happy to discuss another time all the reasons why, but I’m making this post a simple and direct request to drop this idea completely. Nothing we design can rely on symlinks being usable on Windows.

3 Likes

If you’re going to make the statement I’d like you to also publish all those reasons when you have time for it. Thanks! PS. I wasn’t saying to make symlinks the only way, but a possible way with some benefits when available.

Immediate answer - corporate PC with admin rights and developer mode locked down so the user can’t make the necessary change.

4 Likes

That sounds to me more like shouldn’t be the only way to do it, not that it shouldn’t be allowed at all. So I’m waiting for some other reasons here that I’m sure Steve has.

This is correct, but in all the other times you’ve referred to it, you’ve never included this nuance.

Feel free to take advantage of symlinks if they’re enabled. Do not require them to be enabled or assume that they will be.

(Another answer is that the vast majority of student laptops these days are as locked down as corporate PCs, so anything that doesn’t work without symlinks - even if it barely works and “we’ll display a message telling them to enable them” - is going to exclude the next generation of devs.)

1 Like

I’ve spent a little time creating a library for the frontend as suggested above, which you can find here. Hopefully, this will help inform our choice of editable installation and will form the basis of a more robust implementation should this PEP or an equivalent be adopted.

1 Like

Thanks for your effort!

I have a concern about the example:

import sysconfig

import frontend_editables

path_mapping = ...  # Will have been returned by the backend.
installed_files = frontend_editables.install(
    sysconfig.get_path("purelib"),
    path_mapping,
    frontend_editables.EditableStrategy.lax,
)
# Then append the ``installed_files`` to the distribution's ``RECORD``,
# optionally by passing ``append_to_record=<path to RECORD>`` to ``install``.

The sysconfig.get_path("purelib") bit is misleading as the frontend will most likely not be running with the Python of the intended environment so you’ll actually need a subprocess call.

Yeah, that’s just a placeholder. In actual practice the frontend (pip) will pass the output path to frontend_editables.

I’ve started doing the pip+setuptools POC (that could use @layday library to do the path link) but it’s not yet ready:

However, I plan to use that as POC for the PEP.

While Bernát works on setuptools and pip, I have added support for frontend editables in flit at flit@feat-frontend-editables, combining ideas that have been thrown around in this thread and in a way that strays (rather significantly) from the PEP. Specifically:

  1. build_editable has been renamed build_wheel_for_editable.
  2. build_wheel_for_editable builds an installable wheel.
    The return value is the filename of the wheel. This wheel differs from
    a regular wheel in two ways:
    • It must not contain files and folders which it wishes to register
      as being editable in editable.json.

    • In its metadata directory, it must contain one additional JSON file,
      editable.json, with the following schema:

      {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "properties": {
          "paths": {
            "type": "object",
            "additionalProperties": {"type": "string"}
          }
        },
        "required": ["paths"]
      }
      

      paths maps the paths of files the backend has omitted from the wheel
      to their absolute path on disk.

  3. build_wheel_for_editable does not make reference to scheme paths. These are the
    responsibility of the frontend performing the installation.
  4. build_wheel_for_editable takes two arguments: wheel_directory and config_settings.
    These have the same meaning as they do in build_wheel.
  5. get_requires_for_build_editable is not implemented.
    The build requirements are the same as for build_wheel and frontends
    must call get_requires_for_build_wheel prior to calling build_wheel_for_editable.

This all gives us, in what you can experiment with today:

from pathlib import Path
import json
import sys
import sysconfig

import flit_core.buildapi
import frontend_editables
from installer import install as install_wheel
from installer.destinations import SchemeDictionaryDestination
from installer.sources import WheelFile


## BUILD AN EDITABLE WHEEL ##

editable_directory = Path() / "editable"
editable_directory.mkdir(exist_ok=True)
editable_wheel = flit_core.buildapi.build_wheel_for_editable(str(editable_directory))


## INSTALL THE WHEEL ##

destination = SchemeDictionaryDestination(
    sysconfig.get_paths(),
    interpreter=sys.executable,
    script_kind="posix",
)
with WheelFile.open(editable_directory / editable_wheel) as wheel:
    dist_info_dir = wheel.dist_info_dir
    install_wheel(source=wheel, destination=destination, additional_metadata={})


## INSTALL THE EDITABLE FILES AND UPDATE THE RECORD ##

# installer does not return the installation location, let's assume it's
# "purelib" for now.
root = Path(sysconfig.get_path("purelib"))
frontend_editables.install(
    root,
    json.loads((root / dist_info_dir / "editable.json").read_bytes()),
    frontend_editables.EditableStrategy.lax,
    append_to_record=root / dist_info_dir / "RECORD",
)

I’ve not addressed PEP 610, but assume that the frontend has to create a direct_url.json with a file URL and "dir_info": {"editable": true} at the end of all this.

I’d strongly disagree with this approach (and my implementation also differs in this sense). The backend is not generating a JSON. It returns the content. The frontend might decide to use a JSON file to communicate with the backend, but there’s no reason to mandate that file, it’s fine to use any type of inter-process communication technique. This is in line with how get_requires_for_x works. Similarly, there’s no need for build_wheel_for_editable to take the wheel directory argument.

I again strongly disagree. To achieve an editable mode the backend might take additional dependencies, and as such we should not conflate wheel dependencies with editable dependencies. The backend can alias those to the same if it wishes, but should be allowed to differ.

My implementanion rests on producing a PEP 660-like wheel for simplicity of installation and interoperability with existing tools. This is mainly to assess the viability of frontend_editables - not that of the PEP as a whole. The backend could return something like a two-tuple of (wheel_filename, extra_paths) rather than create an editable.json in the wheel; I don’t think it matters too much but it means that the editable installation has to occur in the same execution cycle. Any kind of IPC with the backend other than to request an editable wheel has been omitted on purpose. What this comes down to is our previous disagreement over whether the backend should be able to influence the editable installation. The PEP can go in a different direction - please don’t take my implementation to be normative.

One complication that became apparent when I tried to interface with @pf_moore’s editables is that the frontend editable library might require additional dependencies to be installed in the target environment, at which point it stops being a simple post-installer kind of thing that the frontend can call and forget. Of course, the editable library could bundle its dependencies and e.g. make a copy on install, but it’s a limitation to consider.

editables simply needs to be declared as a dependency of the editable wheel in a PEP 660 world, and the front end’s normal dependency resolution mechanisms will handle it. But with the virtual wheel approach, it seems like any additional dependencies in support of the mechanism being used will have to be installed by a separate dedicated mechanism (as the installation isn’t being done via the standard “install a wheel” route).

FWIW, I have no plans to bundle dependencies in editables, or make the runtime parts available as anything other than a standard wheel.

To clarify, I meant that my library could bundle editables, not the other way around. Sorry for the confusion.

1 Like