Offer a "dumb" PEP 517 develop hook

dholth · April 18, 2020, 4:37pm

Five years ago we got PEP 517, but it lacks a develop() or editable_install() hook. Instead, users of alternative build systems have to include a “setup.py develop” command to do exactly the same thing.

Add a develop() hook to PEP 517 build backends that would do the same thing. When called, it’s the build system’s responsibility to add that package to PYTHONPATH in the best way it knows how. If we can imagine a better, more complicated way to do this than having the backend produce its own symlinks or .egg-link but cannot produce it then we should allow the simplest thing that could possibly work.

pf_moore · April 18, 2020, 4:42pm

I don’t have a problem with this, but the devil is almost certainly in the details. The only way we’ll know for sure is if someone submits a PEP, which no-one has done yet.

One obvious question that I can think of is what if the front-end isn’t trying to install to sys.path? There are --user, --target, --prefix and --root options in pip, for all of which it would be wrong for the backend to install to "somewhere on sys.path" without more co-ordination with the frontend. But yes, a PEP could address this, and any other questions that came up.

dholth · April 18, 2020, 5:01pm

The idea is that it is no more specified than pip executing “setup.py develop”.

It’s galling to keep setup.py around just to emulate this one command. Give an easier interface for a build system to do exactly the same thing.

pf_moore · April 18, 2020, 6:43pm

The actual command pip uses is

setup.py develop [--no-user-cfg] --no-deps [--prefix dir] [--home dir] [--user --prefix=]

Plus any global and install options the user might have specified.

And no, I don’t know how that --prefix= that we add as part of the --user processing works, but there’s no following argument

So at a minimum we need to decide how we’d include all of that flexibility in the hook, or we’d need to deprecate some or all of pip’s options for editable mode. Or a bit of both.

dholth · April 18, 2020, 7:55pm

I added output.write(' '.join((sys.executable, sys.prefix, str(sys.argv) + "\n"))) to my setup.py shim and ran pip install -e . ; pip install -e . --user ; pip install -e . --prefix=/usr/local/ and got this.

~/opt/anaconda3/bin/python ~/opt/anaconda3
    ['-c', 'develop', '--no-deps'] 

~/opt/anaconda3/bin/python ~/opt/anaconda3
    ['-c', 'develop', '--no-deps', '--user', '--prefix=']

~/opt/anaconda3/bin/python ~/opt/anaconda3
    ['-c', 'develop', '--no-deps', '--prefix=/usr/local/']

~/opt/pypy3/bin/python ~/opt/pypy3
    ['~/prog/enscons/setup.py', 'develop', '--no-deps']

~/opt/pypy3/bin/python ~/opt/pypy3
    ['~/prog/enscons/setup.py', 'develop', '--no-deps', '--prefix', '/usr/local/']

~/opt/pypy3.6-v7.3.1-osx64/bin/pypy3 ~/opt/pypy3.6-v7.3.1-osx6
    ['~/prog/enscons/setup.py', 'develop', '--no-deps', '--user', '--prefix=']

(I replaced my home directory with ~/)

The pypy virtualenv doesn’t support --user installs. I don’t know how to get --home passed in there. For me --user is only what VScode does to unsuccessfully add pytest or rope to my environment so it is not useful.

There’s pip’s implementation https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/install/editable_legacy.py

enscons does develop by asking distutils where the purelib directory is and putting a file there.

>>> enscons.paths.get_install_paths('enscons')                              
{'purelib': '~/opt/py3env/lib/python3.7/site-packages', ... }

Since this hook would exist to provisionally get something working my preference would be to underengineer it, and have develop() take no arguments.

--no-deps is always implied since an installer is involved.

Second choice. It could take the directory in PYTHONPATH that I’m supposed to touch to link my package in.

Third choice. Pass the existing arguments prefix : str, user : bool, minus “home” if we don’t even know what it does. This would be good for setuptools compatibility. enscons would raise an Unsupported exception if those were passed.

The hook could be called link instead of develop; personally all I’m looking for is that python -m mymodule works afterwards when installing for development in a virtualenv.

If we do this then the relatively tiny number of non-setuptools packages that don’t work at all with pip install -e . start to work, in the hopefully most-common case.

In distutils Distribution:

        # Ignore install directory options if we have a venv
        if sys.prefix != sys.base_prefix:
            ignore_options = [
                'install-base', 'install-platbase', 'install-lib',
                'install-platlib', 'install-purelib', 'install-headers',
                'install-scripts', 'install-data', 'prefix', 'exec-prefix',
                'home', 'user', 'root']

Distutils paths for beaglevote. Presumably these change when you pass user, home, or root.

--root may be so that you can build and install your package to an independent root, and then package those files with RPM.

>>> enscons.paths.get_install_paths('beaglevote'))
{'data': '/Users/daniel/opt/py3env',
 'headers': '/Users/daniel/opt/py3env/include/python3.7m/beaglevote',
 'platlib': '/Users/daniel/opt/py3env/lib/python3.7/site-packages',
 'purelib': '/Users/daniel/opt/py3env/lib/python3.7/site-packages',
 'scripts': '/Users/daniel/opt/py3env/bin'}

brettcannon · April 18, 2020, 7:57pm

I believe @pganssle has some ideas on standardizing editable installs but hasn’t had the time to tackle it.

dholth · April 18, 2020, 8:00pm

When the better solution is ready we deprecate this one.

Here’s the proposal from one year ago. It is not going to happen. Specification of editable installation

pganssle · April 18, 2020, 11:24pm

I’m strongly against this idea. We know basically exactly what standardized editable installs will look like, we just need someone to have the time to implement a proof of concept.

We already have way too much churn to be introducing new PEP 517 hooks that we plan to deprecate. This is basically a brand new workaround when we already have a workaround for this - notably, you just need to add a setup.py shim if you want to use editable installs.

I suspect it will take more effort to design a suitable specification than it would to create a proof of concept for the standardized editable installs. From my estimation it’s not an enormous amount of work to be done, just requires some significant refactoring.

I’m also worried that this will cause additional confusion in the messaging around adopting pyproject.toml. Right now people are already furious that there are sometimes bugs when the adopt PEP 517 inadvertently. Now we’ll be adding a hook that will change the behavior of -e for PEP 517 builds, which we will then later change again.

dholth · April 19, 2020, 12:32am

I thought the idea was overegineered and vague. I didn’t understand what advantages it would have over a simpler hook, and it’s been a year. An ordinary person complained about the missing feature in this forum.

Will you let me implementat my proposal in April 2021 if it’s still missing?

pf_moore · April 19, 2020, 9:16am

OK, let’s be clear here. Pip’s existing behaviour is fine. Annoying for backend developers who have to include a setup.py shim to support editables, but it works. The next version of editable support in pip will be a standardised mechanism. Pip’s not going to implement a new form of editable support without a standard to back it.

If a proposed standard explicitly states that it’s a temporary solution, and tools will have to change again when the “real” version of the standard comes along, don’t be surprised if that proposal gets rejected.

I get the argument about lack of time, but I’m genuinely not sure I know what standardised editable installs will look like - do you have a link to any sort of post describing where we’re at? If the answer is “no, it’s in the thread Daniel linked to” then that’s fine, but then I think there’s also someone’s time needed to pull together and summarise that thread.

merwok · April 20, 2020, 8:36pm

I think I remember that one! --prefix= is the same as --prefix "", meaning the parameter value is an empty string, which pip does to override the value that may be defined in a global config file like ~/.pydistutils.cfg or site-packages/distutils/distutils.cfg.

pganssle · April 20, 2020, 9:31pm

Yeah, the answer is in that thread, I think @pradyunsg has been champing at the bit to actually write a PEP for this (I was hoping for a proof of concept first, but if we’re at the stage where we need someone to implement it, maybe we should just do a draft PEP waiting on implementation). A simple summary is in this post.

A lot of the devil is in the details, but I think one of the key issues that makes this “dumb” hook unacceptable to me is that it blurs the line between frontend and backend in a way that we’ve been explicitly trying to move away from. This requires backends to know how to install the things they build. The proposed standard in the other thread was pretty simple: the build back-end prepares a “virtual wheel” that maps location-in-wheel to location-on-disk, plus the wheel metadata, and hands that all over to the front-end for installation. This is a clean separation of concerns between front-end and back-end, because the front-end doesn’t need to know anything about how to build the package and the back-end doesn’t need to know anything about how to expose packages to the system.

dholth · April 20, 2020, 9:42pm

If I could return { “purelib”: “.” } or { “purelib” : “./src” } and then all the installer has to do is add that directory to a .pth file I’d be happy, don’t even need the build to happen in the same step. I think some people were thinking about having the installer create a tree of symlinks to each individual file in your package in site-packages? and I’m not sure you can have both features.

sbidoul · May 4, 2020, 9:01pm

Given the complexity of the debate in the other thread, which IMO stems out of the vast number of use cases, why not coming back to this proposal? It may be considered somewhat “impure” in the sense that the back-end may undertake some install steps (if any, see below), but it certainly sounds practical to me. Apologies if I have missed other arguments against this approach.

Besides simplicity, its biggest advantage is that it gives back-ends full freedom to satisfy the requirements of their users, as niche or sophisticated as they could be.

To flesh it out a little bit, it could look like this, with two new back-end methods.

build_wheel_for_develop(wheel_directory, config_settings=None, metadata_directory=None)

The only requirement for that method is that the built (partial) wheel must include .dist-info metadata. For the rest it may be empty, or contain any other content typically found in a wheel.

develop(target_directory, config_settings=None)

This method is responsible for additionally making sure, if needed, that the project is importable when, at runtime, target_directory is in sys.path.

This method must return the list of files (or symlinks, it does not matter) created in target_directory and any of its sub-directories.

This method is optional, since a way to handle simple cases is for the back-end to add a .pth file in the wheel generated by build_wheel_for_develop.

The front-end would work like this for editable installs:

Same behavior as a normal install except that it calls build_wheel_for_develop in place of build_wheel. The partial wheel is installed normally, including dependencies, console scripts creation, etc.
When the partial wheel is installed, the front-end calls develop(target_directory) with target_directory depending on front-end options (–user, --target, --root).
The front-end appends the files returned by develop to RECORD (with hash, except for symlinks which are recorded without hash).
The front-end records a PEP 610 direct_url.json with the project directory as url and dir_info['editable'] = True.

In terms of POC, I personally glimpse quite clearly how that would work in pip and flit, and I assume implementing that in setuptools is not too hard (at least to achieve the equivalent of the current setup.py develop).

Hard hat on, send feedback please

dholth · May 4, 2020, 9:29pm

I apologize for generating this much discussion, the “third try” thread should be the most up to date. This is a POC referenced in another thread. It uses an existing hook to create the metadata, pip installs that *.dist-info and pip adds a .pth to a directory returned by the build system. Like my last comment in this thread. I like it better than the initial ‘let the build system poke site-packages’ idea. It is easier to implement for the build system if pip creates the .pth file.

sbidoul · May 4, 2020, 9:38pm

@dholth yep I’ve seen your POC.

That’s the part that is too restrictive IMO. And I believe it’s not necessary since that .pth file could be put in the partial wheel by the backend (edit - made a simple test, that seems to work just fine). The develop hook then optionally comes on top for more advanced use cases not covered by the simple .pth solution. So we get the best of both approaches.

dholth · May 4, 2020, 9:46pm

I would be open to any hook that lets enscons perform a develop install without having to provide setup.py. I bet you haven’t seen enscons’ current setup.py develop implementation. It delegates to easy_install, probably very close to what setuptools develop does. https://github.com/dholth/enscons/blob/master/enscons/setup.py#L9

Searched for .egg-link in poetry:

github.com

python-poetry/poetry/blob/e01564775bf58b3f11a5aa67841e9b4cb79de176/poetry/masonry/builders/editable.py

from __future__ import unicode_literals

import os
import shutil

from collections import defaultdict

from poetry.semver.version import Version
from poetry.utils._compat import decode

from .builder import Builder
from .sdist import SdistBuilder


class EditableBuilder(Builder):
    def build(self):
        return self._setup_build()

    def _setup_build(self):
        builder = SdistBuilder(self._poetry, self._env, self._io)

This file has been truncated. show original

Flit install_directly

github.com

takluyver/flit/blob/9a9e641e6e1c4ab3f19c9adc62c2280534800212/flit/install.py#L281




def _get_dirs(self, user):
    if self.python == sys.executable:
        return get_dirs(user=user)
    else:
        import json
        path = osp.join(osp.dirname(__file__), '_get_dirs.py')
        args = ['--user'] if user else []
        return json.loads(self._run_python(file=path, extra_args=args))


def install_directly(self):
    """Install a module/package into site-packages, and create its scripts.
    """
    dirs = self._get_dirs(user=self.user)
    os.makedirs(dirs['purelib'], exist_ok=True)
    os.makedirs(dirs['scripts'], exist_ok=True)


    dst = osp.join(dirs['purelib'], osp.basename(self.module.path))
    if osp.lexists(dst):
        if osp.isdir(dst) and not osp.islink(dst):
            shutil.rmtree(dst)

Setuptools develop.py

github.com

pypa/setuptools/blob/master/setuptools/command/develop.py

from distutils.util import convert_path
from distutils import log
from distutils.errors import DistutilsError, DistutilsOptionError
import os
import glob
import io

from setuptools.extern import six

import pkg_resources
from setuptools.command.easy_install import easy_install
from setuptools import namespaces
import setuptools

__metaclass__ = type


class develop(namespaces.DevelopInstaller, easy_install):
    """Set up package for development"""

This file has been truncated. show original

sbidoul · May 5, 2020, 5:32am

Note the develop hook as I propose it is NOT meant to call setup.py develop.

I had not seen the enscons setup.py develop implementation indeed, thanks for the pointer. I believe my proposal above does enable what you want, so in enscons you’d need to:

not do the easy_install.pth update
not generate .egg-link
not generate .egg-info in project directory
make a partial wheel with .dist-info metadata only, and a .pth file pointing to the project directory
the develop hook does not seem to be needed in that case since all you need is a .pth entry pointing to your project directory

For setuptools and poetry it would be similar.

Flit has two modes:

.pth file: same as above, .pth in partial wheel
symlink: add a develop hook that creates the symlink

The tree of symlinks approach can be implemented in the develop hook by any back-end that wants it.

uranusjr · May 5, 2020, 8:20am

Would it work if it put the symlinks in the partial wheel for the frontend to install, just like the pth? I know zipfile does not support symlinks natively, but symlinks are always a platform-specific thing anyway and it’s easy to do if you don’t care about portability.

I think what I’m getting at is maybe the develop hook isn’t even needed for the initial proposal, since no existing backends actually need it.

sbidoul · May 5, 2020, 9:43am

The partial wheel would be temporary and only a mean of communication between the back-end and front-end, so portability should not be an issue. If there exists a mean to record symlinks in a ZipFile that is supported by the python versions we want that could be sufficient indeed.