Pip 19.1 and installing in editable mode with pyproject.toml

Packaging
1

So I’ll just start this thread since a number of people have brought it up on the tracker in various places since pip 19.1 was released, and it’s better to have the discussion happening in one place.

I don’t have time to write up a full email with background (I think most of you already know the background anyways), and I’m not necessarily the best one to do so, but the basic question people are asking to revisit / discuss is how pip should behave if pip install -e is run on a project containing a pyproject.toml file.

The PEP 517 code in pip 19.0.3 didn’t really contemplate editable mode and led to something breaking in one case, which is what spurred making the behavior for editable mode explicit, as opposed to continuing longer with undefined behavior.

Two cases to distinguish when deciding this are–

  1. with the build-backend key in the [build-system] table

  2. without the build-backend key in the [build-system] table

In the former case, PEP 517 says the source tree is a pyproject.toml-style source tree and follows the PEP 517 specification. In the latter, PEP 517 says it is setup.py-style and is not using the specification.

When considering what to do, IMO there should also be a plan for what will happen when editable mode is standardized, and that plan should include what any migration / deprecation paths will look like.

1 Like
2

I think that the with and without build-backend cases do not really need to be distinguished here. I think we should fall back to PEP 518 and invoke setup.py develop in case a build-backend.requires key is found, and otherwise just invoke setup.py develop.

This is not like the case with setuptools.build_meta:__legacy__, where we have some behavior that we want to deprecate, this is just a feature we haven’t gotten around to standardizing yet, so we can’t really implement it properly. We should be doing our best to make sure that editable installs continue to work as they did before to the best extent possible. Under the hood the implementation will probably change once PEP 517 gets the capacity to handle editable installs, but I don’t think that should affect users for whom it was already working.

For people who rely on PEP 517 for the build editable installs were already properly broken, so they are seeing no change anyway.

2 Likes
3

When PEP 517 opted out of defining an API for editable mode, the intent was “Keep using setup.py develop for this feature”, not “It’s OK to break it until there’s a defined backend API for it”. So until there’s a new PEP expanding the build backend API to support editable mode, pip should treat -e as implying --no-use-pep517.

Rationale:

  • -e has historically required python setup.py develop to work, so projects currently using the flag will support that invocation
  • projects using a PEP 517 backend that doesn’t support python setup.py develop (likely because they don’t include a setup.py file) won’t be using -e either (since it has historically relied on setup.py develop)
  • after the build backend API does gain optional editable mode support via a new PEP, then setuptools can implement it for build_meta and build_meta_legacy, pip can bump the version of the implicit setuptools build requirement, and drop the interim workaround of "-e implies --no-use-pep517"

And all of this should be completely transparent to end users, seamlessly switching them from “no PEP 517 at all” to “mostly installing via the build backend API, but falling back to direct setup.py invocation for editable mode” to “all installation requests are handled via the build backend API, even editable mode ones”.

13 Likes
4

Who did not follow a lot has already been said here https://github.com/pypa/pip/issues/6434

5

Would it be helpful if I just showed you an actual use case of how the (apparently incidental) behavior of pip 19.0.3 worked for us?

Example pyproject.toml:

[build-system]
requires = ["setuptools>=40.1.0", "wheel", "cython", "numpy<1.16", "toml"]
build-backend = "setuptools.build_meta"

The corresponding setup.py use those build dependencies to cythonize select modules and compile some extension modules against numpy. Toml is for reading additional build metadata.

The advantage of this setup is that I can tell my entire team to “just run pip install -e .” to do three things:

  1. Get all necessary build dependencies (PEP 518)
  2. Install any new dependencies
  3. Run any necessary build or compilation steps

And it stopped all complaints and time wasted about not having a functional dev environment.

In CI, I can use pep517 to do the same thing to build a wheel and sdist from a source tree and publish that to our internal package index. My understanding is this is a preview of an anticipated pip build feature. This is again nice because it’s one step to collect all dependencies and generate a build artifact. This is, I believe, the entire point of PEP 517.

The convenience of this that of having build frontend, because it means I can use flit or poetry or setuptools as I see fit for the project and my team can be trained to lean on that rather than worrying about the lower-level details of how a project is built. It means the steps for getting a functioning dev environment are:

$ git clone git@domain.com:path/to/repo.git my-repo
$ cd my-repo
$ virtualenv -p python3.6 .env
$ pip install -e .

This simplicity even in the face of complex build requirements is made possible by PEP 518 and the work you guys have done on that side. I could live without 517 (even if I want it) because that’s mostly a CI/package maintainer concern, and I can dockerize or script that however I want. But requiring that all my devs need to pass --no-use-pep517 every time they execute a use case that PEP 517 does not define…well, that just seems inane.

I get that pip needs to move forward, but I think the concern that “once we define a PEP 517-compliant version of editable installs, it will break even harder” is misplaced.

And all of this should be completely transparent to end users, seamlessly switching them from “no PEP 517 at all” to “mostly installing via the build backend API, but falling back to direct setup.py invocation for editable mode” to “all installation requests are handled via the build backend API, even editable mode ones”.

Like @ncoghlan said, this should be a bit more seemless

1 Like
6

And I think I mentioned this elsewhere, I’m more than willing to help start drafting specs for a PEP 517-compliant editable install mode, but I don’t really have the history or full understanding of the use cases beyond my own.

As far as I can tell we need to define hooks for in-place builds analagous to the existing PEP 517 ones: get_requires_for_build_inplace and build_inplace. And probably define who is resposnible for adding the source tree to sys.path (and how this gets done).

I won’t be at the mini-summit, since I can’t make it to PyCon. Is there a good way I can follow what’s going on?

7

Thanks, @ncoghlan. To clarify, in your suggestion are there any combinations or scenarios where either an error should be raised or a warning logged in current pip?

Also, is there any concern that when it comes time to standardizing editable mode, we won’t be able to (or for some reason might not want to) match setuptools’ current behavior using backends, in which case we would have to break compatibility in some other, possibly bigger way at a later date to make the API work?

Some people on the tracker were suggesting the use of a different flag / option name for the future PEP 517-defined editable mode (e.g. --develop, --source-install, --inplace, etc. as opposed to --editable). With the seamless approach you described, you’re saying no new option name would be needed in the future?

8

I’d say we don’t need an additional flag and just falling back to PEP 518 style isolation for editable installs should be fine.

3 Likes
9

FYI, so something can be ready, earlier I posted a PR that would revert the pyproject.toml-related changes and restore things to the 19.0.3 state: https://github.com/pypa/pip/pull/6449

Just as before, I’m happy to do whatever PyPA decides. However, I’d still personally be interested in hearing answers to the questions I asked in my post above (mainly the second paragraph). I asked those questions because I think it’s the crux of what I understood to be the reason for wanting the --no-use-pep517 flag to be required.

10

Personally, I would be fine if a message is emitted (something like “no-pep517 assumed since editable is given; this may change in the future”), but a warning or error would be very confusing to an end user until we can offer a PEP 517 compatible alternative.

2 Likes
11

My current intuition is that a pep517 editable install can end-up being little more than a pass-through to the backend. IOW, something like “Dear backend, please install metadata and do whatever you think best for the source code in this local directory to be executable in place”. The current .egg-link method of setup.py develop would still be allowed, although considered legacy with maybe a long term deprecation path. In that view there should not be any backwards compatibility issue, because --editable + setuptools would still be allowed to work exactly as it does today.

In the worst case a compatibility issue is discovered while standardizing editable installs, it will still be possible to add a new pip option (--inplace?) for the new behaviour and keep --editable, as a setuptools-only function.

1 Like
12

The Packaging summit topic PyPA governance is probably relevant here.

My thoughts (as they relate to this specific issue) as as follows: The levels of decision making for something like this are:

  1. Whatever the pip developers involved in the discussion, and especially the person writing the PR, think is best.
  2. The consensus of the pip developers as a group.
  3. The PyPA as a whole
  4. Standards-level discussion

Getting pip developer consensus is a matter of availability (for example, @dstufft hasn’t participated in this issue yet) but once enough of us get involved, the majority view is usually reasonably easy to identify. Ideally, there shouldn’t ever be any need to go beyond this for any other reason than the pip devs asking for a decision from the PyPA - I hope we don’t ever reach a point where we need a process for users to ask the PyPA to override the decisions of a project’s owners :frowning:

The problem is that there is no PyPA decision making process at the moment - there’s not even an easy to identify list of who is in the PyPA in order to take their views into consideration. So the decision here has to reside with the pip developers.

So I guess the question is, what is the pip developers’ consensus (taking into account the feedback here and on the tracker)? I was in favour of the solution @cjerdonek implemented (as, I assume, was he :wink:) but both of us have said we’d be happy to accept an alternative decision. @pradyunsg and @xafer have both expressed the view that -e should imply --no-use-pep517 (although there’s still the details to be worked out - this all started because of "conflict with the backend dependencies" error running pip install in editable mode with pyproject.toml / PEP-517 enabled project · Issue #6314 · pypa/pip · GitHub, so we can’t just revert without taking a view on what we do about that issue).

I’d say that so far, pip-dev consensus is in favour of -e implying --no-use-pep517. So if we assume that, what are the next steps?

I don’t honestly have the time over the next couple of weeks to do any of this - and I know that PyCon is coming up, so I suspect time is short for others as well. It might be nice, though, if we could set some sort of realistic expectation for the people waiting for this as to when we’d hope to have a fixed release out (it’s hard to get the decisions made when people are looking for a release in a matter of hours :slightly_frowning_face:).

1 Like
13

Unfortunately I think this ship has sailed, because we’ve opted a large number of people in to PEP 517/518, so we can no longer say, “Oh this just won’t work if you are using a marker of PEP 517”, because it has been working for a ton of people already.

That said, if we do find that we must make breaking changes in order to support editable installs, there’s always the (not exactly palatable) option of adding a new “PEP XXX-compatible editable install” option, like pip -E, where pip -e invokes setup.py develop and raises a deprecation warning, and pip -E uses the new backend methods. Depending on the precise degree and type of incompatibilities, we can probably go the other way as well, where -e uses the new version and -E gives you the legacy “bug-compatible” version of editable installs.

1 Like
14

My current intuition is that a pep517 editable install can end-up being little more than a pass-through to the backend. IOW, something like “Dear backend, please install metadata and do whatever you think best for the source code in this local directory to be executable in place”. The current .egg-link method of setup.py develop would still be allowed, although considered legacy with maybe a long term deprecation path. In that view there should not be any backwards compatibility issue, because --editable + setuptools would still be allowed to work exactly as it does today.

Exactly, it’s really not very complicated. The hook could be:

build_inplace(site_packages_directory, config_settings=None):
    ...

Existing backends basically already do what is necessary:

  • setuptools executes its complex hierarchy of build commands initiated by setup.py develop and then creates an .egg-link file
  • poetry has poetry install, which uses poetry’s resolver and optionally calls a build script, and creates an .egg-link file generates a setup.py and calls pip install -e $pkg under the hood :unamused:
  • flit has a very minimal build and has the --symlink and --pth-file options for linking to site-packages

The question for me is if we want to go with real isolation of the build environment from the install environment, shouldn’t we let the build front-end be the one that actually makes the links? We could probably deprecate .egg-link that way, and the the build frontend make a platform-appropriate decision about how to include the source in site packages. For that, the signature would be:

def build_inplace(config_settings: Optional[dict] = None) -> Iterable[str]:
    ...

Where the resulting collection of str would be a list of paths to link onto site-packages. What do you think @sbidoul ?

In the worst case a compatibility issue is discovered while standardizing editable installs, it will still be possible to add a new pip option ( --inplace ?) for the new behaviour and keep --editable , as a setuptools-only function.

Or I think that might be the appropriate time to apply a --no-use-pepXXX flag, or just remove build-backend = setuptools.build_meta from pyproject.toml to force the old behavior (as the incompatibility would inevitably be with setuptools, I think). You wouldn’t really need a new name for the option. Although I like pip build -i|--inplace at some point

15

Warnings that an end user can’t do anything about aren’t useful, so I see the preferred outcome as:

  • for now, silently use direct setup.py invocation for -e, and only emit an error if that isn’t possible, or doesn’t work
  • eventually, silently use a build backend API, with a fallback to direct setup.py invocation that emits a warning to upgrade setuptools

I don’t see any reason why we’d accept a backend API spec update that didn’t allow the setuptools backend to correctly reproduce the behaviour of “setup.py develop”

2 Likes
16

(Cross-posted from here, for better visibility.)

tl;dr I think we need editable installs (or something of similar simplicity) in order to ease the entry pathway for novice Python package developers.

Long version Something I don’t think I’ve seen in the discussion so far is this: I think it’s also important to keep in mind that editable installs represent by far one of the easiest, most-minimal-config approaches for beginning package developers to get their work under reliable tests. @pfmoore’s comment about running everything through tox, including local dev testing, got me thinking about how I might rework my project setup along those lines… but properly configuring tox to cleanly handle even just one environment for a local test suite is not trivial, and I figure it’s not going to be obvious to a new package developer why the best practice is wrapping the primary testing tool with tox. pip install -e . followed by pytest is simple, quick, and obvious, even if it’s not the most robust approach.

So, I think that editable install support should be eliminated only if another alternative is identified that is comparably simple for new developers. (It seems like the consensus is moving toward retaining editable install support, so this extra argument may be somewhat superfluous, but I wanted to make sure it was made regardless.)

17

If you want simplicity for newcomers, it’s hard to beat simply running pytest without running pip install -e… that’s what I do. There are some confusing details about when this does and doesn’t work, but AFAICT that’s true of pytest no matter how you run it.

1 Like
18

It’s not just for newcomers. If you’re using a non-pure python package (e.g., any thing with a build step like cythonize, extension module compilation, webpack for building web assets), it’s extremely important for efficient development, since this requires that a build be run at some point. And if this build is not editable (i.e., just pip install .), then every single change you make from that point requires another pip install ..

The alternative to this is just interfacing directly with the build backend, which not only requires that all build dependencies be pre-installed the same environment, but also skips pip-based checking of dependencies. This is ugly and goes against the spirit of PEP 517/518.

I’m pretty sure that only works because the current directory is implicitly added to sys.path. If you’re in the wrong directory, or if you use a src/ layout, this doesn’t work. It’s also not just tests. If you’re doing any kind of REPL-based development, having easy access to source changes matters.

It seems like you guys are missing a bunch of real use cases for which editable mode is a godsend:

  1. Development on any non-pure python package
  2. Working in a team with a large variance in skill level (expansion of your “newcomer” use case)
  3. Having a complex library-like project with lots of dependencies that you don’t necessarily want to pin with requirements files. This becomes an “ensure I have a valid dev environment” switch that…
    1. Ensures the more general (non-pinned) project dependencies are met
    2. Tends to catch bugs in dependencies faster than any other method I can think of
    3. Tests the build process itself thoroughly

I want to hammer home how useful this “give me a valid dev environment” feature is. For many more complex projects, maintaining a bunch of consistent, valid development environments is not a trivial task. Being able to build all these checks into the behavior of a single build frontend command is an enormous relief and productivity multiplier.

3 Likes
19

I cannot overstate this. At work, we use invoke so we have a single invoke build task that takes care of everything needed to have the developer ready to work on a project: install dependencies, using an editable install, generating code, etc. This is a significant boost to the entire team.

I would love to see something similar in pip, which I thought, until reading this thread, was the job of pip install -e .. If we get this fixed, this could become generally widespread in the community: for almost every project, it would be reasonable to assume that you can get started to work on a project by creating a virtual environment and running pip install -e. (or pip install -e .[dev]).

2 Likes
20

I’m sure I’m misunderstanding something, because I feel it’s completely the other way around. If your software is pure Python, and editable install makes whatever you edit reflect directly in the installation, and makes development much easier. A compiled module, however, needs to be compiled to be importable in Python, so the editable install isn’t that useful since you need that extra build step anyway, and can just add pip install . as a build step.

But one way or the other, there’s no doubt the editable install is a valuable part of Python packaging, and needs to be spec-ed eventually.