Pip 19.1 and installing in editable mode with pyproject.toml

Thanks, @ncoghlan. To clarify, in your suggestion are there any combinations or scenarios where either an error should be raised or a warning logged in current pip?

Also, is there any concern that when it comes time to standardizing editable mode, we won’t be able to (or for some reason might not want to) match setuptools’ current behavior using backends, in which case we would have to break compatibility in some other, possibly bigger way at a later date to make the API work?

Some people on the tracker were suggesting the use of a different flag / option name for the future PEP 517-defined editable mode (e.g. --develop, --source-install, --inplace, etc. as opposed to --editable). With the seamless approach you described, you’re saying no new option name would be needed in the future?

I’d say we don’t need an additional flag and just falling back to PEP 518 style isolation for editable installs should be fine.

3 Likes

FYI, so something can be ready, earlier I posted a PR that would revert the pyproject.toml-related changes and restore things to the 19.0.3 state: https://github.com/pypa/pip/pull/6449

Just as before, I’m happy to do whatever PyPA decides. However, I’d still personally be interested in hearing answers to the questions I asked in my post above (mainly the second paragraph). I asked those questions because I think it’s the crux of what I understood to be the reason for wanting the --no-use-pep517 flag to be required.

Personally, I would be fine if a message is emitted (something like “no-pep517 assumed since editable is given; this may change in the future”), but a warning or error would be very confusing to an end user until we can offer a PEP 517 compatible alternative.

2 Likes

My current intuition is that a pep517 editable install can end-up being little more than a pass-through to the backend. IOW, something like “Dear backend, please install metadata and do whatever you think best for the source code in this local directory to be executable in place”. The current .egg-link method of setup.py develop would still be allowed, although considered legacy with maybe a long term deprecation path. In that view there should not be any backwards compatibility issue, because --editable + setuptools would still be allowed to work exactly as it does today.

In the worst case a compatibility issue is discovered while standardizing editable installs, it will still be possible to add a new pip option (--inplace?) for the new behaviour and keep --editable, as a setuptools-only function.

1 Like

The Packaging summit topic PyPA governance is probably relevant here.

My thoughts (as they relate to this specific issue) as as follows: The levels of decision making for something like this are:

  1. Whatever the pip developers involved in the discussion, and especially the person writing the PR, think is best.
  2. The consensus of the pip developers as a group.
  3. The PyPA as a whole
  4. Standards-level discussion

Getting pip developer consensus is a matter of availability (for example, @dstufft hasn’t participated in this issue yet) but once enough of us get involved, the majority view is usually reasonably easy to identify. Ideally, there shouldn’t ever be any need to go beyond this for any other reason than the pip devs asking for a decision from the PyPA - I hope we don’t ever reach a point where we need a process for users to ask the PyPA to override the decisions of a project’s owners :frowning:

The problem is that there is no PyPA decision making process at the moment - there’s not even an easy to identify list of who is in the PyPA in order to take their views into consideration. So the decision here has to reside with the pip developers.

So I guess the question is, what is the pip developers’ consensus (taking into account the feedback here and on the tracker)? I was in favour of the solution @cjerdonek implemented (as, I assume, was he :wink:) but both of us have said we’d be happy to accept an alternative decision. @pradyunsg and @xafer have both expressed the view that -e should imply --no-use-pep517 (although there’s still the details to be worked out - this all started because of "conflict with the backend dependencies" error running pip install in editable mode with pyproject.toml / PEP-517 enabled project · Issue #6314 · pypa/pip · GitHub, so we can’t just revert without taking a view on what we do about that issue).

I’d say that so far, pip-dev consensus is in favour of -e implying --no-use-pep517. So if we assume that, what are the next steps?

I don’t honestly have the time over the next couple of weeks to do any of this - and I know that PyCon is coming up, so I suspect time is short for others as well. It might be nice, though, if we could set some sort of realistic expectation for the people waiting for this as to when we’d hope to have a fixed release out (it’s hard to get the decisions made when people are looking for a release in a matter of hours :slightly_frowning_face:).

1 Like

Unfortunately I think this ship has sailed, because we’ve opted a large number of people in to PEP 517/518, so we can no longer say, “Oh this just won’t work if you are using a marker of PEP 517”, because it has been working for a ton of people already.

That said, if we do find that we must make breaking changes in order to support editable installs, there’s always the (not exactly palatable) option of adding a new “PEP XXX-compatible editable install” option, like pip -E, where pip -e invokes setup.py develop and raises a deprecation warning, and pip -E uses the new backend methods. Depending on the precise degree and type of incompatibilities, we can probably go the other way as well, where -e uses the new version and -E gives you the legacy “bug-compatible” version of editable installs.

1 Like

My current intuition is that a pep517 editable install can end-up being little more than a pass-through to the backend. IOW, something like “Dear backend, please install metadata and do whatever you think best for the source code in this local directory to be executable in place”. The current .egg-link method of setup.py develop would still be allowed, although considered legacy with maybe a long term deprecation path. In that view there should not be any backwards compatibility issue, because --editable + setuptools would still be allowed to work exactly as it does today.

Exactly, it’s really not very complicated. The hook could be:

build_inplace(site_packages_directory, config_settings=None):
    ...

Existing backends basically already do what is necessary:

  • setuptools executes its complex hierarchy of build commands initiated by setup.py develop and then creates an .egg-link file
  • poetry has poetry install, which uses poetry’s resolver and optionally calls a build script, and creates an .egg-link file generates a setup.py and calls pip install -e $pkg under the hood :unamused:
  • flit has a very minimal build and has the --symlink and --pth-file options for linking to site-packages

The question for me is if we want to go with real isolation of the build environment from the install environment, shouldn’t we let the build front-end be the one that actually makes the links? We could probably deprecate .egg-link that way, and the the build frontend make a platform-appropriate decision about how to include the source in site packages. For that, the signature would be:

def build_inplace(config_settings: Optional[dict] = None) -> Iterable[str]:
    ...

Where the resulting collection of str would be a list of paths to link onto site-packages. What do you think @sbidoul ?

In the worst case a compatibility issue is discovered while standardizing editable installs, it will still be possible to add a new pip option ( --inplace ?) for the new behaviour and keep --editable , as a setuptools-only function.

Or I think that might be the appropriate time to apply a --no-use-pepXXX flag, or just remove build-backend = setuptools.build_meta from pyproject.toml to force the old behavior (as the incompatibility would inevitably be with setuptools, I think). You wouldn’t really need a new name for the option. Although I like pip build -i|--inplace at some point

Warnings that an end user can’t do anything about aren’t useful, so I see the preferred outcome as:

  • for now, silently use direct setup.py invocation for -e, and only emit an error if that isn’t possible, or doesn’t work
  • eventually, silently use a build backend API, with a fallback to direct setup.py invocation that emits a warning to upgrade setuptools

I don’t see any reason why we’d accept a backend API spec update that didn’t allow the setuptools backend to correctly reproduce the behaviour of “setup.py develop”

2 Likes

(Cross-posted from here, for better visibility.)

tl;dr I think we need editable installs (or something of similar simplicity) in order to ease the entry pathway for novice Python package developers.

Long version Something I don’t think I’ve seen in the discussion so far is this: I think it’s also important to keep in mind that editable installs represent by far one of the easiest, most-minimal-config approaches for beginning package developers to get their work under reliable tests. @pfmoore’s comment about running everything through tox, including local dev testing, got me thinking about how I might rework my project setup along those lines… but properly configuring tox to cleanly handle even just one environment for a local test suite is not trivial, and I figure it’s not going to be obvious to a new package developer why the best practice is wrapping the primary testing tool with tox. pip install -e . followed by pytest is simple, quick, and obvious, even if it’s not the most robust approach.

So, I think that editable install support should be eliminated only if another alternative is identified that is comparably simple for new developers. (It seems like the consensus is moving toward retaining editable install support, so this extra argument may be somewhat superfluous, but I wanted to make sure it was made regardless.)

If you want simplicity for newcomers, it’s hard to beat simply running pytest without running pip install -e… that’s what I do. There are some confusing details about when this does and doesn’t work, but AFAICT that’s true of pytest no matter how you run it.

1 Like

It’s not just for newcomers. If you’re using a non-pure python package (e.g., any thing with a build step like cythonize, extension module compilation, webpack for building web assets), it’s extremely important for efficient development, since this requires that a build be run at some point. And if this build is not editable (i.e., just pip install .), then every single change you make from that point requires another pip install ..

The alternative to this is just interfacing directly with the build backend, which not only requires that all build dependencies be pre-installed the same environment, but also skips pip-based checking of dependencies. This is ugly and goes against the spirit of PEP 517/518.

I’m pretty sure that only works because the current directory is implicitly added to sys.path. If you’re in the wrong directory, or if you use a src/ layout, this doesn’t work. It’s also not just tests. If you’re doing any kind of REPL-based development, having easy access to source changes matters.

It seems like you guys are missing a bunch of real use cases for which editable mode is a godsend:

  1. Development on any non-pure python package
  2. Working in a team with a large variance in skill level (expansion of your “newcomer” use case)
  3. Having a complex library-like project with lots of dependencies that you don’t necessarily want to pin with requirements files. This becomes an “ensure I have a valid dev environment” switch that…
    1. Ensures the more general (non-pinned) project dependencies are met
    2. Tends to catch bugs in dependencies faster than any other method I can think of
    3. Tests the build process itself thoroughly

I want to hammer home how useful this “give me a valid dev environment” feature is. For many more complex projects, maintaining a bunch of consistent, valid development environments is not a trivial task. Being able to build all these checks into the behavior of a single build frontend command is an enormous relief and productivity multiplier.

3 Likes

I cannot overstate this. At work, we use invoke so we have a single invoke build task that takes care of everything needed to have the developer ready to work on a project: install dependencies, using an editable install, generating code, etc. This is a significant boost to the entire team.

I would love to see something similar in pip, which I thought, until reading this thread, was the job of pip install -e .. If we get this fixed, this could become generally widespread in the community: for almost every project, it would be reasonable to assume that you can get started to work on a project by creating a virtual environment and running pip install -e. (or pip install -e .[dev]).

2 Likes

I’m sure I’m misunderstanding something, because I feel it’s completely the other way around. If your software is pure Python, and editable install makes whatever you edit reflect directly in the installation, and makes development much easier. A compiled module, however, needs to be compiled to be importable in Python, so the editable install isn’t that useful since you need that extra build step anyway, and can just add pip install . as a build step.

But one way or the other, there’s no doubt the editable install is a valuable part of Python packaging, and needs to be spec-ed eventually.

Yeah it’s hard to understand until you work with one in practice a lot. A non-pure package is part python (often like 95% regular python), and part something else that needs to be compiled or built somehow. This means the “part something else” has to be compiled at least once (necessitating a pip install [-e] of some kind). This is often a part that changes far less often then the rest of your code. For day-to-day, you want to be able to edit python like normal, and see saved changes be reflected immediately in tests, scripts, and REPLs. In a pure python project you can get a way with using the right working directory or PYTHONPATH mangling or other sys.path tricks, but with non-pure, you have to build at least once for the project to work at all. So if there’s no editable install, you lose the instant feedback on the other 95% of your code.

In a way, it’s about being able to leverage two of Python’s advantages at once: extensibility and rapid developer feedback.

Absolutely. I just want to help clarify the use cases.

I agree with Nick here, on both fronts.

We don’t need to add a -E to pip for this. With my pip maintainer hat on, that’s an additional option that exists and I don’t see any strong reasoning for it so I’ll be a -1 for it.

Regarding the standardization and all, we have a better place for that conversation.

2 Likes

There seems to be a subthread here arguing for keeping editable installs, but I am fairly certain that deprecating editable installs was never on the table, so this argument doesn’t really need to be made.

The question in this thread is that PEP 517 does not support editable installs because it was out of scope of the original standardization effort. Prior to pip 19.1, attempting an editable install on a PEP 517 project would fall back to invoking setup.py develop (which means it would not be using the normal PEP 517 mechanism). The question is that while we work to build the standard that will add editable support, should you have to explicitly say that you want to circumvent PEP 517 (or invoke with a different front-end) in order to get an editable install?

2 Likes

To be clear, my suggestion for -E or --legacy-editable or something is a fallback position. I was addressing @cjerdonek’s (legitimate) concern that during the standardization process, we may find that the setup.py develop workflow is fundamentally broken in some way and that we’ll be causing more pain by not breaking things now.

My points are in general agreement with @ncoghlan, I don’t think this is a likely outcome, but even if it does happen, we have the option of adding a new option and deprecating the old one to avoid breaking existing code (or adding a “bug compatible” legacy option of some sort).

1 Like

The original goal of pip was basically to be just an installer. Over time, that scope has grown a lot to cover all sorts of workflow related tasks. I’m genuinely unsure on how far pip should go down that route - as a maintainer, it’s a lot of added complexity, as a user it’s a mix (I use some of the features, but don’t use others).

I’d love to see better tool support for development workflows. I don’t know how far the tool in question should be pip. I’d love to see a more clearly defined scope for pip.

This is a super important use case to address. But I think it’s asking too much of pip to expect pip install -e address it – it’s just not high enough level to handle the whole job in the long run (though I see why it’s one of the better options right now). Some things it can’t do:

  • Create/manage the venv itself
  • Distinguish between unlocked and locked dependency ranges. When creating a package, we use unlocked ranges. In development, we use locked ranges. pip install -e is an extension to the “creating a package” part of the system, so it uses the wrong dependencies for development.
  • Make sure your environment is actually up to date before running tests

A pipenv-like tool can take care of all of this seamlessly for you.

And if you have a pipenv-like tool, editable installs are actually less compelling in general… if pipenv test always does a quick incremental rebuild before invoking the test tool, then that’s just as easy to use as an editable install, and far more friendly to beginning users who aren’t equipped to keep track of which kinds of edits are ok with an editable install, and which ones force you to re-run pip (e.g. compiled code, version bumps, dependency bumps). The tradeoffs here are tricky, because speed matters too. But if I were building a tool like this, I’d probably support editable installs, but I’d leave then disabled by default, so experts could get the speed boost of skipping rebuilds, but only if they’re prepared to track cache consistency in their head.

(Note: a combined build-and-test command is actually the workflow that I’ve always used for projects with native code. E.g. numpy and scipy have a runtests.py tool that takes care of this. So this isn’t something I’m just inventing out of thin air.)

Hmm, so here’s a potentially interesting feature for a future version of editable installs. This suggests editable installs are really just a optimization to make reinstalling really fast, and they sacrifice some accuracy to do that. Maybe we can make them a faster or more accurate optimization. For example, what if the backend could provide some extra metadata to hint to the frontend when a rebuild is required ("if any of these files change, do a full reinstall: setup.py, lib/foo.c, …). Then a pipenv-like tool could stay fast when only .py files are updated, but also take on some of the cache consistency tracking that we currently have to do in our heads.

2 Likes