Yeah it’s hard to understand until you work with one in practice a lot. A non-pure package is part python (often like 95% regular python), and part something else that needs to be compiled or built somehow. This means the “part something else” has to be compiled at least once (necessitating a pip install [-e] of some kind). This is often a part that changes far less often then the rest of your code. For day-to-day, you want to be able to edit python like normal, and see saved changes be reflected immediately in tests, scripts, and REPLs. In a pure python project you can get a way with using the right working directory or PYTHONPATH mangling or other sys.path tricks, but with non-pure, you have to build at least once for the project to work at all. So if there’s no editable install, you lose the instant feedback on the other 95% of your code.
In a way, it’s about being able to leverage two of Python’s advantages at once: extensibility and rapid developer feedback.
Absolutely. I just want to help clarify the use cases.
We don’t need to add a -E to pip for this. With my pip maintainer hat on, that’s an additional option that exists and I don’t see any strong reasoning for it so I’ll be a -1 for it.
Regarding the standardization and all, we have a better place for that conversation.
There seems to be a subthread here arguing for keeping editable installs, but I am fairly certain that deprecating editable installs was never on the table, so this argument doesn’t really need to be made.
The question in this thread is that PEP 517 does not support editable installs because it was out of scope of the original standardization effort. Prior to pip 19.1, attempting an editable install on a PEP 517 project would fall back to invoking setup.py develop (which means it would not be using the normal PEP 517 mechanism). The question is that while we work to build the standard that will add editable support, should you have to explicitly say that you want to circumvent PEP 517 (or invoke with a different front-end) in order to get an editable install?
To be clear, my suggestion for -E or --legacy-editable or something is a fallback position. I was addressing @cjerdonek’s (legitimate) concern that during the standardization process, we may find that the setup.py develop workflow is fundamentally broken in some way and that we’ll be causing more pain by not breaking things now.
My points are in general agreement with @ncoghlan, I don’t think this is a likely outcome, but even if it does happen, we have the option of adding a new option and deprecating the old one to avoid breaking existing code (or adding a “bug compatible” legacy option of some sort).
The original goal of pip was basically to be just an installer. Over time, that scope has grown a lot to cover all sorts of workflow related tasks. I’m genuinely unsure on how far pip should go down that route - as a maintainer, it’s a lot of added complexity, as a user it’s a mix (I use some of the features, but don’t use others).
I’d love to see better tool support for development workflows. I don’t know how far the tool in question should be pip. I’d love to see a more clearly defined scope for pip.
This is a super important use case to address. But I think it’s asking too much of pip to expect pip install -e address it – it’s just not high enough level to handle the whole job in the long run (though I see why it’s one of the better options right now). Some things it can’t do:
Create/manage the venv itself
Distinguish between unlocked and locked dependency ranges. When creating a package, we use unlocked ranges. In development, we use locked ranges. pip install -e is an extension to the “creating a package” part of the system, so it uses the wrong dependencies for development.
Make sure your environment is actually up to date before running tests
A pipenv-like tool can take care of all of this seamlessly for you.
And if you have a pipenv-like tool, editable installs are actually less compelling in general… if pipenv test always does a quick incremental rebuild before invoking the test tool, then that’s just as easy to use as an editable install, and far more friendly to beginning users who aren’t equipped to keep track of which kinds of edits are ok with an editable install, and which ones force you to re-run pip (e.g. compiled code, version bumps, dependency bumps). The tradeoffs here are tricky, because speed matters too. But if I were building a tool like this, I’d probably support editable installs, but I’d leave then disabled by default, so experts could get the speed boost of skipping rebuilds, but only if they’re prepared to track cache consistency in their head.
(Note: a combined build-and-test command is actually the workflow that I’ve always used for projects with native code. E.g. numpy and scipy have a runtests.py tool that takes care of this. So this isn’t something I’m just inventing out of thin air.)
Hmm, so here’s a potentially interesting feature for a future version of editable installs. This suggests editable installs are really just a optimization to make reinstalling really fast, and they sacrifice some accuracy to do that. Maybe we can make them a faster or more accurate optimization. For example, what if the backend could provide some extra metadata to hint to the frontend when a rebuild is required ("if any of these files change, do a full reinstall: setup.py, lib/foo.c, …). Then a pipenv-like tool could stay fast when only .py files are updated, but also take on some of the cache consistency tracking that we currently have to do in our heads.
I agree. I have used the example of what we have at work just to illustrate that it is powerful to be able to clone a project and execute a single command to get your work copy ready for development; I didn’t mean to imply that pip should be expanded to support all kinds of customization (like “generating code” in my example). Sorry that I was unclear.
My intention was to show that by making pip install -e . work as expected (install dependencies and the code in develop mode) it would greatly simplify the contribution learning curve.
You are right definitely. As I commented above, my example was merely to illustrate the power of a single command to have dependencies installed and code ready to work with. I believe a working editable install would sufficient for many projects.
I don’t mind personally, but since it was often claimed (IIRC) that pip install -e . supercedes python setup.py develop, it’s a net annoyance for users if the former ceases to function and they have to re-learn the latter (and online resources have to be updated etc.).
Is there anyone even on the side of "pip should always be incapable of doing editable installs"? I didn’t think that was up for discussion, and if someone wants to propose it, I suggest a new thread.
But, the manner in which editable installs were broken, via (AFAICT) a breaking regression of pip that was not announced ahead of time, certainly gave me the impression that the pip/PyPA decision-makers considered them a minor and/or tangential feature. I posted here to try to add my voice to a message from the community, trying to communicate that the editable install (or something with comparable functionality) is an important feature of the packaging tooling, for a variety of reasons.
I would guess this is where much of any tone of ‘arguing in favor of pip support for editable installs’ is coming from.
Well, you can rest assured that this is not true, and I think even if it were considered a “minor/trivial feature” (or even an undesirable anti-pattern), I think most/all the pip maintainers would not deliberately break backwards compatibility with zero notice.
I think my suggestion from my initial comment is still the best way to go - falling back to setup.py develop whenever -e is invoked, and using a PEP 518 isolated build environment if build-system.requires is present and --no-build-isolation is not specified.
This suggests editable installs are really just a optimization to make reinstalling really fast, and they sacrifice some accuracy to do that
Editable installs allow for hot reloading of pure Python web apps running in a local web server, so they offer more than just faster build steps - they enable completely different development styles.
As a lowest common denominator, for single directory pure Python packages, an editable install only needs to do three things:
ensure the top level package is available on sys.path
generate an “installation” metadata directory (only without a valid RECORD file)
ensure the metadata directory can be found via sys.path (note that this is not required to be in the same path segment as the importable package)
Of those steps, it’s only the second one that necessarily requires backend involvement for pure Python projects (for other projects, there’s also a need to support in-tree builds so extension modules can be imported)
It’s not clear to me what’s being asked for 19.1.1. Do people want more to be done than what’s proposed in PR 6449 (namely just rolling back the 19.1 changes)? If so, that new code needs to be written, and it will be inherently riskier than simply rolling back to a previously released state.
Also, on a meta-level there are currently three different topics in the packaging channel related to editable installs that are happening, all of which are important and demanding attention, not to mention other important topics in the channel as well as PyCon US coming up, which also means travel plans for some. While the enthusiasm and interest is great, I think it’s important that we allow enough time for people to participate in these other conversations, especially for people already dedicating some of their volunteer time towards maintainer activities. Remember that people got burned out during the PEP 517 discussion, which lasted months, even without editable installs in the picture. There is also the negativity and mischaracterizations to deal with, which also have an effect.
Well, you can rest assured that this is not true, and I think even if it were considered a “minor/trivial feature” (or even an undesirable anti-pattern), I think most/all the pip maintainers would not deliberately break backwards compatibility with zero notice.
Oh, absolutely! From following the subsequent discussion here and elsewhere, my mind is completely at ease about the matter. I apologize if my post above contributed to the negativity of the situation; I was trying to relay my theory about your question as neutrally as I could.
It’s not clear to me what’s being asked for 19.1.1.
Given that I’m on the outside of all this, I don’t have a good feel for what the technical, PEP-adherence, future-planning, etc. aspects are of the decision.
But, from a PR perspective, two thoughts:
I think it’s crucial to restore pre-19.1 behavior ASAP. It’s been nearly a week since things blew up; IMO the last thing you want is pockets of a “they broke it and they don’t care” notion crystallizing out there. A simple reversion of the editable install changes of 19.1 would seem like the easiest way to do this, but I don’t know the codebase.
I would also recommend not putting any PEP517 warning messages into 19.1.1. I imagine most people have researched what happened, especially the loudest/most negative voices out there, and they now know that PEP517 may change things about editable installs in the future. An extra warning message about this is going to show up continuously in their consoles and CI outputs, and the last thing you want—for the near future, at least—is to add something that will keep reminding people of how upset this situation made them.
I think a simple reversion to the 19.0.3 behavior is good enough for the short term, and we can revisit the transition plan after PyCon (when hopefully we’ll have a better sense of what the trajectory is going to be).
I just checked and it looks like pip install -e .does install the build-system.requires requirements on 19.0.3, so I think the “fall back to PEP 518” logic is there already.
I don’t think this is solid reasoning for dropping the warning. I think it’s easier to just revert the change (and thus not raise the warning), but I do think a warning is justified in this situation. People who actually opt in to PEP 517 will probably be surprised to find that some of the features they may be counting on may not work correctly for editable installs.
Given that nobody in this thread (unless I miss some comments) disagrees that straight reversion is at least good enough, I would make the change and release 19.1.1 first, and put off discussion about warnings after that. PEP 517 has been around before 19.1, so it’s not we’re confusing many more people without the message. A 19.1.2 is always possible if we agree emitting a message is a good idea.
I’ve not been able to keep up with the discussion here (there’s a hole in my thumb right now ) so apologies if I repeat someone else’s points further down.
I think we can safely revert to the 19.0.3 behavior. I’m not sure how that would affect the code changes/test improvements made by Chris for that bit of code but some code debt for getting this change reverted and released should be fine.
Further, once that reverted behavior is released, we should probably close this thread to keep discussions easier to follow.
We all agree that we want pre-pip 19.1 behaviours (outcome wise) with editable installs. Aka editable installs should keep work as they did before. The nuance of if with warning/without warning/with flag/without a flag is probably a decision best made by pip maintainers. However, the decision should be made fast as this still breaks a lot of CIs and breaks workflows for a lot of developers. The longer we drag this nuance discussion out the more people will be hurt without any sane workaround. And as @uranusjr pointed out we can still do follow up changes.
pip is a critical piece of software for python package management. We should not need to wait a week to get a fix for a very popular stuff that causes breaking changes in peoples workflows, otherwise, end-users will start hating, even more, the even now not much liked, python packaging ecosystem.
Slightly OT, but let’s please try to keep the tone as understanding as everyone here are volunteers. So it isn’t that the pip maintainers “should” do something, it “would be appreciated if they could” get this fixed quickly.
) so apologies if I repeat someone else’s points further down.