What? Setting up a dev environment is too hard now?
At least you’ve finally found the way to push me into writing my own backend that’ll do what I need 
Right. This was always my impression too. Pip remains the frontend for installing published packages, but doesn’t automatically become part of the dev’s inner loop (as we call it in Developer Division at work). A better front end for that is the IDE, or the test runner, etc.
Trying to stuff all scenarios into pip is just not going to end well.
pip only has to be involved to the extent that it does not actively make it impossible to do incremental builds. If we add a cache directory and pip always passes it a temporary directory with no option to say, “Please persist this somewhere”, then it’s not like you can leave it up to the backend, particularly if the cache directory PEP says that backends must put any build artifacts in the cache directory.
Setting up separate isolated build and dev environments is definitely a pain and basically no one does it. It also relies on a lot of state, which is why we are flooded with questions from people who can’t execute bdist_wheel or where some random build-time plugin isn’t installed or fundamentally changes something about the installed package. We took the time to standardize a way to declare build-time dependencies separate from runtime dependencies specifically to solve these kinds of problems and we cannot solve them from a setup.py invocation, so we would need some sort of front-end.
I’m also hoping that in the long run the majority of people won’t even have a setup.py in the same way that the majority of cargo users don’t have a build.rs — we should be able to handle the 99% case with declarative metadata alone. That means there’s no such thing as “invoke setup.py bdist_ext” because there’s no setup.py.
Sure. Though presumably what you’re looking for here is a front-end, even if it it’s one that can invoke your build backend in a non-standard way.
I don’t think we’ve ever supported incremental builds outside of -e except in one edge case which was entirely by accident (if you built via setup.py first, those built bits would get copied along with everything else into a temporary location and the backend would presumably use them but that was never supported it just happened accidentally).
Personally I think a lot of what we’ve done over the past so many years is to detangle things so that pip doesn’t have to be the way you install stuff. So that if some particular use case needs a tool that makes different decisions, they are free to do so. A lot of that came from our experience of people trying to make pip (and setuptools) be everything to everyone because that was the only way things worked. I think the better future here is if someone wants something different then what pip provides, it’s easy for them to use another tool that does (or writes one! this is also why we’ve been trying to extract stuff out of pip into reusable libraries to make this all easier).
Yep, that’s why I’m arguing for the behaviour where the frontend can pass nothing and have a safe in-place build without losing the ability to do incremental builds.
Let me bluntly ask the question that’s behind my concerns. I have the feeling that we’re both making assumptions here, and I want to get the facts out in the open.
Is there any intention for setuptools to change its default behaviour to make in-place builds safe, in the sense of not causing the various issues that pip’s users encountered when pip switched to in-place builds?
If that’s going to happen anyway, then there won’t really be any need for pip to pass a temporary cache directory. The only reason we’d do that is as a workaround for setuptools’ default behaviour to enable us to reinstate in-place builds1. And I’d much rather not have to work around setuptools’ behaviour just to do that. If nothing else, it means that pip still has code in it to handle implementation details of setuptools2 (and worse still, in a way that goes directly against the principle in PEP 517 that frontends should just call the backend hooks and not try to manage how the backend delivers the results being asked for 
)
While I’m being blunt, I’ll further state that as a pip developer, I don’t actually have much personal interest in whether pip supports incremental builds or not. I’m not in the camp that feels that building projects should be part of pip’s core functionality - we need to do basic builds so that we can install projects that don’t supply wheels, but I would hope that for more complex builds, projects would use the backend directly, or use specialised tools3. It’s a shame those tools don’t exist yet, but PEP 517 was supposed to make writing them easier, so I’d like to think that it’s just a matter of time (and resources, of course…)
With my interoperability standards hat on, of course, I’m strongly in favour of making the standards capable of supporting tools that want to handle incremental builds - as you say, not having that ability is deterring projects from adopting them.
1 There is of course the other approach that pip could take - to implement in-place builds, and direct people affected by this issue, to raise it with setuptools directly. But that doesn’t feel like a very constructive approach, so I’d rather find a better solution.
2 After all, other backends don’t - to my knowledge - have any problem doing an in-place build.
3 And ship wheels, so that users don’t need access to those tools, of course.
It’s very unclear what you mean by “safe in-place build” here, which is what everything hinges on. What is unsafe about the current behavior? The fact that it puts build artifacts in the repository root?
I don’t see that there’s any reasonable way to have incremental builds without a persistent cache directory and without altering the repository root. Where would we put the build artifacts? We can’t persist temporary directories between builds, and if we put the build artifacts anywhere other than a temporary directory or the current directory, it would create invisible state that end-users wouldn’t know about or have any way to clear.
The build directory setuptools uses does not encode enough information to not produce incorrect results when run twice from the same build cache in every case. For instance, if you have two Python 2.7 builds one with wide unicode one without and run setup.py build twice, one with each, the build cache will get reused between them, but the second build will be wrong, because it isn’t ABI compatible.
Feels to me like a nice setuptools bug 
Maybe. But it’s not really unique to setuptools and not perfectly solvable either. AFAIK nobody has ever implemented incremental builds for C code in a way that didn’t occasionally require manually purging the cache from time to time for one reason or another. Even within the Python ecosystem when I tested scikit-build, it just straight up gave an incomprehensible error whenever you tried to run it twice with the same build directory.
So while you’re correct in that in this particular instance it’s a problem with setuptools, the over all class of problem is not, and those people running into it are almost certainly going to blame pip, which we will then need to expend effort to educate and redirect.
The only way pip can solve this is to pass in different cache directories per python interpreter, and the backend to adhere to that because here you point at setuptools as the culprit:
But then you just said that setuptools cannot be expected to solve this, and this sentence puts the ball very much back in setuptools court.
We can also solve it by just not caching anything between builds, which I think is both Paul and I’s preferred option.
But that would be a setuptools feature request, and one that would will be very unpopular for source tree builds I’d guess. 
I get less problem for the build frontend but is really what’s best for the end user? Is what I think @pganssle is getting at, and I agree with him on that.
Incremental builds are not inherently best for the user. They are faster on rebuilds, but they also introduce subtle edge cases where things will break for users, particularly in strange, hard to debug ways. I could just as easily say that what’s best for end users is that the output of the build takes a little longer, but more consistently produces the correct artifact.
I tend to agree on the little. But I define little in ~1s. In practice on c-extension rebuild little may mean a lot more.
Most C extensions in the Python ecosystem are relatively quick to build. The ones where it takes an appreciable amount of time are the exception not the rule.
I’m confused. I don’t think I said that, but if I did, I’ll stick with “I’m confused”, because I guess I must be 
From a front end point of view, I still think that pip should just be calling build_wheel and getting a wheel back, and that’s it. Pip’s users have pointed out that we can’t just do that in the case of setuptools, and so we currently copy the build tree. We’d like to not do that (because copying the build tree triggers a whole load of other issues our users care about). But if setuptools doesn’t work in such a way that we can call build_wheel multiple times on the source directory the user supplies, then so be it. PEP 517 isn’t explicit on the backend’s responsibilities so we have to live with that.
The current proposal may or may not be linked to that issue. It’s designed to help with incremental builds, at least for setuptools (I don’t know if any other backends would benefit). I’m fine with that. The proposal doesn’t say anything about how front ends would use the interface, so to that extent “what pip will do” is irrelevant here. It’s possible that pip will choose to use the interface in a way that doesn’t make incremental builds any easier with pip (maybe as a result of trying to use it as a workaround for the issue of copying the source tree). If that’s a problem, then the proposal needs to make more demands on how front ends use it. It’s the converse of the problem we have with build_wheel not specifying enough restrictions on the back end, I guess. And I suspect we’ll hit the same problem, that we won’t be able to get consensus on over-constraining front ends.
I’m not sure this aspect of the discussion is particularly productive any more, though, so I suggest we let it drop.
I was referring to @dstufft under:
Ah, OK. Yes, it’s quite possible @dstufft knows more about this than I do 
(But I suspect there’s still a middle ground where setuptools could make changes to reduce the chance of issues, if not eliminate them totally).
Anyways I’ll write up a PEP by next Monday 
On the need to add this cache interface and we can go from there.
Can you link the PEP?
I just wanted to give a heads up that I actually took the effort the write up the PEP, and I will post it for public review later this week once I get the first batch of reviews on it.