This is mostly intended as an open ended discussion, that might lead to a meaningful user experience improvement.
Currently, if you try to do something like pip install amazing-package and the package requires a pyproject.toml-based build, there’s no mechanism for the build-backend to indicate to the frontend that it needs to present some output. This means that a user gets effectively no output when they have a long running build within pip, unless they pass -v.
Is this something that we’d want to improve, adding some way to show progress while a build is happening?
In principle, this would be good, and having some sort of callback that backends can use to say “I’m X% done” might be nice.
Do backends know how far they are through the process? If a backend calls a C compiler, it would rely on the compiler returning progress data.
The PEP 517 hooks get called in a subprocess, so simple callbacks wouldn’t work, we’d need some form of IPC.
This could be prototyped by having a backend support being passed something in config_options that acts as a progress reporting channel. Pip could then set that option to interact with the backend. It would be clumsy and not production-ready, but it would allow us to experiment with the idea before trying to standardise an interface.
My instincts say that it would be a major user interface improvement to have this, but almost impossible to actually achieve. I’d be happy to be proved wrong on the latter part of that statement, though
You mean, when the output is to a pipe not a console? No idea, I’ve not tried it. Probably about the same as a rich progress bar does. It might need some fixing up, but in my experience rich is pretty good at dealing with that sort of thing.
The Docker buildx UI does something similar. Under a normal run, it shows the last few lines of the log for each step. If a flag is passed, it no longer use clever terminal scrolling and outputs everything directly. That way you can see output but not be overwhelmed by it unless you ask for that.
One thing to consider is adding something like cargo’s cargo:warning=MESSAGE, which allows build scripts to show warnings to the user even if the output is otherwise hidden.
A lot of modern build tools (e.g. ninja and cargo) show single line progress bars, but I don’t know how easy it is to extract that information and pass it back to pip. For cargo at least it involves parsing json and counting yourself, while for ninja I don’t know about any structure progress output. Again I think writing a specially formatted message to stdout/stderr could be a nice way to communicate that.
I also love the docker buildx output, even though I’m not sure if that much should be shown by default or only with -v.
The fundamental question here is probably whether we want to amend the PEP 517 hooks to include progress reporting, or to simply try to do something with the existing stdout/stderr streams from the backend (which would be essentially just a QOL improvement for pip).
Personally, my impression is that most of the reporting people are interested in is compiler output, which we (neither the frontend nor the backend) have no control over, so I suspect there’s limits to what we can usefully do. Including a scrolling display in pip of the last N lines of backend output seems like the most feasible option that’s come up here so far. Would it help? I’m not sure…
Me neither. I’ve gone ahead and closed the open requests on pip’s issue tracker for this stuff, directing folks to here so that they can provide feedback here instead of on the issue tracker – I think anything we do here is bound to affect more than just pip (at the very least, some build backends will care about how pip’s output presentation changes).
For setuptools there’s no progress indication in stdout and its output tends to be very noisy, so there it won’t help. Many of the most build-heavy projects will be using a build system like CMake or Meson in the future (or now already), and those know the number of targets they are building and output a counter like [127/1500] target-name-being-compiled. So the last N lines (can be short, even N=1) would help quite a bit.
from rich.progress import Progress, SpinnerColumn, ProgressColumn
from time import sleep
self.cols = 
def render(self, task):
col = Col()
progress = Progress(SpinnerColumn(), col)
for n in progress.track(range(100)):
It displays the last 5 lines, scrolling, just as I think we’re discussing. And when redirected to a file, it shows the final display, but nothing else. It does the same when piped into something like less. Which may not be exactly what we’d want, but it’s certainly a reasonably friendly, and understandable behaviour.