J4yPr4tt
November 26, 2023, 10:32pm
1
Hoping that I can request some reviews and a merge for the following PR.pip install times.
pypa:main ← NeilBotelho:parallel-downloads
opened 07:12PM - 04 Nov 23 UTC
This PR adds parallel download support to `BatchDownloader` and adds a cli optio… n `--parallel-downloads <n>` to the install and download commands. If the option is not specified, or if set to 1, it doesn't change pip's behaviour or UI.
`BatchDownloader` is only called by [`_complete_partial_requirements`](https://github.com/pypa/pip/blob/fd77ebfc742de4d76ff976de22e86d116e0faad3/src/pip/_internal/operations/prepare.py#L445) after resolution is completed, so it wouldn't affect the resolution process as far as I can tell. It is used to download wheels where only metadata access was available for resolution. So as more packages adopt PEP 658 the number of wheels we would be able to download in parallel would also increase :)
### Note
Importantly, this PR does not add UI/progress bars for parallel downloads, it just logs the `Downloading <wheel-file-name>` message. I am working on a separate PR that adds a progress bar for parallel downloads. These 2 PR's can then be combined. I don't expect this PR to be merged until that PR is opened.
I've done it this way because the UI requires some discussion, due to limitations of how rich and even tqdm render parallel progress bars in Jupyter notebooks([see this issue](https://github.com/Textualize/rich/issues/3183)). Currently the best way I can think to circumvent this issue is to add a flag (something like --jupyter) to be used when downloading in parallel in jupyter notebooks that will disable the progress bar only for the parallel downloads. But this isn't ideal.
Also, if someone could tell me where I should ideally initiate this sort of discussion(discourse/IRC/discord etc.) I would appreciate it :)
3 Likes
ofek
November 27, 2023, 3:13pm
2
I am very excited about this! I thought about parallelism in the context of lock file implementations in Hatch and I’m happy I won’t have to implement that myself now after this gets merged
2 Likes
pradyunsg
November 27, 2023, 5:17pm
3
Speaking with my pip maintainer hat on, this is something that’s in my radar to review but I expect that I won’t have the bandwidth to do so until next year personally.
3 Likes
