I’d like to follow up on a recent issue I encountered in a project of mine (issue #644) and understand a few more details on how pip wheel works exactly.
For context: given a venv where I’ve installed a package in editable mode and all of its dependencies (for local development) chances are I’ve also installed other packages to toy and tinker and experiment. However, now I’d like to prune back all installed packages and remove those which aren’t direct or indirect deps of my editable package. Or phrased differently: I’d like to remove those packages that can’t be reached from my package’s dependency graph.
To do that, I came up with the following (actual source here):
Create a requirements.txt file of all currently installed packages.
Build wheels for all of them into a local wheelhouse folder.
Uninstall all packages using the requirements.txt file.
Install the package from the local wheelhouse folder and without a package index.
Most of the time that works quite well. Except — and hence the issue and this question — on some (rare) occasion step 2 seems to pull in more package versions of an already installed package to run the deps resolver. And that can go awry and take a long time.
Which raises two questions: Why would pip wheel need more dependency information (and download packages) than it already has available in my venv? And is there a better way to prune away non-dependent packages in a venv?
Regarding the point about pruning venvs, I can think of two better approaches:
Just delete the venv and recreate it with only the package(s) you want. Virtualenvs are meant to be disposable, after all.
Parse the output from pipdeptree’s JSON mode to determine which packages aren’t needed by the packages you want to keep, and then unininstall those. You don’t even need to install pipdeptree in the venv you’re analysing.
As for your primary question, I think it’d be hard to answer without at least the pip logs from an actual occurrence of the problem. One thing I can think of is that some package that doesn’t provide wheels for your platform has an sdist that builds nondeterministically.
Alternatively, depending on what exactly you’re seeing, it may be that pip’s resolver isn’t taking the exact versions in the requirements file fully into account from the start, and pip ends up backtracking through some package’s versions in order to find one that’s compatible with the packages processed so far, which may? (I’m not sure) involve building wheels for intermediate versions.
I think there is a fundamental misunderstanding going on here. pip wheel does not build a wheel from an installed package – there’s just no way you can do that, as a wheel must be built from the source code of the package (for example, your venv doesn’t contain the pyproject.toml of each package). If I understand your post correctly, you’re trying to do it in such a way that PyPI is never involved, which you think you’ve solved because step 4 does indeed not involve it – but step 2 very much does. And it does not use your venv in any way.
pip wheel only looks at the dependencies you give it (e.g., requirements file).
Pip does not know if the dependencies you pass needs dependencies themselves, so it must run the resolver to check. Adding --no-deps will skip this.
If your environment still have the pip cache for all wheels built/downloaded during your venv install, then no downloads are done. I’m not sure if you can disable downloads for offline builds.
— on some (rare) occasion step 2 seems to pull in more package versions of an already installed package to run the deps resolver. And that can go awry and take a long time.