That does sound like an important technical problem! Is it specificallythe number of files that’s an issue? Because if so, then it sounds like we should look into using zipimport for ensurepip, and any future ensureX libraries.
Installing packages in zip form is messy in the general case. But if we only worry about ensureX packages, maybe it’s not so bad. For example, for ensurepip we would install:
site-packages/
…/pip.zip
…/pip.pth
…/pip-VERSION.dist-info
And the .dist-info dir would have a few more files, including a RECORD that just lists pip.zip and pip.pth. If you later upgrade pip, it reverts back to the normal layout.
For ensuremultiprocessing, it would be similar, but adding _multiprocessing.pyd.
The trick being: since we don’t have to support arbitrary packages, we can be sure that no one is playing games with file. And we’re just trying to reduce the number of files rather than necessarily put everything in a single file like eggs do, so it’s ok to keep extensions outside of the zip.
I guess we should actually do this right now for pip (8 seconds is a lot!), but it would become even more urgent if we start moving more stuff into site-packages.
Another possibility: add a default-site-packages directory to the standard sys.path, where all the default-installed packages live. It goes after site-packages. If you upgrade a package, the new version goes in site-packages as usual, so it shadows the version in site-packages. For venvs, the parent environment’s default-site-packages gets unconditionally added to the child environment’s path, not copied, and not affected by --system-site-packages. So child venvs revert back to the original version of the default packages, which seems reasonable, it’s free, and then you can upgrade them in the venv if you want.
I guess the problem with this is that if you try to uninstall in the venv, then you instead just unshadow the original version … Maybe we’d want some kind of tombstone mechanism to prevent that?