Unexpected interaction between pip install --user and virtualenv

It seems that when running pip install . --user on a project, it installs all dependencies completely outside the virtualenv. I noticed this while following the installation instructions of keras-retinanet:

1. Clone this repository.
2. In the repository, execute pip install . --user. Note that due to inconsistencies with how tensorflow should be installed, this package does not define a dependency on tensorflow as it will try to install that (which at least on Arch Linux results in an incorrect installation). Please make sure tensorflow is installed as per your systems requirements.

After doing these steps, I added the virtualenv path to my Pycharm IDE project preferences, but it didn’t recognize any of the dependencies. When running pip list inside my virtualenv, none of the dependencies showed up, though I do see them in /Users/<username>/Library/Python/3.7/lib/python/site-packages.

I tried to “undo” this via pip uninstall . --user , but that gave an error saying that the uninstall subcommand didn’t recognize the --user flag. How can I clean this up?

Should trying to pip install --user while in a virtualenv just throw an error? It seems misleading in that it appears to work fine, but as far as I can tell the virtualenv is completely ignored, and everything escapes into the global state.

Don’t specify --user if you don’t want to install into site.USER_BASE.

--user is the default when running as non-root outside of a virtualenv.

You can run python -m site to print sys.path, site.USER_BASE, site.USER_SITE, and site.ENABLE_USER_SITE.

https://pip.pypa.io/en/stable/user_guide/#user-installs :

pip install --user follows four rules:

  1. When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
  2. When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv).
  3. pip will not perform a --user install in a --no-site-packages virtualenv (i.e. the default kind of virtualenv), due to the user site not being on the python path. The installation would be pointless.
  4. In a --system-site-packages virtualenv, pip will not install a package that conflicts with a package in the virtualenv site-packages. The --user installation would lack sys.path precedence and be pointless.

The uninstall behavior is a bit wonky: IMHO, pip uninstall --user from within a virtualenv should work; it shouldn’t be necessary to deactivate first (before running pip uninstall (without specifying --user)). But --user is the default when running as non-root outside of a virtualenv.

FWIW, pipx is generally a better way to install things user-globally than installing everything into USER_SITE because pipx creates per-install virtualenvs for console_scripts that get symlinked into ~/.local/bin, too.

1 Like

What version of pip are you using? pip hard-errors when you specify --user in a virtual environment now.

2 Likes

It looks like I’m on pip version 19.2.3, and there’s a newer version (20.2.3) that I will upgrade to. Thanks again for the help @uranusjr and @westurner