Pip upgrade to 22.3 windows10

Packaging
10

Pip deliberately defaults to a user install if the system location is not writeable. This is because when we used to fail in this case, users would try sudo, which would work but break their distro managed installation. After many years of problems, we settled on the current approach as a solution.

In this case, it looks like the OP installed python as admin (presumably an all users installation) but then tried to upgrade pip as a non-admin user.

In general, I’d recommend sticking with the default (a per-user install) when installing python on windows, unless you have a specific reason for needing an all-users install, and you are a sysadmin (so you have a good understanding of the permission issues involved).

11

I think it’s implemented badly. By default, pip should prompt for consent before switching to a --user install. If users don’t want to be prompted, there can be a setting to either always allow or never allow automatic --user installs. Personally, I never want it. When I have Python installed at the system level, that’s not by accident. The only accident is that I forgot to use a shell with admin access when upgrading, for which pip punishes me by making me uninstall an automatic --user install that I don’t want.

12

:person_shrugging: Feel free to raise an issue on pip proposing a better behaviour. I personally have no particular stake in the current behaviour, it was designed mainly to handle the concerns of Unix users and distro maintainers, where the ergonomics may be different. But any further discussion is probably off-topic for this thread.

13

Personally, when I’m on Linux, I’d especially prefer pip giving an error or prompting for confirmation to fall back to a user install, though for slightly different reasons as @eryksun on Windows. I almost always work in venvs, so if I forget to activate one, I’d rather pip fail or prompt than silently install into --user if I don’t deliberately sudo it; at least one time this happened, it resulted in a very laborious cleanup and reinstall effort.

Honestly, if it were up to me, now that pip has a solver and dry run support, I’d much rather it act like most package managers (either by default or at least with an option) and display what it is going to do (i.e. --dry-run, preferably in a more readable format) before doing it and asking for confirmation. That would have avoided the overwhelming majority of issues (on different platforms and on both *conda and Python.org) I’d ran into where an environment got screwed up due to either user error (installing in the wrong env, wrong interpreter, with/without sudo, the wrong spec, or some other issue) or packaging issues, and is the single biggest feature I miss vs. conda (and I typically run pip with --dry-run first for that reason, despite it taking twice the time and effort). But that’s getting a bit out of scope here…

1 Like
14

I have some real world experience of this causing issues, I managed an Anaconda distribution at a large enterprise and for technical and cultural reasons it made sense to make the base install of Anaconda read-only and users who wanted Python with other packages could use venv or conda to create their own environments.

The problem was that pip --user install is fundamentally at odds with conda environments, and would often break even the base environment. A user would pip install something, pip would default to user install because the base directory is read only, and then all of a sudden their Spyder installation would no longer work.

The solution was fairly easy though, we updated the default pip.ini we distributed with the Anaconda install to include a user=false flag (forget the specific config). And it was my opinion that because we controlled the environment and had specific needs it wasn’t worth bringing up or arguing it should be default. But thought people might be interested in a real world example.

2 Likes
15

I haven’t personally seen that too much, but our other developers have seen that happen, so we specifically mention it and how to fix it in some of our tutorial/help videos.

Also, just FYI, it is for reasons like this that we’ve been recommending our standalone installers to avoid the many issues with users doing all sorts of ill advised things like this and breaking Spyder. We also advising that if they do install Spyder with Anaconda, they use a separate isolated env instead of base. In addition to our stable pynsist and py2app based ones, we now have new experimental conda-based installers for all three major platforms, which will allow for auto-updating, installing Spyder plugins and creating/modifying user environments all in a controlled manner, combining the benefits of both a conda and a standalone install while opening the door to adding package, environment and plugin management functionality right within Spyder.

16 
pip config set global.require-virtualenv true

See also

2 Likes
17

See also: It should be possible for a Python environment to disable the "user site" functionality without patching the stdlib · Issue #99312 · python/cpython · GitHub

18

Neat!
I’ve been preventing accidental --user installs by making my ~/.local/lib/python3* read-only :sweat_smile:

1 Like
19

Thanks for the advise but this wasn’t really about Spyder, as Spyder breaking was a symptom of pip --user installs being fundamentally incompatible with using conda to manage your Python environments. It could have happened with any package installed with conda, Spyder was just more likely to break due to have more dependencies.

The problem is that you may only install packages via conda in one conda environment but in another conda environment you might be using conda to just bootstrap Python and use pip install to install everything else. If you use pip --user install in the latter environment those packages are still picked up by the former environment.

Anaconda was the main way that our users (which had a very wide range of technical backgrounds) got access to Python, and they weren’t allowed to install software manually, so some users were always going to use Spyder via Anaconda to do their work. So I tried hard to make the default behavior as safe as possible, although I never made the configs read only so if a user wanted to force break their own environment I wasn’t going to stop them.

Well to at least stop the automatic user behavior you can also do what I described in my previous post which if I remember rightly was set like this:

pip config set global.user false
2 Likes
20

Ok. I appreciate all the feedback from everyone.

If I uninstall anaconda, would this solve my problem? Or best to uinistall BOTH CONDA & VANILLA PYTHON?

Thanks for the help

21

It seems to me like the problems with --user have very little to do with conda specifically. It’s equally problematic, and should not be used, when you have multiple virtualenvs. I think --user came from a world where users would always have a single Python (system) install, and not multiple environments. Now that the latter is the recommend way of operating, and most users understand that, --user is more harmful than helpful. It should probably be disabled by default, with an understandable error message for users that run into permissions errors.

If you remove all your installs, the problem will go away. But that’s a pretty destructive action. Instead, I think the one thing you have to understand is where Python and Pip are coming from. If this is the Pip from a Conda environment, you should upgrade it with conda (so conda upgrade pip rather than python -m pip install --upgrade pip).

3 Likes
22

I don’t think this is correct as my understanding is that Python in an activated virtual env ignores user installed packages.

I just tried:

python -m venv .venv
pip install attrs --user
source .venv/bin/activate
python -c “import attrs”

And I got an import error. Whereas when I do the same with a conda environment I do not get an import error.

p.s this site is basically unusable to write a long post on mobile, sorry for the edits the UI keeps making random choices about what I want it to do.

1 Like
23

@rgommers @notatallshaw
Ok, thanks all. I uninstalled BOTH my vanilla AND conda installations.

I’ve installed the vanilla python. I think that’s good enough for now. But I’d like to install CONDA in the near future. Any tips on how to prevent the pip permissions disaster?

Also, I had py 3.9 installed and now the python site has 3.11? Not sure why the version went DOWN? I assume I can get the py 3.9 and pip install from 3.11?

Thanks again.

24

It’s not a decrease. Like most software version numbers, they are not to be read as decimals, but rather each “element” (between dots) is read as an integer. So 3.11 is two minor versions newer than 3.9 (eleven being two more than nine) and is correctly read as “three dot eleven” rather than “three dot one one”.

1 Like
25

I interpreted what @rgommers meant with that was that regardless of environment type (conda or venv), the main problem with --user by default is the same, making it easier to install things into the base Python installation when an environment is not properly activated (either due to user error or another issue). But perhaps he meant something different? I believe there might be an issue with --user when using a pip-installed (i.e. unpatched) pip inside a conda environment, but that wasn’t what I took Ralf to be referring to there.

Also, to note, both system and user-installed packages will be ignored when a venv is activated so long as --system-site-packages is not passed when creating the venv (and I would assume, and believe I saw before, but have not tested again to confirm, that the same is true of user packages when that option is passed in addition to system packages).

The ultimate problem likely didn’t have anything to do with permissions, but rather due to basic user errors and/or simple confusion (I cannot be sure specifically, since there wasn’t enough information provided about either your environment or what specifically you did).

To avoid problems in the future, I suggest first running, from your regular system command prompt (not Anaconda prompt):

python.exe -m pip install --upgrade pip
pip config set global.require-virtualenv true

This will ensure you don’t accidentally install things into your globally-installed Python directly.

Then, when using Python.org Python (i.e. from a non-Anaconda prompt), use venv virtual environments to install your packages instead. This will ensure that if something goes wrong, you won’t need to reinstall Python, just recreate the environment, will avoid any permissions issues, and will ensure that you won’t accidentally install packages with pip either with or without Anaconda anywhere that your *conda Python could pick them up. Also, always use python -m pip install --upgrade pip to upgrade pip itself in Python.org installed Python.

In general, when using any Python install:

  • Never use setup.py <anything> or easy_install <anything>. These are long-obsolete, legacy install methods that are likely to break things.
  • Always triple-check that the venv/conda environment you expect is activated before installing anything. It should show in parenthesis in/around your command prompt and you can also use where python to determine the path to the currently activated Python environment.
  • Ensure the install is doing what you expect before committing to it, by looking at the conda install output or the output of pip install --dry-run. Are a bunch of packages being installed that you didn’t ask for, or you know are already installed, or at a different path than you expect? Does something else look off? This may be a little tough at first, but as you gain more experience, you’ll learn what to expect.
  • If you aren’t sure about something, before doing anything, ask someone you trust that is an experienced Python/Conda user to give it a second look. If you don’t have someone, find one :slight_smile:

When using Conda (e.g. via Anaconda prompt):

  • Either download minforge instead of Miniconda (or Anaconda), or run the following to set your default channel to conda-forge, for a wider and more up to date package selection and avoiding having to mix channels or use pip (which both cause a multitude of problems):

    conda config --add channels conda-forge
conda config --set channel_priority strict

  • Never touch the base conda environment. Always create new conda environments and install specific packages you need in them, for the same reasons as pip (namely, if something goes wrong, just delete the env and recreate it, no need to reinstall, and it greatly decreases the chance of package conflicts and makes it easier to see what you have installed).

  • Avoid using pip to install anything in a Conda environment, always use conda unless there are no conda-forge packages available. If you do use pip, I suggest just creating a fresh environment with conda create -n pip-env python, activating it and then installing everything with pip from the get-go (except for pip itself; always manage that with conda). Especially avoid installing/upgrading pip using pip itself in a conda environment.

While nominally aimed at Spyder users, our Don’t mix pip and conda video walks you through most of the latter steps and provides more guidance on fixing and avoiding these situations.

2 Likes
26

@CAM-Gerlach - I owe you a beer or a coffee :grin:

This is very valuable information and tips. I appreciate it. I took your advice about setting the global requirement to true. Hopefully this will prevent any future mishaps.

I really need to do a deep dive into virtual environments. I’m not a professional coder (but I wannabe)! I code at home with some of my professional data(I can take the data home and work on it, but not in a production environment at work.

I’m very leery of installing conda again. It seems like it’s more trouble than its worth. But I’ll get there, I think the COURSERA classes I’ve been looking at have some CONDA requirements.

Again, thanks.

27

Just to give credit where credit was due, that was @pradyunsg 's advice to me above, I just stole shared it with you :slight_smile:

Really, the parts you actually need to know as a user are pretty straightforward. You can think of a Python venv as an isolated set of Python packages, where packages you install/upgrade/remove inside the environment don’t affect anything outside it, and likewise those you install/upgrade/remove outside of it don’t affect anything inside it. I.e., anything you do with pip in the environment, stays in the environment.

You can have as many as you want and can create them wherever you want; each of them lives in a specific directory that you can name whatever and put wherever you want, but for basic usage, you usually create it inside your top-level directory for a project (that contains your scripts, modules, data etc that you’re working on). So, to create a venv, cd to that directory and run

python -m venv <env_name>

Replace <env_name> with the name of the directory you want Python will create inside your project dir to contain the venv; they are often named env, .env, venv or .venv, though you can also give the directory a more descriptive name for your project, so you can tell it apart from other environments when its activated (since that’s the name shown in the command prompt).

You only need to create it once, but whenever you open a new command prompt and are ready to start working in that environment (running python, installing packages with pip, etc), you need to activate it. On Windows, you do so by running, while in your project directory,

<env_name>/Scripts/Activate.bat

You can confirm its activated by entering where python, and you’ll see the path to the Python inside the <env_name> directory instead of the global install.

That’s it; now any time you run python, pip, etc. in the same command prompt window, they will run inside the virtual environment. To deactivate the environment, simply type deactivate.

Hey, I’m not a professional programmer either…I just play one on the internet.

1 Like
28

It also comes from a world without pipx, where installing reused tools into --user was a thing when it doesn’t cause issues with the code you were running.

1 Like
29

To note, this doesn’t work with conda environments (or other types of non virtualenv/venv environment management), so it doesn’t actually work for my particular situation after all, and PEP 668 is probably the way forward to address that…which, going back and reading the linked issues and discussions, it seems you all already figured out :slight_smile:

1 Like