Pip upgrade to 22.3 windows10

Packaging
21

It seems to me like the problems with --user have very little to do with conda specifically. It’s equally problematic, and should not be used, when you have multiple virtualenvs. I think --user came from a world where users would always have a single Python (system) install, and not multiple environments. Now that the latter is the recommend way of operating, and most users understand that, --user is more harmful than helpful. It should probably be disabled by default, with an understandable error message for users that run into permissions errors.

If you remove all your installs, the problem will go away. But that’s a pretty destructive action. Instead, I think the one thing you have to understand is where Python and Pip are coming from. If this is the Pip from a Conda environment, you should upgrade it with conda (so conda upgrade pip rather than python -m pip install --upgrade pip).

3 Likes
22

I don’t think this is correct as my understanding is that Python in an activated virtual env ignores user installed packages.

I just tried:

python -m venv .venv
pip install attrs --user
source .venv/bin/activate
python -c “import attrs”

And I got an import error. Whereas when I do the same with a conda environment I do not get an import error.

p.s this site is basically unusable to write a long post on mobile, sorry for the edits the UI keeps making random choices about what I want it to do.

1 Like
23

@rgommers @notatallshaw
Ok, thanks all. I uninstalled BOTH my vanilla AND conda installations.

I’ve installed the vanilla python. I think that’s good enough for now. But I’d like to install CONDA in the near future. Any tips on how to prevent the pip permissions disaster?

Also, I had py 3.9 installed and now the python site has 3.11? Not sure why the version went DOWN? I assume I can get the py 3.9 and pip install from 3.11?

Thanks again.

24

It’s not a decrease. Like most software version numbers, they are not to be read as decimals, but rather each “element” (between dots) is read as an integer. So 3.11 is two minor versions newer than 3.9 (eleven being two more than nine) and is correctly read as “three dot eleven” rather than “three dot one one”.

1 Like
25

I interpreted what @rgommers meant with that was that regardless of environment type (conda or venv), the main problem with --user by default is the same, making it easier to install things into the base Python installation when an environment is not properly activated (either due to user error or another issue). But perhaps he meant something different? I believe there might be an issue with --user when using a pip-installed (i.e. unpatched) pip inside a conda environment, but that wasn’t what I took Ralf to be referring to there.

Also, to note, both system and user-installed packages will be ignored when a venv is activated so long as --system-site-packages is not passed when creating the venv (and I would assume, and believe I saw before, but have not tested again to confirm, that the same is true of user packages when that option is passed in addition to system packages).

The ultimate problem likely didn’t have anything to do with permissions, but rather due to basic user errors and/or simple confusion (I cannot be sure specifically, since there wasn’t enough information provided about either your environment or what specifically you did).

To avoid problems in the future, I suggest first running, from your regular system command prompt (not Anaconda prompt):

python.exe -m pip install --upgrade pip
pip config set global.require-virtualenv true

This will ensure you don’t accidentally install things into your globally-installed Python directly.

Then, when using Python.org Python (i.e. from a non-Anaconda prompt), use venv virtual environments to install your packages instead. This will ensure that if something goes wrong, you won’t need to reinstall Python, just recreate the environment, will avoid any permissions issues, and will ensure that you won’t accidentally install packages with pip either with or without Anaconda anywhere that your *conda Python could pick them up. Also, always use python -m pip install --upgrade pip to upgrade pip itself in Python.org installed Python.

In general, when using any Python install:

  • Never use setup.py <anything> or easy_install <anything>. These are long-obsolete, legacy install methods that are likely to break things.
  • Always triple-check that the venv/conda environment you expect is activated before installing anything. It should show in parenthesis in/around your command prompt and you can also use where python to determine the path to the currently activated Python environment.
  • Ensure the install is doing what you expect before committing to it, by looking at the conda install output or the output of pip install --dry-run. Are a bunch of packages being installed that you didn’t ask for, or you know are already installed, or at a different path than you expect? Does something else look off? This may be a little tough at first, but as you gain more experience, you’ll learn what to expect.
  • If you aren’t sure about something, before doing anything, ask someone you trust that is an experienced Python/Conda user to give it a second look. If you don’t have someone, find one :slight_smile:

When using Conda (e.g. via Anaconda prompt):

  • Either download minforge instead of Miniconda (or Anaconda), or run the following to set your default channel to conda-forge, for a wider and more up to date package selection and avoiding having to mix channels or use pip (which both cause a multitude of problems):

    conda config --add channels conda-forge
conda config --set channel_priority strict

  • Never touch the base conda environment. Always create new conda environments and install specific packages you need in them, for the same reasons as pip (namely, if something goes wrong, just delete the env and recreate it, no need to reinstall, and it greatly decreases the chance of package conflicts and makes it easier to see what you have installed).

  • Avoid using pip to install anything in a Conda environment, always use conda unless there are no conda-forge packages available. If you do use pip, I suggest just creating a fresh environment with conda create -n pip-env python, activating it and then installing everything with pip from the get-go (except for pip itself; always manage that with conda). Especially avoid installing/upgrading pip using pip itself in a conda environment.

While nominally aimed at Spyder users, our Don’t mix pip and conda video walks you through most of the latter steps and provides more guidance on fixing and avoiding these situations.

2 Likes
26

@CAM-Gerlach - I owe you a beer or a coffee :grin:

This is very valuable information and tips. I appreciate it. I took your advice about setting the global requirement to true. Hopefully this will prevent any future mishaps.

I really need to do a deep dive into virtual environments. I’m not a professional coder (but I wannabe)! I code at home with some of my professional data(I can take the data home and work on it, but not in a production environment at work.

I’m very leery of installing conda again. It seems like it’s more trouble than its worth. But I’ll get there, I think the COURSERA classes I’ve been looking at have some CONDA requirements.

Again, thanks.

27

Just to give credit where credit was due, that was @pradyunsg 's advice to me above, I just stole shared it with you :slight_smile:

Really, the parts you actually need to know as a user are pretty straightforward. You can think of a Python venv as an isolated set of Python packages, where packages you install/upgrade/remove inside the environment don’t affect anything outside it, and likewise those you install/upgrade/remove outside of it don’t affect anything inside it. I.e., anything you do with pip in the environment, stays in the environment.

You can have as many as you want and can create them wherever you want; each of them lives in a specific directory that you can name whatever and put wherever you want, but for basic usage, you usually create it inside your top-level directory for a project (that contains your scripts, modules, data etc that you’re working on). So, to create a venv, cd to that directory and run

python -m venv <env_name>

Replace <env_name> with the name of the directory you want Python will create inside your project dir to contain the venv; they are often named env, .env, venv or .venv, though you can also give the directory a more descriptive name for your project, so you can tell it apart from other environments when its activated (since that’s the name shown in the command prompt).

You only need to create it once, but whenever you open a new command prompt and are ready to start working in that environment (running python, installing packages with pip, etc), you need to activate it. On Windows, you do so by running, while in your project directory,

<env_name>/Scripts/Activate.bat

You can confirm its activated by entering where python, and you’ll see the path to the Python inside the <env_name> directory instead of the global install.

That’s it; now any time you run python, pip, etc. in the same command prompt window, they will run inside the virtual environment. To deactivate the environment, simply type deactivate.

Hey, I’m not a professional programmer either…I just play one on the internet.

1 Like
28

It also comes from a world without pipx, where installing reused tools into --user was a thing when it doesn’t cause issues with the code you were running.

1 Like
29

To note, this doesn’t work with conda environments (or other types of non virtualenv/venv environment management), so it doesn’t actually work for my particular situation after all, and PEP 668 is probably the way forward to address that…which, going back and reading the linked issues and discussions, it seems you all already figured out :slight_smile:

1 Like
30

There is a lot of discussion and maybe some of it went off the topic. I found a fix here. The solution is as follows:

> python -m pip uninstall pip
> python -m ensurepip
> python -m pip install -U pip

Havent tested this in conda, but it worked for python 3.7.6 (64 bit), running from a non-admin cmd prompt.