PEP 668: Marking Python base environments as "externally managed"

It’s worth mentioning Spack I think. It works in a very similar way to Conda.

(Conda [3] is a bit of a special case, as the conda command can install much more than just Python packages, making it more like a distro package manager in some senses.

I don’t actually think about it like this at all. The conda base environment is like a Linux distro, and you should never mess with it as a user. However, user-created conda environments are conceptually similar to Python’s virtual environments, just more powerful. They do environment activation, and have a “one environment for one development task” philosophy. The environment activation does a bunch of magic, that’s why you can’t layer a virtualenv on top of an activated conda env. And a major part of the reason why you need environments is because the package manager/repository provides multiple versions of every package - just like PyPI, and unlike Linux distros and Homebrew.

The above all applies to Spack environments too I believe.

This makes the “using a virtual environment is best-practice” advice more generalizable. We have two kinds of Python installs:

  1. “base” or “system”: macOS system, Linux distros, Homebrew, Python.org installers, Python built from source, conda base env Python, etc.
  2. “dev environment”: virtual/conda/spack environments

The advice for all of these is: never mess with (1), always work in (2).

The NixOS Python docs page linked from the PEP is kind of illustrative in this regard: it has a python-with-my-packages where everything is in the “base” OS packages and that’s immutable so you can’t pip install anything, and then further down it explains how to get virtual/conda/micromamba/mach-nix environments - all user-level dev envs where you can install whatever you want.

3 Likes