PEP 668: Marking Python base environments as "externally managed"

Can we get this confirmed? Also, I use make install rather than altinstall, but it’s still separate from the system Python (/usr/local/bin/python3 separate from /usr/bin/python3).

This is correct. The canonical CPython source is actually entirely unchanged by this PEP, so make install is also unchanged. The PEP only recommends Python redistributors to add the marker file to the compiled Python.

Cool, thanks. Will this then remove the scary warning on running pip as root, replacing it instead with the new warning about running pip against the system Python? I’ve always just ignored that warning since it’s my own Python that I’m installing into.

(Though I am also in the “expecting breakage” camp, happily running unreleased Python versions and then seeing magnificent segfaults and gorgeous desynchronization errors as things get updated and the packages haven’t been recompiled. It’s good fun.)

Yes, replacing the warning is one of the main goals. It can already be disabled now: https://github.com/pypa/pip/pull/10990

I’m not sure when (or if) it will be entirely dropped; feel free to open an issue on GitHub to kick off the discussion.

1 Like

Excellent, thanks for the clarity!

Now, that my account got finally unblocked, I’d be happy to correct the inaccuracies here.

First, my prediction (not “supposition”) is motivated by the contents of the overwhelming majority of self-help pages returned by a simple Google search on the topic. I included a sample link in my first post above. And,while I did repeat my prediction a few times, I always decorated with “predictably”, i.e. did not present it “as a fact”.

Second, I’m not at all “content to risk mucking up your system python”; BUT, I do have a few simple rules of hygiene that completely eliminate that risk, which makes me a collateral victim of this PEP. That’s also what “gets under my skin”, as @fungi so aptly put it. BUT, I can see the value in stopping people without such self-imposed rules from shooting themselves in the foot; especially from the perspective of increased support burden. I just wish that this had been done without forcing the same remedy down the throat of those who don’t actually need it… (see more on that in my next post below)

I can also sympathize with the idea that a big scary flag is likely to make some people think twice before they copy and paste the solution from StackOverflow or some other trendy page. I’d still predict a good 50% rate (or higher) of people who will simply turn off the intended protection, but I obviously cannot know, and on that point you are exactly correct. Time will tell how effectively this PEP works its magic.

I know. I’m glad we build our Python images from source at work.

Wrong; although one could argue that packages installed with PIP_USER set are “sort of” like a virtual environment, they really aren’t. One might also argue that those packages are installed for “the system Python”, which is partially correct, since they are, in fact, intended to be used in conjunction with the same Python executable and some of the same system-provided packages that this so-called “system Python” uses, but they do not end up in the system’s site-packages.

Says who? That sounds like a fairly arbitrary distinction to me. If I’m using the same Python version anyway, why would I need to create a virtual environment and duplicate stuff unnecessarily.

The problem is that the distro is not only preventing me from “installing into the system Python”, it also stops me from installing packages into my user site-packages. THAT is what all this noise is about.

My simple rules of hygiene I alluded to in my previous post above are as follows:

  • unprivileged users have PIP_USER set in their shell’s start-up configuration
  • system commands are only ever executed as a privileged user

These guarantee that my user installs NEVER pollute system packages, or break system commands built in Python for a privileged user. Yes, running some system-provided Python-based commands as a non-privileged user might break, but there I’d argue that those were not intended to be used by them; in fact, that’s one motivation behind sudo.

So, from my perspective, I wouldn’t complain if that scary flag was only needed when privileged users attempted to run pip install (which I’d never do; so, had that approach been taken, I wouldn’t even have realized this PEP existed). My issue is with the unpleasantries of this PEP being forced on those (like myself) who prefer user installs over virtual environments.

AND, make no mistake, this preference is NOT based on the storage overhead, although that’s a clear waste too, no matter how small, but the convenience of a friction-free user experience; but, I know that’s very subjective.

One the on hand, I’m stunned that people would do that. On the other hand, I’m not really surprised, since people do all kinds of weird stuff they shouldn’t. BTW, this is exactly why I predicted that this PEP’s approach to protection is unlikely to be effective.

Well, I like to think that my foolproof rules of hygiene outlined above are more akin to using provably correct loop invariants that prevent bound violations; but I’d be the first to admit that as a beginner in C a few decades ago, I probably did much worse than just letting pointers step out of bound… :wink:

Presumably, you realise that:

  1. Setting PIP_USER will cause attempts to do pip install in a virtual environment will break, as --user is not allowed in a virtual environment. While you may not use virtual environments yourself, this may cause issues if you use tools that use them “under the hood”.
  2. Pip has for some releases now defaulted to doing a user install if the site-packages is not writeable. So you don’t even need PIP_USER in the first place.

None of this alters the fact that PEP 668 causes a problem for you. That’s because it’s intended to give distributions a way to protect against issues which you don’t accept are problems. That’s your choice, and similarly you have the option to remove the distribution-provided EXTERNALLY_MANAGED file if you want. But it does suggest that you might want to be sure that your processes are appropriate for pip’s current behaviour before arguing about what pip “should” do…

2 Likes

I had no idea; probably because I don’t use virtual environments. Thanks for the heads up!

Yes, this one I have noticed on some systems and in recent containers.
Time to replace PIP_USER=1 by PIP_BREAK_SYSTEM_PACKAGES=1, I guess.
BTW, this is a welcome change (and seems to help resolve the first issue you noted).

That’s correct, unfortunately.

That’s putting words in my mouth I never uttered. I accept that distributions
and their users may have such problems, even if I never had them.
What I do not appreciate is that the purported solution causes unnecessary grief.

I prefer the less drastic option I mentioned above; especially because
I can turn that on selectively on a per-user basis.

This comment seems to imply that my “processes” may NOT be
“appropriate for pip’s current behaviour” peppered with a note
that I can only interpret as an attemtp to discourage me from
arguing against the current solution and potentially exploring
a better alternative. Interesting…

I’d argue that the essence of my “rules” remain unchallenged,
even if user installs no longer require PIP_USER being set:

  1. only unprivileged users run pip install resulting in user installs
  2. only privileged users run system-provided Python-based commands

#1 assumes that the system’s site-packages is not writable
by unprivileged users. If it is, we have a much bigger problem.
In the more likely case that it isn´t, this seems to be the default
behavior now. Kudos for that!

Distributions have very simpe means to enforce #2 by simply
removing execute permissions on the relevant binaries for
unprivileged users.

With that in place, it should suffice to prevent privileged users
from running pip. Doing so would avoid inconveniencing just
about everyone who is not a fan of virtual environments.

Noteworthy is the fairly trivial observation that those are,
in essence, the ONLY folks who are affected by this PEP,
since those who already use virtual environments aren’t
bothered by pip (unnecessarily, I might add, unless we’re
talking about a privileged user running pip).

Yes. If you want to follow those rules, go for it. Your distribution chooses to flag your use of user installs as potentially problematic. Your rule (2) means they are probably[1] wong in your specific case, so you have to explicitly say “I know what I’m doing”. As you are aware, PIP_BREAK_SYSTEM_PACKAGES=1 does what you want.

I fail to see the problem. Yes, your distribution has done something that inconveniences you. You now have to acknowledge that you’ve taken precautions to keep things safe. They presumably feel that the benefits (to everyone) outweigh the downsides (to presumably a few people). You can be annoyed that you have to suffer for the benefit of others, and you can even believe that the benefits aren’t as big as your distro maintainers think. Take it up with them if you think you have a case.

But why are you arguing against the simple existence of a mechanism that addresses a real need, that distro maintainers have been asking us to provide for a long time?

If you have an alternative proposal, you probably should have made it when the feature was being discussed. If you missed that discussion, by all means make a proposal now (but you’d have to cover transition, as distros are using the PEP 668 mechanism already, so you can’t ignore that).

Anyway, I don’t think I have much else to add here, and I don’t want to fuel this whole argument. You’ve made your statement, let’s leave it at that. Feel free to respond to any points I made above if you feel the need, but I won’t continue this discussion any further.


  1. I’m not an expert, so I won’t try to second guess them here. ↩︎

6 Likes

Given the current status quo, my rules are not water-tight.
They also expressly ignore potential user-facing applications
(i.e. not system programs or admin tools) written in Python
that a distro might carry. Those can break with incompatible
packages in user installs. BUT, such breakage should be fairly
trivial to avoid during packaging with a simple wrapper that sets
PYTHONNOUSERSITE, it would seem.

In either case, PIP_BREAK_SYSTEM_PACKAGES=1 covers
my needs. So, I could certainly leave it at that.

What prompts me to answer nonetheless is the feeling that
my gripes are being totaly misunderstood.

My point is that the approach taken by this PEP is suboptimal;
and the claimed benefits could have been achieved without
the inconvenience that the current solution implies.

If this were the only reasonable way, I wouldn’t react.

BUT, as you also said, 'nough said… I’m late to the party
and not sufficiently motivated to shave this yak.

12 posts were split to a new topic: Alternative approaches to distros providing a “system Python”

I think there is one significantly negative aspect here, which is that people who are new to Python are already somewhat well-protected by pip install --user, as long as distros are careful to use the -s flag in shebangs, which arguably they should do anyway.

I would hope at least that PyPA and/or Python has a concise informative document describing the use of a venv (or a non-distro installation tool like Pyenv) and that Pip offers an informative error message in the event that pip install --user is blocked, ideally directing to that informative document. Part of the difficulty in getting started with Python is that there is no unambiguous series of steps to get started in a manner that is considered safe. Otherwise there will be even more confused people than before wondering why Python is so hard to use.

Compare to e.g. Haskell or OCaml. Their installation instructions unambiguously and overtly recommend GHCUp and OPAM respectively [1] [2]. There is no beating around the bush, long HOWTO docs, etc. Python already has a bit of a documentation problem where the docs are really hard to use unless you already know what you’re doing. As the Python tooling improves, and the language itself adopts features to prevent bad usage, it seems important to also consider the new-user experience and the state of documentation in general.

I don’t really know how documentation normally is contributed and written in cases like this, especially when it’s content that might go on python.org and not just buried in a howto on docs.python.org. I’d like to help if I can.

Yup, there’s a creating virtual environments section at the top of the official Installing Python Packages tutorial in the Python Packaging User Guide (packaging.python.org).

As discussed in the PEP and above, the error message by default is fairly descriptive and includes appropriate links, and more importantly is customizable by the distro that controls it to include whatever description and links most relevant to the distro’s specific use case for choosing to enable the EXTERNALLY-MANAGED flag.

There’s been a lot of conversation about this across a number of threads in the past few months. You can read through those if you want/dare. Suffice it to say that there is some disagreement about whether Python can or should provide something as unambiguous as the Haskell/OCaml docs you mention, but it does seem there is increasing momentum towards providing some more opinionated guides of that sort, and giving them more prominent placement in the docs.

I think that there is an option for a win-win compromise which preserves both the system safety and the ability of the users to use Python and pip.

Stakeholder requirements:

  1. Running pip install does not affect the system state. (Desired by the distro owners)
  2. Running pip install works. (Desired by Python users and devs.)

A solution that satisfies both of these requirements can be implemented.

In fact this would have been the case automatically if the distos by deafult only used system pacakges and did not touch the user-local packages. (A pretty sane approach for security and stability.)
Distros where system apps are using user-local packages are arguably broken and should fix their bug instead of breaking the users.
Such user-breakable distros use the PEP668’s EXTERNALLY-MANAGED flag file to disable installing user-local packages (among other things). These distros want the user-local packages to be empty so that the system apps do not use the user-local packages. In that case maybe the system apps must not use the user-local packages? If you don’t want to use these packages, then do not use them.

The distros that for some reason refuse to stop self-breaking by continuing to use the user-local packages, should then provide an alternative user-local virtual environment that is automatically activated by default at user login. (But just stopping using the user-local packages would be better of course.)

Formal scenarios:

Normal user wants to install package and use it:

  1. User logs in.
  2. User runs python -m pip install some-package
  3. User uses/runs the new package.

Expected:

  • Installation completes successfully
  • System packages are not affected

Admin wants to change system packages:

  • Admin uses distro-provided package tools to install package
    or
  • Admin runs system-python -m pip install some-package

Are there any reasons why such win-win solution not desirable?

1 Like

This would be quite challenging, as ‘login’ encompasses many things:

  • Interactive sessions (local or remote)
  • Non-interactive sessions (e.g. ssh <host> <command>)
  • Scheduled tasks (cron, anacron, systemd timers, etc.)
  • ‘hooks’ run by desktop environments, NetworkManager, and other tools

A user who has run pip install to install some package, without realizing that it got installed into a virtual environment because that was done implicitly, would have a reasonable expectation that scripts they invoke from cron/anacron, or other non-interactive environments, would make use of that package.

Additionally, upgrading existing virtual environments when a new version of the distro-provided interpreter is installed can be a non-trivial exercise. Attempting to hide that complexity from the user by having an implicitly-used virtual environment could lead to very hard-to-diagnose failures.

For the co-authors on this PEP and maintainers of packaging tools that have had implementation changes due to this PEP: are there any outstanding implementation pieces left here?

If not, I’d like to mark the PEP as final and move it into packaging.python.org. :slight_smile:

1 Like

This specific approach is discussed in the PEP.

I would appreciate it if we don’t end up going into a cyclical debate about the proposal at this point though, especially about something already discussed at length here and over the years. The reason this PEP is long is because it discusses a large matrix of effects of the design choices and picks a design with the most sensible tradeoffs that aren’t too painful outside of one specific workflow that was deemed problematic.


As far as I can tell, we’ve discussed and looked at all the possible solutions in the years leading up to this PEP, especially the ones that people who seemingly haven’t read the PEP have proposed as the “obvious” solutions after the changes described in this PEP rolled out affecting their workflows that relied on mangling system packages due to not using virtual environments.

To be clear, I’m not singling out @Ark-kun here – many people on the Internet have proclaimed that a system Python/separated distro scheme/magic pixie dust is the obvious technical solution with no tradeoffs; other than of course the complexity of actually implementing it on shoe-string resourcing that most of these foundational pieces of open source software operate on, as well as the amount of coordination it’d require by oh-so-many people aside from the aspects that the PEP already covers. They aren’t wrong that those solve the problem of pip install failing in a manner that requires additional user action, but we’re not locked out of them with this proposal either.

2 Likes

Yes, we’ve deployed it in Debian and Ubuntu stable releases. Now that we have implemented and documented the overrides, I am happy with how things are looking.

This reminds me that we should drop our pip patch protecting the system, now.

3 Likes