PEP 668: Marking Python base environments as "externally managed"

Yes. If it’s externally managed, it’s supposed to not be modified by pip and friends (unless a flag is passed).

To be precise, as I understand it, it’s a scheme that is externally managed, not a directory.

1 Like

Exactly. And we want to mark a scheme externally managed, but the way to mark it is to put a marker file to the stdlib directory, which is unfortunately shared with other schemes. It would make my life much easier if I could put the marker to purelib or platlib instead.

It’s not like pip is ever going to install into stdlib anyway.

Ah I see. I don’t think the case of two schemes where part of the scheme is shared was considered in the discussion. Is this something new that Fedora is doing?

Probably not. But wasn’t that sort of the point? Because tools shouldn’t be installing there, the EXTERNALLY-MANAGED file is protected? If it’s in purelib, I could install a wheel that includes an EXTERNALLY-MANAGED file, and my environment would then be locked.

Anyway, I think it’s fair to say that this wasn’t something covered by the PEP so it needs a discussion and PEP revision if we want to allow it. I don’t personally have a strong opinion on this, so I’ll leave it to interested parties to agree how best to solve the issue.

2 Likes

Fedora is currently using one scheme. But when we experimented with two schemes, this was always the case. There is no standard library in /usr/local/... so the “local” scheme cannot point stdlib there.

1 Like

If I recall the discussion at the time, this is intended by design. Libs installed to /usr/local/lib* still affect the entire Python installation, so it doesn’t make much difference to allow it and only disallow installing into /usr/lib*.

I’ve re-read Create separate distro and local directories and indeed it does not explicitly say whether the “local” directory should or should not be externally managed. However, in my head, it doesn’t make sense to have the “local” directory marked as externally managed if the external tool does not manage it.

If installing into it can result in the system Python seeing different versions of packages than the ones installed in the distro directory, then surely that shouldn’t be allowed for the same reason that we don’t allow packages to be installed in the distro directory itself? (If that makes the local directory useless in practice, that’s a separate issue IMO).

So does installing packages to your HOME directory.

The externally-managed (i.e. RPM-installed) Python software in Fedora uses -s in Python shebangs[1] and that excludes the “local” site-packages directory as well as the “home” site-packages directory from sys.path. As a result, when Fedora users run Python directly, the “local” and “home” packages are visible, but when they run software installed via RPM, the “local” and “home” packages cannot shadow the RPM-installed libraries.

Indeed, a “local” directory that does not allow installing into indeed seems useless, which is why I feel so confused about this.


  1. Except for software that is packaged incorrectly (and can be fixed) or software that supports pip-installed plugins (and hence it needs to explicitly opt-out from the protection). ↩︎

Yes, I too find this very confusing.

Is that standard Python behaviour, or caused by Fedora-specific patches? I can’t find a Python build (Windows and WSL/Ubuntu) with “distro” and “local” directories on sys.path, so I can’t really guess what the “expected” behaviour here should be. And Python’s documentation for sys.path doesn’t help, as it says that the directories included are “system dependent”.

I’m also confused as to the interaction of sysconfig and sys.path here. After all, the sysconfig install path for stdlib isn’t exactly an install path, as you shouldn’t be installing to it. So what’s the point in having that key in sysconfig - the sys.path initialisation code doesn’t use it as far as I can tell.

So I guess my take on this (to reiterate, I don’t have any particular preferred outcome here, beyond it being broadly consistent and understandable) is that the current PEP is probably as good as it can be, but it’s based on an underlying Python model which doesn’t actually reflect the realities of what distros do. Therefore, I’d argue that the “right” fix is to get sysconfig changed to better reflect the reality of how Python is configured by distributors, and then update the PEP to match that.

One problem is that there doesn’t seem to be much interest from core Python in this issue - sysconfig doesn’t get much love, and the pressure seems to be to move packaging concerns out of core Python. I can understand that up to a point, but IMO sysconfig is the key point of interaction - packaging tools can be independent as long as they have a way of introspecting the installation’s layout - and that introspection has to be part of the stdlib, otherwise we have a bootstrapping issue.

Of course, we can find workarounds for this specific problem, but at some point I think we need to address the root cause here…

1 Like

With an ubuntu container I get “dist-packages” in both /usr/lib/ and /usr/local/lib/ on sys.path:

$ podman run --rm -it ubuntu
root@44888ff75ae5:/# apt-get update > /dev/null
root@44888ff75ae5:/# apt-get install -y python3-pip > /dev/null
debconf: delaying package configuration, since apt-utils is not installed
root@44888ff75ae5:/# python3
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']

Ubuntu packages install to one, pip installs to the other:

root@44888ff75ae5:/# apt-get install -y python3-requests > /dev/null
debconf: delaying package configuration, since apt-utils is not installed
root@44888ff75ae5:/# python3 -m inspect -d requests
Target: requests
Origin: /usr/lib/python3/dist-packages/requests/__init__.py
[...]

root@44888ff75ae5:/# pip install bresenham
[...]
root@44888ff75ae5:/# python3 -m inspect -d bresenham
Target: bresenham
Origin: /usr/local/lib/python3.10/dist-packages/bresenham.py
[...]

FWIW, on Ubuntu, sysconfig schemes include 'deb_system':

>>> sysconfig.get_default_scheme()
'posix_local'
>>> sysconfig.get_path('platlib', scheme='posix_local')
'/usr/local/lib/python3.10/dist-packages'
>>> sysconfig.get_path('platlib', scheme='deb_system')
'/usr/lib/python3/dist-packages'

And Fedora currently has rpm_prefix:

>>> sysconfig.get_default_scheme()
'posix_prefix'
>>> sysconfig.get_path('platlib', scheme='posix_prefix')
'/usr/local/lib64/python3.10/site-packages'
>>> sysconfig.get_path('platlib', scheme='rpm_prefix')
'/usr/lib64/python3.10/site-packages'

To answer this: sysconfig reflects the paths that CPython uses (primarily in make install and getpath.c), rather than defining them. So changes to sysconfig will usually require changes elsewhere in order to be consistent, because there’s no single source of truth.

1 Like

Is the intended goal for Fedora to allow /usr/local installations with pip, without flags?

If so, are there any specific reasons for preferring this over the PEP’s recommendations to nudge users toward virtual environments by default?

Normally I’d say “each environment needs it’s own /lib path”, but since this topic is specifically about the “base environment”, I think whatever makes up the base environment[1], including the /lib that goes along with it, needs to be externally managed.


  1. to me, this includes the system python, but we may have different interpretations of “base environment” ↩︎

It might not be the best long-term solution, but it works now and changing it will break things for some users. I’d be happier with nudging users rather than breaking them.

The PEP 668 way covers use cases I can think of, like:

  • venvs with --system-site-packages should solve issues with system libraries that pip can’t install (e.g. dnf, selinux).
  • applications with pip-installed plugins should probably manage a virtualenv, or add its own path entry for the plugins

I probably forgot something, but anyway: switching to the new way will take time & effort. If we add EXTERNALLY-MANAGED now and pip decides to honor it, things will break for users (both installing to /usr/local/ and with --user). I’m worried that many users will reach for the --break-system-packages hammer and leave that in their scripts forever.
Things would be easier if we could e.g. agree on some kind of INSTALLING-HERE-IS-DEPRECATED-BUT-SAFE marker file for individual path entries, and USER-HOME-INSTALLS-ARE-DEPRECATED-BUT-SAFE, at least for a few releases.

The high-level goal in Fedora that I thought this PEP will help me achieve is:

  • local means /usr/local/lib.../python.../site-packages
  • system means /usr/lib.../python.../site-packages

sudo pip install will install to local

This is already the case in Fedora due to our patch to sysconfig.

sudo pip install --upgrade will install to local and only uninstall from there, never uninstall from system

This is already the case in Fedora due to our patches to sysconfig and pip.
Users who pip install --upgrade pip unfortunately lose this protection because they undo our pip patch [1].

sudo pip install --prefix=/usr will error with the message from EXTERNALLY-MANAGED

This is not the case for Fedora yet.

sudo pip install --prefix=/usr --break-my-system will let users install to system

This is currently moot for Fedora, due to the previous point.


Maybe I simply had bad expectations about this PEP. Sorry for not making this clearer before it was approved. I tried to stay on top of this and then I missed the train when it suddenly got moving.


  1. I’ve just noticed the KeyError comment in the patch is bogus, feel free to ignore it. ↩︎

I’m gonna use this to sneak in a possibly-paraphrased quote that I really like (I first heard it from a John Green):

Long term systemic problems require long term systemic solutions.

This is tackling a UX/correctness problem that has calcified over a long time. I promise I’m not trying to hurry this along in a disruptive manner.


FWIW, I think it’s sensible for redistributions to disable user installs with site.ENABLE_USER_SITE being modified (it supports a default of False, instead of the regular None). I don’t recall if we stated that in the PEP as a recommendation.

These together would basically mean that the system-provided Python can only be managed with a system package manager and all other install-via-pip use cases are delegated to virtual environments.


I guess one thing we could do to ease transitions is enable externally managed environments to present warnings and continue, instead of just erroring out, by supporting a Warning key in addition to Error. I’m unsure how useful that would be TBH, but doesn’t hurt to throw the idea out there.

Right now, this PEP says that we will basically disallow installing in both local and system, if the marker file exists.

Broadly, this PEP isn’t trying to solve the /usr vs /usr/local problem (that’s a sysconfig design problem to be solved separately). It’s trying to remove any possibility for users to break their system-package-manager-managed environment by adding/modifying/removing arbitrary files using (a new-enough) pip.

Another way to think of this: this gives redistributors the ability to tell Python tooling “hey, don’t meddle with files in /usr based schemes” (which… as noted affects /usr/local).

That said, part of the motivation behind suggesting setting the schemes explicitly and separately (at least, as far as I remember) was to make it easier to support the sysconfig changes and to eventually have better support the /usr vs /usr/local situation via a CLI argument in pip in the future. It could enable a hypothetical pip install --scheme=posix_local/posix_global to modify files there (which could imply ignoring externally-managed, or require an extra flag). I’m imagining that a future functional change to sysconfig would implement the posix_local as a default (or simplify implementing/patching it) and make some other changes to actually properly support this.

PS: The --prefix approach is a neat idea, however (without patches) regular/vanilla pip installs already use /usr as userbase and posix_prefix as the scheme.

But there are enough core devs here to make changes happen. I think a clear goal for sysconfig along with expectations of how it is to be used would be good. Then we can make the changes upstream since it’s one of those things that has to ship in-box.

1 Like

Agreed. It should probably be a separate topic dedicated to collecting use cases for sysconfig, rather than derailing this thread, though.

1 Like

Adding a cross reference to Linux distro patches to `sysconfig` are changing `pip install --prefix` outside virtual environments since the distro needs for /usr vs /usr/local was discussed there too, and the point of that thread is to figure out/discuss a design for a solution to the problem.