I’m OK with the support expressed in this thread, I would take that as demonstrating consensus, and that’s sufficient.
BTW, I guess as no-one else volunteered as PEP delegate, this is going to be me, so could one of the authors update the PEP with the appropriate PEP-delegate header?
Added Paul as the PEP-Delegate in:
Assuming no one has concerns with Paul being the PEP Delegate on this, I think this PEP is ready for pronouncement.
OK. I’ll give this a week for anyone who wants to raise any final concerns, and I’ll aim to make a decision by next weekend.
OK, I have reviewed the PEP and the discussion here, and I’d like to formally accept this PEP. Congratulations to everyone involved on a well-written PEP and a well-managed discussion 
Could someone update the PEP to reflect this decision, please?
I do have a couple of minor points which might be worth clarifying in the PEP, neither of which affect the decision as they only relate to the recommendations part of the document.
Hurrah! Thanks @pf_moore! I’ve filed:
Thanks folks!
I am going to try to implement this in Fedora 38.
There is one bit of the spec I don’t fully understand, but maybe I have missed this somewhere else in the PEP. I’m trying to follow the recommendation for the distros. tl;dr I want pip to install to /usr/local/lib(64)/python3.X/site-packages for the users, but RPM to install to /usr/lib(64)/python3.X/site-packages. We currently do this already, but we use the same syconfig installation scheme – I understand that we will need to change that to have two schemes and I accept that.
To follow the specification, we will put the EXTERNALLY-MANAGED marker file to /usr/lib(64)/python3.X/ because that is what sysconfig.get_path("stdlib", sysconfig.get_default_scheme()) returns.
However, for the new installation scheme that uses /usr/local/lib(64)/python3.X/site-packages, stdlib is still located at /usr/lib(64)/python3.X – there is no stdlib in /usr/local/lib(64)/python3.X. How do we make the marker not affect installations to /usr/local/lib(64)/python3.X/site-packages?
There’s no way of doing that IIRC. If you mark an installation as externally managed, you won’t be able to install to any scheme that is enabled in that installation. You can keep carrying your sysconfig patching, but users will need to tell the installer to ignore this flag to install anything outside a virtual environment.
Was that the intended outcome of the PEP?
Yes. If it’s externally managed, it’s supposed to not be modified by pip and friends (unless a flag is passed).
To be precise, as I understand it, it’s a scheme that is externally managed, not a directory.
Exactly. And we want to mark a scheme externally managed, but the way to mark it is to put a marker file to the stdlib directory, which is unfortunately shared with other schemes. It would make my life much easier if I could put the marker to purelib or platlib instead.
It’s not like pip is ever going to install into stdlib anyway.
Ah I see. I don’t think the case of two schemes where part of the scheme is shared was considered in the discussion. Is this something new that Fedora is doing?
Probably not. But wasn’t that sort of the point? Because tools shouldn’t be installing there, the EXTERNALLY-MANAGED file is protected? If it’s in purelib, I could install a wheel that includes an EXTERNALLY-MANAGED file, and my environment would then be locked.
Anyway, I think it’s fair to say that this wasn’t something covered by the PEP so it needs a discussion and PEP revision if we want to allow it. I don’t personally have a strong opinion on this, so I’ll leave it to interested parties to agree how best to solve the issue.
Fedora is currently using one scheme. But when we experimented with two schemes, this was always the case. There is no standard library in /usr/local/... so the “local” scheme cannot point stdlib there.
If I recall the discussion at the time, this is intended by design. Libs installed to /usr/local/lib* still affect the entire Python installation, so it doesn’t make much difference to allow it and only disallow installing into /usr/lib*.
I’ve re-read Create separate distro and local directories and indeed it does not explicitly say whether the “local” directory should or should not be externally managed. However, in my head, it doesn’t make sense to have the “local” directory marked as externally managed if the external tool does not manage it.
If installing into it can result in the system Python seeing different versions of packages than the ones installed in the distro directory, then surely that shouldn’t be allowed for the same reason that we don’t allow packages to be installed in the distro directory itself? (If that makes the local directory useless in practice, that’s a separate issue IMO).
So does installing packages to your HOME directory.
The externally-managed (i.e. RPM-installed) Python software in Fedora uses -s in Python shebangs[1] and that excludes the “local” site-packages directory as well as the “home” site-packages directory from sys.path. As a result, when Fedora users run Python directly, the “local” and “home” packages are visible, but when they run software installed via RPM, the “local” and “home” packages cannot shadow the RPM-installed libraries.
Indeed, a “local” directory that does not allow installing into indeed seems useless, which is why I feel so confused about this.
Except for software that is packaged incorrectly (and can be fixed) or software that supports pip-installed plugins (and hence it needs to explicitly opt-out from the protection). ↩︎
Yes, I too find this very confusing.
Is that standard Python behaviour, or caused by Fedora-specific patches? I can’t find a Python build (Windows and WSL/Ubuntu) with “distro” and “local” directories on sys.path, so I can’t really guess what the “expected” behaviour here should be. And Python’s documentation for sys.path doesn’t help, as it says that the directories included are “system dependent”.
I’m also confused as to the interaction of sysconfig and sys.path here. After all, the sysconfig install path for stdlib isn’t exactly an install path, as you shouldn’t be installing to it. So what’s the point in having that key in sysconfig - the sys.path initialisation code doesn’t use it as far as I can tell.
So I guess my take on this (to reiterate, I don’t have any particular preferred outcome here, beyond it being broadly consistent and understandable) is that the current PEP is probably as good as it can be, but it’s based on an underlying Python model which doesn’t actually reflect the realities of what distros do. Therefore, I’d argue that the “right” fix is to get sysconfig changed to better reflect the reality of how Python is configured by distributors, and then update the PEP to match that.
One problem is that there doesn’t seem to be much interest from core Python in this issue - sysconfig doesn’t get much love, and the pressure seems to be to move packaging concerns out of core Python. I can understand that up to a point, but IMO sysconfig is the key point of interaction - packaging tools can be independent as long as they have a way of introspecting the installation’s layout - and that introspection has to be part of the stdlib, otherwise we have a bootstrapping issue.
Of course, we can find workarounds for this specific problem, but at some point I think we need to address the root cause here…
With an ubuntu container I get “dist-packages” in both /usr/lib/ and /usr/local/lib/ on sys.path:
$ podman run --rm -it ubuntu
root@44888ff75ae5:/# apt-get update > /dev/null
root@44888ff75ae5:/# apt-get install -y python3-pip > /dev/null
debconf: delaying package configuration, since apt-utils is not installed
root@44888ff75ae5:/# python3
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']
Ubuntu packages install to one, pip installs to the other:
root@44888ff75ae5:/# apt-get install -y python3-requests > /dev/null
debconf: delaying package configuration, since apt-utils is not installed
root@44888ff75ae5:/# python3 -m inspect -d requests
Target: requests
Origin: /usr/lib/python3/dist-packages/requests/__init__.py
[...]
root@44888ff75ae5:/# pip install bresenham
[...]
root@44888ff75ae5:/# python3 -m inspect -d bresenham
Target: bresenham
Origin: /usr/local/lib/python3.10/dist-packages/bresenham.py
[...]
FWIW, on Ubuntu, sysconfig schemes include 'deb_system':
>>> sysconfig.get_default_scheme()
'posix_local'
>>> sysconfig.get_path('platlib', scheme='posix_local')
'/usr/local/lib/python3.10/dist-packages'
>>> sysconfig.get_path('platlib', scheme='deb_system')
'/usr/lib/python3/dist-packages'
And Fedora currently has rpm_prefix:
>>> sysconfig.get_default_scheme()
'posix_prefix'
>>> sysconfig.get_path('platlib', scheme='posix_prefix')
'/usr/local/lib64/python3.10/site-packages'
>>> sysconfig.get_path('platlib', scheme='rpm_prefix')
'/usr/lib64/python3.10/site-packages'
To answer this: sysconfig reflects the paths that CPython uses (primarily in make install and getpath.c), rather than defining them. So changes to sysconfig will usually require changes elsewhere in order to be consistent, because there’s no single source of truth.