The table in “Use cases” for “Distro Python in Docker” says “assuming the Docker image removes the marker file”, but the later recommendation section suggests that distros keep the marker in container images. From the discussion here, I understand the recommendation section to be the preferred option, so the table should probably be clarified to reflect that.
There is one bit of the spec I don’t fully understand, but maybe I have missed this somewhere else in the PEP. I’m trying to follow the recommendation for the distros. tl;dr I want pip to install to /usr/local/lib(64)/python3.X/site-packages for the users, but RPM to install to /usr/lib(64)/python3.X/site-packages. We currently do this already, but we use the same syconfig installation scheme – I understand that we will need to change that to have two schemes and I accept that.
To follow the specification, we will put the EXTERNALLY-MANAGED marker file to /usr/lib(64)/python3.X/ because that is what sysconfig.get_path("stdlib", sysconfig.get_default_scheme()) returns.
However, for the new installation scheme that uses /usr/local/lib(64)/python3.X/site-packages, stdlib is still located at /usr/lib(64)/python3.X – there is no stdlib in /usr/local/lib(64)/python3.X. How do we make the marker not affect installations to /usr/local/lib(64)/python3.X/site-packages?
There’s no way of doing that IIRC. If you mark an installation as externally managed, you won’t be able to install to any scheme that is enabled in that installation. You can keep carrying your sysconfig patching, but users will need to tell the installer to ignore this flag to install anything outside a virtual environment.
Exactly. And we want to mark a scheme externally managed, but the way to mark it is to put a marker file to the stdlib directory, which is unfortunately shared with other schemes. It would make my life much easier if I could put the marker to purelib or platlib instead.
It’s not like pip is ever going to install into stdlib anyway.
Ah I see. I don’t think the case of two schemes where part of the scheme is shared was considered in the discussion. Is this something new that Fedora is doing?
Probably not. But wasn’t that sort of the point? Because tools shouldn’t be installing there, the EXTERNALLY-MANAGED file is protected? If it’s in purelib, I could install a wheel that includes an EXTERNALLY-MANAGED file, and my environment would then be locked.
Anyway, I think it’s fair to say that this wasn’t something covered by the PEP so it needs a discussion and PEP revision if we want to allow it. I don’t personally have a strong opinion on this, so I’ll leave it to interested parties to agree how best to solve the issue.
Fedora is currently using one scheme. But when we experimented with two schemes, this was always the case. There is no standard library in /usr/local/... so the “local” scheme cannot point stdlib there.
If I recall the discussion at the time, this is intended by design. Libs installed to /usr/local/lib* still affect the entire Python installation, so it doesn’t make much difference to allow it and only disallow installing into /usr/lib*.
I’ve re-read Create separate distro and local directories and indeed it does not explicitly say whether the “local” directory should or should not be externally managed. However, in my head, it doesn’t make sense to have the “local” directory marked as externally managed if the external tool does not manage it.
If installing into it can result in the system Python seeing different versions of packages than the ones installed in the distro directory, then surely that shouldn’t be allowed for the same reason that we don’t allow packages to be installed in the distro directory itself? (If that makes the local directory useless in practice, that’s a separate issue IMO).
So does installing packages to your HOME directory.
The externally-managed (i.e. RPM-installed) Python software in Fedora uses -s in Python shebangs and that excludes the “local” site-packages directory as well as the “home” site-packages directory from sys.path. As a result, when Fedora users run Python directly, the “local” and “home” packages are visible, but when they run software installed via RPM, the “local” and “home” packages cannot shadow the RPM-installed libraries.
Indeed, a “local” directory that does not allow installing into indeed seems useless, which is why I feel so confused about this.
Except for software that is packaged incorrectly (and can be fixed) or software that supports pip-installed plugins (and hence it needs to explicitly opt-out from the protection). ↩︎
Is that standard Python behaviour, or caused by Fedora-specific patches? I can’t find a Python build (Windows and WSL/Ubuntu) with “distro” and “local” directories on sys.path, so I can’t really guess what the “expected” behaviour here should be. And Python’s documentation for sys.path doesn’t help, as it says that the directories included are “system dependent”.
I’m also confused as to the interaction of sysconfig and sys.path here. After all, the sysconfig install path for stdlib isn’t exactly an install path, as you shouldn’t be installing to it. So what’s the point in having that key in sysconfig - the sys.path initialisation code doesn’t use it as far as I can tell.
So I guess my take on this (to reiterate, I don’t have any particular preferred outcome here, beyond it being broadly consistent and understandable) is that the current PEP is probably as good as it can be, but it’s based on an underlying Python model which doesn’t actually reflect the realities of what distros do. Therefore, I’d argue that the “right” fix is to get sysconfig changed to better reflect the reality of how Python is configured by distributors, and then update the PEP to match that.
One problem is that there doesn’t seem to be much interest from core Python in this issue - sysconfig doesn’t get much love, and the pressure seems to be to move packaging concerns out of core Python. I can understand that up to a point, but IMO sysconfig is the key point of interaction - packaging tools can be independent as long as they have a way of introspecting the installation’s layout - and that introspection has to be part of the stdlib, otherwise we have a bootstrapping issue.
Of course, we can find workarounds for this specific problem, but at some point I think we need to address the root cause here…
To answer this: sysconfigreflects the paths that CPython uses (primarily in make install and getpath.c), rather than defining them. So changes to sysconfig will usually require changes elsewhere in order to be consistent, because there’s no single source of truth.