PEP 668: Marking Python base environments as "externally managed"

Packaging Standards
10

I am one of the maintainers of the Python interpreter in Fedora, Red Hat Enterprise Linux (RHEL), and CentOS. My main focus is on Fedora, but Fedora eventually defines what’s it gonna be like in RHEL/CentOS. I’ve been selected to summarize our feedback.

tl;dr we want to participate as much as possible, but we have some small concerns and we want to test it out in practice before it is approved

What we currently do

This also applies to Python 3.6+ in RHEL 7, 8, and future 9 as well as the appropriate CentOS Linux/Stream releases.

  1. We patch distutils, as is mentioned in the PEP. The patch makes sure that we install to /usr/lib(64)/python3.X/site-packages when we create RPM packages, but to /usr/local/lib(64)/python3.X/site-packages for other use cases. The patch is indeed conceptually the sort of hook envisioned by bpo-43976, except implemented as a code patch to distutils instead of as a changed sysconfig scheme. That difference is quite important, stay tuned.

  2. We also patch pip to prevent uninstallation from /usr/lib(64)/python3.X/site-packages when it is upgrading to /usr/local/lib(64)/python3.X/site-packages. This is conceptually what the PEP describes in the Writing to only the target sysconfig scheme, except very hacky and specific to our schemes. I am glad that the PEP addresses this, thank you!

  3. Our modern Python packaging macros use pip to install packages and we remove the RECORD file, as well as set the INSTALLER to rpm. This works nicely to prevent pip uninstalling packages, but will still fail if the user attempts an upgrade if we didn’t have (2). Also, the majority of Python RPM packages in Fedora still use the old macros that install with setup.py install and have egg-info instead of dist-info. Having the marker helps us to address this distro-wide and display a specific actionable error message without patching pip.

  4. Our patch from (1) also patches the site module to include /usr/local... if Python is invoked without -s and we encourage our packagers to use #!/usr/bin/python3 -s in shebangs. Our macros from (3) do that automatically. But some packages need to explicitly see Python packages installed in /usr/local... (e.g. pip-installed plugins) and they cannot use -s.

What problems do we face

  1. Distutils are deprecated and our patch patches distutils instead of sysconfig. Once distutils are gone, we will no longer be able to patch them in the standard library and we would need to apply patches to setuptools/pip instead which is problematic because users tend to upgrade to newer versions of pip/setuptools from PyPI. We have discussed this at length with interested parties (namely @jaraco, @pradyunsg, and a bit with @doko42 and @steve.dower) – a plan emerged and was summarized in PEP 632 discussion.
    We planned to redo our patch in Python 3.10+ to patch sysconfig instead. See our proof of concept as a GitHub pull request. We have submitted several patches to CPython upstream, namely merging distutils.sysconfig into sysconfig and making distutils load install schemes from sysconfig.
    When everything was ready upstream, we have attempted to update our patch to use distutils only to realize there are unexpected consequences – namely the distutils patch was quite limited in its scope and impact (package installation) while the sysconfig one leaked in many different places. Currently, we know how to solve some of the issues we found, but we have not finished this in time for Fedora 35 which introduced Python 3.10 due to other urgent tasks and we plan to continue with Python 3.11.

  2. Pip requires patching. That means once the user does pip install -U pip, the patch is gone. And the next invocation of pip install -U setuptools might as well wipe the RPM-installed setuptools (making it invisible for system tools with #!/usr/bin/python3 -s shebangs, which was a disaster until recently, because entry_point console scripts used to import from pkg_resources (part of setuptools), but even now is a big problem). We were unable to solve this properly and once again, I am very glad that this PEP solves it.

  3. Many people did not understand/like the /usr/local... change when it was introduced, as well as they don’t really understand/like the RECORD-less installations, see for example one recent bug that was reported because get-pip.py is unable to uninstall system-installed pip.

PEP 668 in Fedora

We want to implement this PEP in Fedora, as it will allow us to drop most of our patches (or do them in an upstream-supported way) and hopefully solve the problems. However, our implementation heavily depends on bpo-43976 which is kinda mentioned in the PEP but is not described in much detail. As with our previous attempts to do this, we expect the devil to be in the details and we would like to have a working proof of concept before this PEP is approved. Making bpo-43976 work in Fedora will be my priority for upcoming months, but unfortunatelly we still have some details to figure out (such as the -s behavior). Ideally, we would play with the implementation for a while before this PEP is approved. Is that acceptable to you? A draft pull request for pip and Python would be really helpful.

Our concerns in the current draft

  1. The PEP recommends removing the marker file in container images. It specifically mentions dpkg-based systems, however, this will be really tricky in RPM-based systems. We don’t control all places where Fedora creates container images to be able to explicitly rm that marker. We don’t want to remove it in the post-installation scriptlet, because there is no “configuration flag” here: the installation of packages is not interactive. We could possibly package that marker as a separate package, marking it Recommended by Python, but that only makes it even harder for us to influence what installations will (not) have it.
    We would prefer if the marker is recommended to be installed by distributors even in base container images and people who build on top of those images would explicitly remove it if they want to use pip in a way that conflicts with the purpose of that marker.
  2. The PEP recommends that we make Python require pipx. That is extremely unlikely to happen. Our Python does not even require a system-installed pip. Our Python has venv working out of the box as well as ensurepip because we explicitly want to avoid problems of that thing being broken. pipx is not part of the standard library and we don’t want to pull it into systems that only have Python installed as a dependency of e.g. dnf. There is a certain compromise between “Python is too bloated” and “standard things work out of the box” and requiring pipx is way too far.
  3. bpo-43976 is not part of this PEP’s specification.

Thank you for working on this and trying to standardize things! You rock.

1 Like