PEP 632 - Deprecate distutils module

PEPs
104

Hello. We have met with @jaraco, @pradyunsg, @encukou and Matthias Klose and discussed the issues with the move of distutils from the standard library to setuptools (or any other pip-installable project) affecting Fedora and Debian.

Status quo

Both distributions currently patch distutils.command.install.INSTALL_SCHEMES and have custom logic for selecting the appropriate scheme.

Both patches are designed for the same purpose: When the Python packages are installed by users (e.g. via sudo), they are installed to a location that is not managed by the system package manager (i.e. /usr/local/lib(64)?/pythonX.Y/(site|dist)-packages). When the Python packages are installed to a fake --root by the system package manager build scripts (e.g. rpmbuild), they are installed to the system location (i.e. /path/to/buildroot/usr/lib64/pythonX.Y/(site|dist)-packages, so the system packages (.deb or .rpm) are installed to /usr/lib(64)?/pythonX.Y/(site|dist)-packages). The site module is also patched to ensure proper sys.path.

I might not like it, but users often run sudo pip install, sudo pip install --upgrade or even sudo python setup.py install. The patches exist to prevent systems being bricked by this behavior (e.g. by replacing a dependency for the package manager with an incompatible version).

The problem

If the code moves to setuptools (or pip or pypa/distutils or any other pip-installable project), users will eventually override our patches with the pip-installed version of distutils. This will then let them override system packages.

Hence, if distutils needs to be removed from stdlib, the code that we patch needs to move to another part of the standard library.

The plan

At the meeting, we have come up with the following plan that suits both PyPA representatives and distros representatives. We present the plan to CPython/PEP 632 representatives and would like to see if it works for you as well.

To avoid confusion, I use the term pypa/distutils for the distutils fork that is to be bundled in setuptools. I use the term cpython/distutils for the distutils in Python’s standard library.

The plan is presented first as is, rationales to follow.

  1. pypa/distutils keeps INSTALL_SCHEMES defined for compatibility with Python < 3.10, it also gains a get_default_scheme() function.
  2. cpython/distutils in Python 3.10+ moves INSTALL_SCHEMES to sysconfig.INSTALL_SCHEMES and imports them for backwards compatibility, uses sysconfig._get_default_scheme() to determine the scheme.
  3. distributors backport (2) to older Python versions and patch sysconfig instead of patching cpython/distutils.
  4. pypa/distutils uses INSTALL_SCHEMES and _get_default_scheme() from sysconfig if found, fallbacks to its own definitions if not.
  5. pypa/distutils fallbacks will honor distributors’ intentions to the greatest extent possible.

Rationales

  1. Setuptools would like to evolve its pypa/distutils. To do so, it cannot use cpython/distutils on older Python versions and pypa/distutils on newer; it needs to always use pypa/distutils. Hence pypa/distutils need to be able to work with Pythons that don’t yet have sysconfig.INSTALL_SCHEMES and sysconfig._get_default_scheme(). Hence, it needs to contain a reasonable set of install schemes (as it does today). Nothing needs to change.

  2. sysconfig was selected based on our common understanding of the module’s purpose. We don’t necessarily insist it is sysconfig, as long as the module exists in currently existing Python versions. sysconfig already contains some installation schemes that are not 1:1 copy of the installation schemes from distutils. The idea is to unify those as much as possible and provide a functionality that would allow the deprecated cpython/distutils module to present the installation schemes from sysconfig in a backward-compatible way. (In the worst-case scenario, we namespace the different sets of installation schemes.) I am prepared to work on a proof of concept of this if we agree on the plan. If the proof of concept works, the Python Maintenance team at Red Hat can dedicate a long-term commitment.

  3. The distributions already patch both old and new Python versions. We are prepared to change our patches to move the patched definitions to sysconfig and use read them form there in cpython/distutils. That way, the behavior remains the same, except the source of the information shifts aside to allow the next points to happen.
    There are these edge cases where users might get a broken system:

    • users of end-of-life Fedora/Debian versions who decide to install/update setuptools from PyPI
    • users who don’t update their system and decide to install/update setuptools from PyPI

    There’s not much we can do about that.

  4. pypa/distutils support old and new Python versions. It needs to function if the installation schemes exist in sysconfig as well as if it does not. Preferring feature-detection over version-detection, pypa/distutils fully supports the distributors who backported the changes and respect their patches. Setuptools maintainers are prepared to work on this.

  5. Setuptools maintainers would like to be able to adapt pypa/distutils defaults in a way that would make the patches minimal (ideally unnecessary, however, that might not be possible). If useful, such defaults might propagate to Python’s sysconfig later. This step is not necessarily needed right now.

I’ve emphasized some parts to illustrate that we have the resources to make this happen.

3 Likes