PEP 632 - Deprecate distutils module

create_tree is weird in that it operates on a set of paths, and we don’t have anywhere that does that. Replacing it more direct code is likely to be better in almost every case:

for p in {Path(f).parent for f in filenames}:
    p.mkdir()

change_root seems like a generally bad idea, but that’s just based on reading the code. If someone can specify its behaviour in a way that makes it seem at all useful, I might consider it.

Adding a static strtobool or getboolean to configparser seems fine. I still don’t think it’s worth blocking the distutils deprecation over, and it’s probably easier to get the new function through the python-ideas discussion by pointing out the existing one has been deprecated.

1 Like

I’d just move strtobool to configparser via a bpo issue. I don’t think it needs to fight its way through python-ideas.

It might need some improvements, and people may want to argue against configparser being used as a dumping ground for unsupported functionality. Whoever maintains that module should certainly get a say.

1 Like

Two releases worth of deprecation period is plenty of time to migrate any additional APIs that folks feel are sufficiently hard to replicate correctly that if makes sense for them to stay somewhere in the standard library rather than the projects using them making a copy in their own utility libraries, so I also think it would be fine to stick with Steve’s initial plan, where the suggestion for some APIs is “copy the code you’re using”.

3 Likes

Anyone want to add or reinforce any complaints that they feel haven’t been addressed sufficiently? If not, I’ll ask the SC for a delegate (you can still raise concerns after that).

2 Likes

Hello. We have met with @jaraco, @pradyunsg, @encukou and Matthias Klose and discussed the issues with the move of distutils from the standard library to setuptools (or any other pip-installable project) affecting Fedora and Debian.

Status quo

Both distributions currently patch distutils.command.install.INSTALL_SCHEMES and have custom logic for selecting the appropriate scheme.

Both patches are designed for the same purpose: When the Python packages are installed by users (e.g. via sudo), they are installed to a location that is not managed by the system package manager (i.e. /usr/local/lib(64)?/pythonX.Y/(site|dist)-packages). When the Python packages are installed to a fake --root by the system package manager build scripts (e.g. rpmbuild), they are installed to the system location (i.e. /path/to/buildroot/usr/lib64/pythonX.Y/(site|dist)-packages, so the system packages (.deb or .rpm) are installed to /usr/lib(64)?/pythonX.Y/(site|dist)-packages). The site module is also patched to ensure proper sys.path.

I might not like it, but users often run sudo pip install, sudo pip install --upgrade or even sudo python setup.py install. The patches exist to prevent systems being bricked by this behavior (e.g. by replacing a dependency for the package manager with an incompatible version).

The problem

If the code moves to setuptools (or pip or pypa/distutils or any other pip-installable project), users will eventually override our patches with the pip-installed version of distutils. This will then let them override system packages.

Hence, if distutils needs to be removed from stdlib, the code that we patch needs to move to another part of the standard library.

The plan

At the meeting, we have come up with the following plan that suits both PyPA representatives and distros representatives. We present the plan to CPython/PEP 632 representatives and would like to see if it works for you as well.

To avoid confusion, I use the term pypa/distutils for the distutils fork that is to be bundled in setuptools. I use the term cpython/distutils for the distutils in Python’s standard library.

The plan is presented first as is, rationales to follow.

  1. pypa/distutils keeps INSTALL_SCHEMES defined for compatibility with Python < 3.10, it also gains a get_default_scheme() function.
  2. cpython/distutils in Python 3.10+ moves INSTALL_SCHEMES to sysconfig.INSTALL_SCHEMES and imports them for backwards compatibility, uses sysconfig._get_default_scheme() to determine the scheme.
  3. distributors backport (2) to older Python versions and patch sysconfig instead of patching cpython/distutils.
  4. pypa/distutils uses INSTALL_SCHEMES and _get_default_scheme() from sysconfig if found, fallbacks to its own definitions if not.
  5. pypa/distutils fallbacks will honor distributors’ intentions to the greatest extent possible.

Rationales

  1. Setuptools would like to evolve its pypa/distutils. To do so, it cannot use cpython/distutils on older Python versions and pypa/distutils on newer; it needs to always use pypa/distutils. Hence pypa/distutils need to be able to work with Pythons that don’t yet have sysconfig.INSTALL_SCHEMES and sysconfig._get_default_scheme(). Hence, it needs to contain a reasonable set of install schemes (as it does today). Nothing needs to change.

  2. sysconfig was selected based on our common understanding of the module’s purpose. We don’t necessarily insist it is sysconfig, as long as the module exists in currently existing Python versions. sysconfig already contains some installation schemes that are not 1:1 copy of the installation schemes from distutils. The idea is to unify those as much as possible and provide a functionality that would allow the deprecated cpython/distutils module to present the installation schemes from sysconfig in a backward-compatible way. (In the worst-case scenario, we namespace the different sets of installation schemes.) I am prepared to work on a proof of concept of this if we agree on the plan. If the proof of concept works, the Python Maintenance team at Red Hat can dedicate a long-term commitment.

  3. The distributions already patch both old and new Python versions. We are prepared to change our patches to move the patched definitions to sysconfig and use read them form there in cpython/distutils. That way, the behavior remains the same, except the source of the information shifts aside to allow the next points to happen.
    There are these edge cases where users might get a broken system:

    • users of end-of-life Fedora/Debian versions who decide to install/update setuptools from PyPI
    • users who don’t update their system and decide to install/update setuptools from PyPI

    There’s not much we can do about that.

  4. pypa/distutils support old and new Python versions. It needs to function if the installation schemes exist in sysconfig as well as if it does not. Preferring feature-detection over version-detection, pypa/distutils fully supports the distributors who backported the changes and respect their patches. Setuptools maintainers are prepared to work on this.

  5. Setuptools maintainers would like to be able to adapt pypa/distutils defaults in a way that would make the patches minimal (ideally unnecessary, however, that might not be possible). If useful, such defaults might propagate to Python’s sysconfig later. This step is not necessarily needed right now.

I’ve emphasized some parts to illustrate that we have the resources to make this happen.

3 Likes

The above all make perfect sense. Thank you all so much for taking the effort to come up with a solution!

I have a few questions, probably cosmetic and not really related to the problem faced by distros:

  1. Since sysconfig already has _INSTALL_SCHEMES, would it make sense to reuse it instead of having a new global variable named similarly but for different purposes?
  2. Can other packaging tools make use of _get_default_scheme() and INSTALLED_SCHEMES, or should they be considered a private interface between distros and pypa/distutils?
  3. Following the above, would it make sense to make _get_default_scheme() a guarenteed (public?) API in CPython? (And _INSTALL_SCHEMES if that’s used instead of a new standalone INSTALLED_SCHEMES, see first point.)

Thanks for the write-up!

I’m more than happy to include migrating install schemes into sysconfig as part of the PEP, though I would like to hear from the pip maintainers how it would impact them.

I assume that pip installs wheels according to the install schemes, though @pf_moore listed distutils.command.install.SCHEME_KEYS above, and since IIUC setuptools also wants to get out of the installation business, pip is probably more important in terms of future proofing.

(Also, I assume you considered patching an additional directory into the sys.path, rather than trying to patch out installing into a distro-controlled directory? That shouldn’t require modifying install schemes at all.)

1 Like

Unless I missed something in my quick read of your proposal, I think it would be important to spell out exactly which version(s) of sysconfig you are talking about. Recall that we are in the unfortunate situation of having two almost identical sysconfigs in the standard library: one in Lib/sysconfig.py and one in Lib/distutils/sysconfig.py. Perhaps this is an opportunity (or requirement) to get rid of one of them?

1 Like

My understanding is that the top-level sysconfig was added when distutils was (infomally) deprecated years ago, but has apparently gone out of sync. Hard to say which is more up to date, but if people have been patching the distutils one with an impact, it’s clearly still in use.

Obviously the distutils version will be going away with the rest of distutils.

1 Like

That is the idea, if at all possible.

I’d like to make it “public”, i.e. renaming to get_default_scheme() or get_default_installation_scheme().

2 Likes

Could you please explain this in more detail?

1 Like

It’s been awhile but IIRC Lib/sysconfig.py was added when work on packaging/distutils2 was underway and it appeared that, for at least some number of releases, both distutils and packaging/distutils2 would need to co-exist in the standard library. Eventually it was decided to not go ahead with that plan and most of the changes to distutils were reverted but the new sysconfig remained. (By the way, that’s also why Lib/_osx_support.py exists: to have in one place the macOS customizations needed for both distutils and packaging/distutils2.) I’m not sure the two versions were ever totally in-sync; I recall that some ones (@jaraco and @barry ?) spent some time a few years back trying to get them more into sync but I don’t see in changelogs where or when that happened so I might be dreaming. Anyway, it’s been a problem for a long time (there are a number of open bpo issues about it) so, rather than muddying the waters further, we should try to resolve this in conjunction with or as part of PEP 632.

1 Like

I don’t think distutils2 included its own copy of sysconfig, as Tarek quite rightly pointed out that sysconfig needs to be in the standard library as it provides information about the interpreter installation and shouldn’t change independently of updates to the interpreter and standard library.

When the migration to distutils2 was judged unsuccessful, the deprecation of distutils.sysconfig still held, but until PEP 632 we didn’t have a new plan for how to follow through on it.

1 Like

On re-reading, the Fedora patch is essentially what I was suggesting. Yes, it’s a little bit hacked, but basically unless you’re building your own packages, you always want /usr/local/lib... to be the default library location, provided /usr/lib... is also on sys.path.

What you described sounds like "default to /usr/lib... and also import from /usr/local/lib... and try to install to /usr/local/lib...", which strikes me as more complicated logic than "default to /usr/local/lib... and also import from /usr/lib...".

Yes, both would require patching, but I’m pretty sure that the latter would only add an extra sys.path.append() somewhere (site.py? getpath.c?) and a build-time configure option, compared to the more complex patching you’re doing now.

Just an idea. I don’t understand your constraints as well as you do, so if there are reasons why this doesn’t work then just ignore me :slight_smile:

1 Like

The current text says (approx) "if you’re using distutils.sysconfig, switch to sysconfig". Is that clearing the waters enough? Or is something more necessary?

1 Like

One constraint is that if site packages are disabled ($PYTHONNOUSERSITE, python -I), you should get /usr/lib but not /usr/local/lib.

2 Likes

Perhaps we could just improve support in upstream for this scenario (not in this PEP)? I might have another go at streamlining the startup process at the sprints and could factor this in. The aim being to remove the need for direct patching.

Or maybe deprecating the legacy install tools and working with the modern install tools would be enough?

Either way, my earlier questions stands:

1 Like

I’m 100% on board, but I also don’t maintain anything that uses legacy install tools, soooo… :slight_smile:

I think that works.

Assuming this PEP does come through, I think we’ll likely change pip to use sysconfig.INSTALL_SCHEMES instead of distutils.command.install.SCHEME_KEYS on Python >=3.10. Looking at the existing implementation, I don’t think it’ll be too difficult (famous last words).

2 Likes

Yes, I think it’s enough for the PEP.

2 Likes