How can we keep pip from updating/uninstalling system libraries in /usr/lib/python?

Packaging
1

Hello,
I’m not sure whether to discuss this on the setuptool, distutils or pip tracker, so I’ll ask here.

The distutils (and so, setuptools) install has the concept of install schemes, which control where a package gets installed: system-wide, or to a user’s home.
For a long time, Debian has a patch, distutils-install-layout.diff that adds more install shcemes, unix_local and deb_system, and implements logic to select them. The unix_local scheme installs to /usr/local rather than /usr.

Fedora does a similar thing in a more hacky/minimal (depending on your point of view) way. It also changes the install prefix from /usr/ to /usr/local, in the default case.

These patches assume that the stdlib is tied to an interpreter and can’t be changed. Now that setuptools is replacing stdlib distutils with its own copy, these customizations are starting to disappear.

Is there a supported way to do customizations like this?

I can describe Fedora’s reasons for the patch: it’s to minimize damage to the system caused by sudo pip.
Since critical software including the system package manager is written in Python, when sudo pip install updates a library to an incompatible version, the package manager can stop working. And unfortunately, the Internet is full of advice telling people to use sudo pip.
The solution in Fedora is to:

  • install system software (from RPMs) in /usr
  • make pip (i.e. distutils) install to /usr/local
  • in site.py, put both /usr/lib*/python* and /usr/local/lib*/python* (and their site-packages) on sys.path
    System software then uses Python’s -I option, which skips site customization and thus omit the /usr/local, so it’s not affected by changes from sudo pip.

If newer setuptools replace distutils with its own version using a .pth file, the following can happen:

  • sudo pip install -U setuptools, putting an unpatched copy of distutils in /usr/local
  • sudo pip install -U ..., which installs to /usr directly, possibly breaking system software.

Is there any supported way to solve the this issue?

1 Like
2

Just to clarify, this is happening with pip install, not just pip install -e? Is this because pip is directly using distutils to decide where to install things?

If you are asking for support for patching / patched versions of setuptools, I would say “no”, though @jaraco does more supporting of setuptools than the rest of us combined, so he’s probably the final arbiter on that.

I think reasonable ways forward that could be supported or semi-supported would be:

  1. I think the relevant GH issue ended with, “We don’t want to carry a distro-specific patch, but it’s not inconceivable that this would be of sufficiently general interest that we can support customizing the default install scheme in certain circumstances”. I don’t know the details of how difficult it would be to support this sort of thing upstream without breaking a bunch of other stuff, though.
  2. I think @jaraco always intended devendoring of distutils to be not terribly difficult, which is why GitHub - pypa/distutils: distutils as found in cpython exists. If the main problem is with pip’s use of distutils, it seems like you could devendor and carry the patch on your copy of distutils. I think this will work as long as we assume people are using the system setuptools.
  3. Change pip so it no longer relies on distutils and patch that. This could also possibly involve moving the relevant install scheme stuff into packaging (or a new package entirely), which can be patched if it doesn’t directly support this.

Sorry I am offering brain-storming and not real solutions.

3

From my (maybe incomplete) understanding, pip’s current usage of distutils can all be replaced by sysconfig (and setuptools for the obvious legacy setup.py stuff). I’ve been keeping an eye on this, but couldn’t really do it myself due to the amount of tedious work involved and lack of motivation (the current implementation “works” anyway). I’d happily help if anyone is interested in working on the transition though.

4

Adding and selecting install schemes sounds like it could be solved with configuration instead of patches. Even better that the configuration would be different between root and users, if it were in the configuration file.

1 Like
5

I would love to see configurable “install schemes” to supplement the built in ones. This would simplify and rationalise things like the handling of --target, for example. But honestly, it’s never been something that’s mattered enough to me to do anything about it. I would support efforts in this direction, though.

6

Do you have a link to the relevant GH issue? Unfortunately it’s hard to follow discussion in issues I’m not subscribed to, so I didn’t know about possible new ways of customizing install schemes :‌(
I do agree that carrying distro-specific patches is not good.

Which CPython version is GitHub - pypa/distutils: distutils as found in cpython for? Stdlib code can generally be coupled to the rest of the stdlib, but I guess that’s already known from issues like setuptools#2357.
Anyway, I can’t devendor distutils from the version of setuptools that users install from PyPI.

Users can update pip from PyPI, so changing the system pip (or other libraries) would not solve all of the issues.
So far we relied on the fact that the stdlib can not be updated from PyPI… but with setuptools 50.0, that all changes :‌(

I’m interested in how that would work, since install schemes are part of distutils. (For reference, pip uses them in its locations.py.)
Even if that is possible, it won’t change how older versions of sudo pip work.

7

@jaraco, does that sound like something that could be added in GitHub - pypa/distutils: distutils as found in cpython?
How would configuration of selecting a scheme work? I’m skeptical about doing it declaratively.

8

The same logic (at least the parts pip need) is also implemented (and IIUC better-maintained) in sysconfig, which is available in all supported Python versions. Indeed this wouldn’t fix the older pip versions shpiped with the OS, but everything will stay broken forever unless we start the transition at some point.

9

Right. We can definitely do that in Python 3.10. Where should this be best tracked/discussed?

Currently, to me it seems that the plans for distutils/setuptools are scattered across issues in several projects, so I’m not clear on what is actually happening. Would it be worth it for someone who’s following all this to write a PEP?

1 Like
10

I think there are multiple parts to this issue.

  1. pip needs to “modernise” the install schemes logic to use sysconfig instead, so it installs wheels to thes correct location, and only uninstalls distributions in the right location. This does not need a PEP since it’s project-specific. I don’t think there’s an issue tracking this, but can open one if needed.
  2. setuptools may or may not need to perform the same modernisation (either by switching to sysconfig or modifying the bundled setuptools._distutils), depending on the maintainers’ opinion on this. This only needs solving if setuptools intends to continue support for “legacy” installations (non-PEP-517 setup.py install and setup.py develop).
  3. Implement “configurable install schemes” in sysconfig. This is the part that needs involvement from CPython core developers, and likely requires a PEP to specify and design. I’m not sure how this would look like TBH, but can probably offer opinions if someone lays out the foundation.
11

I can also see other changes that seem PEP-worthy:

  1. distutils should be deprecated and removed from the standard library (right?)
  2. setuptools provides its own copy of distutils (overriding the stdlib copy which might be different across OSes or interpreters or versions – on some older systems, the stdlib distutils is the only place that includes the proper install schemes).

As far as I know, PyPA also uses PEPs for larger changes. A PEP doesn’t need to be limited to changes in CPython.

1 Like
12

PyPA uses PEPs for defining standards. We don’t use PEPs for changes relating to individual projects, no matter how large.

  • Deprecation of distutils from the stdlib is a core Python change, and should probably have a PEP, correct. That’s not for the PyPA to say, but for Python core devs and the SC, though.
  • Setuptools vendoring disutils is a grey area. It’s a project-only matter, but they are (as I understand it) deliberately shadowing the stdlib distutils, which goes against “normal” conventions. I’d say this doesn’t need its own PEP, but should probably be covered in the part of the distutils deprecation PEP that describes the transition process.

So I think one PEP, covering the removal of distutils from the stdlib, should be sufficient¹. I’m surprised that there isn’t one…

This is all just my personal opinion, of course.

¹ There’s a further complication, in that (I think) Python’s build process uses distutils itself, so it’s likely that distutils isn’t being “removed”, but rather “reserved for core use only”, with the user-facing version being handed to setuptools. But again, that needs clarifying and agreeing.

1 Like
13

setuptools is an updateble part of the standard python distribution, since it is included via ensurepip. So if distutils is removed from the stdlib in favor of one provided by setuptools, it should conditioned on setuptools as shipped being sufficient to build CPython, and it should be clear that any updated version of setuptools could continue to build CPython as well.

14

The fact that setuptools is present is an implementation detail of ensurepip and shouldn’t be relied on. See here.

15

It’s because no one has formally proposed it. As of right now I view it as setuptools has forked distutils.

Yes, that wrinkle needs to be solved somehow.

16

I can do the deprecation PEP. Everything that has been said here already about it sounds fine, so I doubt anyone will be surprised by the content.

I’ll draft something out and start a new thread, but should be fairly straightforward (especially when I start threatening to tag opponents as “core distutils maintainers” unless they withdraw their opposition :wink: )

2 Likes