# How can we keep pip from updating/uninstalling system libraries in /usr/lib/python? **URL:** \ **Category:** Packaging\ **Created:** [August 27, 2020, 4:04pm UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059 "2020-08-27T16:04:13Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [August 27, 2020, 4:04pm UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/1 "2020-08-27T16:04:13Z")
Hello, I’m not sure whether to discuss this on the setuptool, distutils or pip tracker, so I’ll ask here. The distutils (and so, setuptools) install has the concept of [install schemes](https://github.com/python/cpython/blob/d9a966ae08258da2ce2a432c943d8194760f09c4/Lib/distutils/command/install.py#L21-L65), which control where a package gets installed: system-wide, or to a user’s home. For a long time, Debian has a patch, [distutils-install-layout.diff](https://salsa.debian.org/cpython-team/python3-stdlib/blob/master/debian/patches/3.8/distutils-install-layout.diff) that adds more install shcemes, `unix_local` and `deb_system`, and implements logic to select them. The `unix_local` scheme installs to `/usr/local` rather than `/usr`. Fedora does a [similar thing](https://src.fedoraproject.org/rpms/python3.9/blob/master/f/00251-change-user-install-location.patch) in a more hacky/minimal (depending on your point of view) way. It also changes the install prefix from `/usr/` to `/usr/local`, in the default case. These patches assume that the stdlib is tied to an interpreter and can’t be changed. Now that setuptools is [replacing stdlib distutils with its own copy](https://setuptools.readthedocs.io/en/latest/history.html#v50-0-0), these customizations are starting to disappear. Is there a supported way to do customizations like this? I can describe Fedora’s reasons for the patch: it’s to minimize damage to the system caused by `sudo pip`. Since critical software including the system package manager is written in Python, when `sudo pip install` updates a library to an incompatible version, the package manager can stop working. And unfortunately, the Internet is full of advice telling people to use `sudo pip`. The [solution in Fedora](https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe) is to: - install system software (from RPMs) in `/usr` - make `pip` (i.e. distutils) install to `/usr/local` - in `site.py`, put both `/usr/lib*/python*` and `/usr/local/lib*/python*` (and their site-packages) on `sys.path` System software then uses Python’s `-I` option, which skips site customization and thus omit the `/usr/local`, so it’s not affected by changes from `sudo pip`. If newer setuptools replace distutils with its own version using a `.pth` file, the following can happen: - `sudo pip install -U setuptools`, putting an unpatched copy of `distutils` in `/usr/local` - `sudo pip install -U ...`, which installs to `/usr` directly, possibly breaking system software. Is there any supported way to solve the this issue? ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [August 27, 2020, 9:46pm UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/2 "2020-08-27T21:46:55Z")
> [@encukou](#): > > If newer setuptools replace distutils with its own version using a `.pth` file, the following can happen: > > - `sudo pip install -U setuptools` , putting an unpatched copy of `distutils` in `/usr/local` > - `sudo pip install -U ...` , which installs to `/usr` directly, possibly breaking system software. Just to clarify, this is happening with `pip install`, not just `pip install -e`? Is this because `pip` is directly using `distutils` to decide where to install things? > [@encukou](#): > > Is there any supported way to solve the this issue? If you are asking for support for patching / patched versions of `setuptools`, I would say “no”, though @jaraco does more supporting of `setuptools` than the rest of us combined, so he’s probably the final arbiter on that. I think reasonable ways forward that could be supported or semi-supported would be: 1. I think the relevant GH issue ended with, “We don’t want to carry a distro-specific patch, but it’s not inconceivable that this would be of sufficiently general interest that we can support customizing the default install scheme in certain circumstances”. I don’t know the details of how difficult it would be to support this sort of thing upstream without breaking a bunch of other stuff, though. 2. I think @jaraco always intended devendoring of `distutils` to be not terribly difficult, which is why [GitHub - pypa/distutils: distutils as found in cpython](https://github.com/pypa/distutils) exists. If the main problem is with `pip`’s use of `distutils`, it seems like you could devendor and carry the patch on your copy of `distutils`. I think this will work as long as we assume people are using the system `setuptools`. 3. Change `pip` so it no longer relies on `distutils` and patch that. This could also possibly involve moving the relevant install scheme stuff into `packaging` (or a new package entirely), which can be patched if it doesn’t directly support this. Sorry I am offering brain-storming and not real solutions. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [August 28, 2020, 12:24am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/3 "2020-08-28T00:24:55Z")
> [@pganssle](#): > > Change `pip` so it no longer relies on `distutils` and patch that. This could also possibly involve moving the relevant install scheme stuff into `packaging` (or a new package entirely), which can be patched if it doesn’t directly support this. From my (maybe incomplete) understanding, pip’s current usage of distutils can all be replaced by sysconfig (and setuptools for the obvious legacy setup.py stuff). I’ve been keeping an eye on this, but couldn’t really do it myself due to the amount of tedious work involved and lack of motivation (the current implementation “works” anyway). I’d happily help if anyone is interested in working on the transition though. ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [August 28, 2020, 2:12am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/4 "2020-08-28T02:12:22Z")
> [@encukou](#): > > more install shcemes, `unix_local` and `deb_system` , and implements logic to select them. Adding and selecting install schemes sounds like it could be solved with configuration instead of patches. Even better that the configuration would be different between root and users, if it were in the configuration file. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [August 28, 2020, 7:35am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/5 "2020-08-28T07:35:18Z")
I would love to see configurable “install schemes” to supplement the built in ones. This would simplify and rationalise things like the handling of `--target`, for example. But honestly, it’s never been something that’s mattered enough to me to do anything about it. I would support efforts in this direction, though. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [September 1, 2020, 9:52am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/6 "2020-09-01T09:52:59Z")
> [@pganssle](#): > > - I think the relevant GH issue ended with, “We don’t want to carry a distro-specific patch, but it’s not inconceivable that this would be of sufficiently general interest that we can support customizing the default install scheme in certain circumstances”. I don’t know the details of how difficult it would be to support this sort of thing upstream without breaking a bunch of other stuff, though. Do you have a link to the relevant GH issue? Unfortunately it’s hard to follow discussion in issues I’m not subscribed to, so I didn’t know about possible new ways of customizing install schemes :‌( I do agree that carrying distro-specific patches is not good. > [@pganssle](#): > > - I think @jaraco always intended devendoring of `distutils` to be not terribly difficult, which is why [GitHub - pypa/distutils: distutils as found in cpython](https://github.com/pypa/distutils) exists. If the main problem is with `pip` 's use of `distutils` , it seems like you could devendor and carry the patch on your copy of `distutils` . I think this will work as long as we assume people are using the system `setuptools` . Which CPython version is [GitHub - pypa/distutils: distutils as found in cpython](https://github.com/pypa/distutils) for? Stdlib code can generally be coupled to the rest of the stdlib, but I guess that’s already known from issues like [setuptools#2357](https://github.com/pypa/setuptools/issues/2357). Anyway, I can’t devendor distutils from the version of setuptools that users install from PyPI. > [@pganssle](#): > > - Change `pip` so it no longer relies on `distutils` and patch that. This could also possibly involve moving the relevant install scheme stuff into `packaging` (or a new package entirely), which can be patched if it doesn’t directly support this. Users can update pip from PyPI, so changing the system `pip` (or other libraries) would not solve all of the issues. So far we relied on the fact that the stdlib can _not_ be updated from PyPI… but with setuptools 50.0, that all changes :‌( > [@uranusjr](#): > > From my (maybe incomplete) understanding, pip’s current usage of distutils can all be replaced by sysconfig (and setuptools for the obvious legacy setup.py stuff). I’ve been keeping an eye on this, but couldn’t really do it myself due to the amount of tedious work involved and lack of motivation (the current implementation “works” anyway). I’d happily help if anyone is interested in working on the transition though. I’m interested in how that would work, since [install schemes are part of distutils](https://github.com/python/cpython/blob/d9a966ae08258da2ce2a432c943d8194760f09c4/Lib/distutils/command/install.py#L21-L65). (For reference, pip uses them in its [`locations.py`](https://github.com/pypa/pip/blob/0b18e21cbe64b1bd089e280fd8ba592cdbfcac61/src/pip/_internal/locations.py#L95).) Even if that is possible, it won’t change how older versions of `sudo pip` work. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [September 1, 2020, 9:58am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/7 "2020-09-01T09:58:25Z")
> [@EpicWink](#): > > Adding and selecting install schemes sounds like it could be solved with configuration instead of patches. Even better that the configuration would be different between root and users, if it were in the configuration file. @jaraco, does that sound like something that could be added in [GitHub - pypa/distutils: distutils as found in cpython](https://github.com/pypa/distutils)? How would configuration of selecting a scheme work? I’m skeptical about doing it declaratively. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [September 1, 2020, 10:40am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/8 "2020-09-01T10:40:52Z")
> [@encukou](#): > > I’m interested in how that would work, since [install schemes are part of distutils](https://github.com/python/cpython/blob/d9a966ae08258da2ce2a432c943d8194760f09c4/Lib/distutils/command/install.py#L21-L65). (For reference, pip uses them in its [`locations.py`](https://github.com/pypa/pip/blob/0b18e21cbe64b1bd089e280fd8ba592cdbfcac61/src/pip/_internal/locations.py#L95).) > Even if that is possible, it won’t change how older versions of `sudo pip` work. The same logic (at least the parts pip need) is also implemented (and IIUC better-maintained) in [`sysconfig`](https://docs.python.org/3.8/library/sysconfig.html), which is available in all supported Python versions. Indeed this wouldn’t fix the older pip versions shpiped with the OS, but everything will stay broken forever unless we start the transition at some point. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [September 1, 2020, 11:24am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/9 "2020-09-01T11:24:35Z")
Right. We can definitely do that in Python 3.10. Where should this be best tracked/discussed? Currently, to me it seems that the plans for distutils/setuptools are scattered across issues in several projects, so I’m not clear on what is actually happening. Would it be worth it for someone who’s following all this to write a PEP? ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [September 2, 2020, 9:00am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/10 "2020-09-02T09:00:44Z")
> [@encukou](#): > > Currently, to me it seems that the plans for distutils/setuptools are scattered across issues in several projects, so I’m not clear on what is actually happening. Would it be worth it for someone who’s following all this to write a PEP? I think there are multiple parts to this issue. 1. pip needs to “modernise” the install schemes logic to use `sysconfig` instead, so it installs wheels to thes correct location, and only uninstalls distributions in the right location. This does not need a PEP since it’s project-specific. I don’t think there’s an issue tracking this, but can open one if needed. 2. setuptools may or may not need to perform the same modernisation (either by switching to `sysconfig` or modifying the bundled `setuptools._distutils`), depending on the maintainers’ opinion on this. This only needs solving if setuptools intends to continue support for “legacy” installations (non-PEP-517 `setup.py install` and `setup.py develop`). 3. Implement “configurable install schemes” in `sysconfig`. This is the part that needs involvement from CPython core developers, and likely requires a PEP to specify and design. I’m not sure how this would look like TBH, but can probably offer opinions if someone lays out the foundation. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [September 2, 2020, 9:37am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/11 "2020-09-02T09:37:10Z")
I can also see other changes that seem PEP-worthy: 1. `distutils` should be deprecated and removed from the standard library (right?) 2. `setuptools` provides its own copy of `distutils` (overriding the stdlib copy which might be different across [OSes](https://github.com/pypa/setuptools/issues/2364) or [interpreters](https://github.com/pypa/distutils/commit/2f1fae1bfceb88cf812a45c44cd18ce604b69c69) or [versions](https://github.com/pypa/setuptools/issues/2357) – on some older systems, the stdlib distutils is the only place that includes the proper install schemes). As far as I know, PyPA also uses PEPs for larger changes. A PEP doesn’t need to be limited to changes in CPython. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 2, 2020, 10:00am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/12 "2020-09-02T10:00:01Z")
PyPA uses PEPs for defining standards. We don’t use PEPs for changes relating to individual projects, no matter how large. - Deprecation of `distutils` from the stdlib is a core Python change, and should probably have a PEP, correct. That’s not for the PyPA to say, but for Python core devs and the SC, though. - Setuptools vendoring `disutils` is a grey area. It’s a project-only matter, but they are (as I understand it) deliberately shadowing the stdlib distutils, which goes against “normal” conventions. I’d say this doesn’t need its own PEP, but should probably be covered in the part of the `distutils` deprecation PEP that describes the transition process. So I think _one_ PEP, covering the removal of distutils from the stdlib, should be sufficient¹. I’m surprised that there isn’t one… This is all just my personal opinion, of course. ¹ There’s a further complication, in that (I think) Python’s build process uses distutils itself, so it’s likely that distutils isn’t being “removed”, but rather “reserved for core use only”, with the user-facing version being handed to setuptools. But again, that needs clarifying and agreeing. ---
**Author:** ![mattip](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mattip/32/330_2.png) [@mattip](https://discuss.python.org/u/mattip)\ **Post date:** [September 2, 2020, 10:17am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/13 "2020-09-02T10:17:36Z")
> [@pf\_moore](#): > > (I think) Python’s build process uses distutils itself, so it’s likely that distutils isn’t being “removed”, but rather “reserved for core use only” setuptools is an updateble part of the standard python distribution, since it is included via `ensurepip`. So if distutils is removed from the stdlib in favor of one provided by setuptools, it should conditioned on setuptools as shipped being sufficient to build CPython, and it should be clear that any updated version of setuptools could continue to build CPython as well. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 2, 2020, 11:18am UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/14 "2020-09-02T11:18:15Z")
> [@mattip](#): > > setuptools is an updateble part of the standard python distribution, since it is included via `ensurepip` . The fact that setuptools is present is an implementation detail of ensurepip and shouldn’t be relied on. See [here](https://www.python.org/dev/peps/pep-0453/#automatic-installation-of-setuptools). ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [September 2, 2020, 6:45pm UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/15 "2020-09-02T18:45:04Z")
> [@pf\_moore](#): > > So I think _one_ PEP, covering the removal of distutils from the stdlib, should be sufficient¹. I’m surprised that there isn’t one… It’s because no one has formally proposed it. As of right now I view it as setuptools has forked distutils. > [@pf\_moore](#): > > There’s a further complication, in that (I think) Python’s build process uses distutils itself, so it’s likely that distutils isn’t being “removed”, but rather “reserved for core use only”, with the user-facing version being handed to setuptools. But again, that needs clarifying and agreeing. Yes, that wrinkle needs to be solved somehow. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 2, 2020, 11:27pm UTC](https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059/16 "2020-09-02T23:27:45Z")
I can do the deprecation PEP. Everything that has been said here already about it sounds fine, so I doubt anyone will be surprised by the content. I’ll draft something out and start a new thread, but should be fairly straightforward (especially when I start threatening to tag opponents as “core distutils maintainers” unless they withdraw their opposition 😉 )