Playing nice with external package managers

Packaging
1

At PyCon this year, there was a lot of discussion and grumbling about how pip can fight against python installations that are managed by other tools (apt, yum, conda, etc.). It was thought that the INSTALLER file in the dist-info folder for a given package’s installation was already something that pip would respect and refuse to alter packages where INSTALLER did not read “pip.” After further discussion, this was decided not to be the right way, and another metadata file more specific to this purpose was desired.

This thread is to discuss and decide on that other file, and come up with a PEP for standardizing on it.

Possible notes that I missed are in @sumanah’s post at Drawing a line to the scope of Python packaging

2

Can you say a bit more about the goal here? My first thought would be that the goal is to avoid situations where pip and some other package manager are stepping on each other’s toes and corrupting their respective package databases. But a package database is a single coherent data structure representing a whole environment, so I’d expect this to be handled by marking an entire python environment as managed-by-pip versus managed-externally.

If you tell pip it can’t touch specific packages, then you can still get in a mess:

  1. Pip installs some package
  2. Apt installs the same package, partially blowing away Pip’s copy
  3. Now you have a mix of two different versions installed in the same environment
3

the goal is to avoid situations where pip and some other package manager are stepping on each other’s toes and corrupting their respective package databases.

That sums it up pretty well, actually.

But a package database is a single coherent data structure representing a whole environment

I don’t think this is representative of the reality that we have today (though I really wish it was). People use pip in conda environments, and they also use pip in system environments. When they do, they often encounter problems, which the maintainers of the external tools have to deal with (“I pip installed tensorflow, and conda broke! It must be conda’s fault!” or worse: “I sudo pip installed tensorflow and now ubuntu won’t boot! Ubuntu sucks!”). I see this topic as a universal agreement to stop trampling packages that are managed by something else. If stopping completely is not feasible (which it probably won’t be), then lay out standards for how to tell users what needs to be done, and that the thing they are doing is unsafe.

I got the sense at PyCon that this was a settled matter that just needed some implementation elbow grease. @ncoghlan @dstufft @kalefranz and Matthias from Canonical were in most of the discussions. Maybe they have more to add - I’m kind of going on second hand discussions with Kale.

1 Like
4

Relevant here: https://github.com/pypa/pip/issues/5605

We’d discussed that we’d use a new “MANAGED-BY” file. I believe @ncoghlan made initial notes on this. If he’s okay with it, someone who has a copy of those notes should share them here. :slight_smile:

1 Like
5

These are my post-PyCon notes - the redacted bit was a specific name in the original emailed notes, but I want to let them chime in themselves if they’re still planning to work on it (since a lot of us have been in that situation of volunteering to work on something, and then just never finding the time to get to it).

Next steps:

Why not use INSTALLER:

  • We want Python level tools to be cooperative by default (install
    with pipenv, uninstall with pip, reinstall with poetry, uninstall with
    pipenv, etc)
  • Making INSTALLER strict would either interfere with that, or else
    we’d see everyone treating “pip” as a magic string that meant “shared
    management”, and we’d lose the tracking of which installer actually
    did the installation

Using a new MANAGED-BY file instead:

  • File is absent by default, indicates shared management by any tool
  • If MANAGED-BY is present, then management is limited to tools that
    understand and support the external management scheme named in the
    file
  • Examples: Debian would set ‘dpkg’; Fedora would set ‘rpm’; conda
    would set ‘conda’
  • Tools that don’t recognise the scheme would refuse to uninstall,
    upgrade, or downgrade the package
  • Tools that convert Python packages to system packages should add the
    new metadata file with the appropriate contents

Legacy metadata formats:

  • egginfo files must be treated as if MANAGED-BY was present with an
    unknown management scheme
  • egginfo directories must be handled the same way as distinfo
    directories (i.e. shared management by default, specific scheme can be
    named in MANAGED-BY file)
6

So does this mean that this plan depends on convincing the dpkg, rpm, etc. maintainers to check for this file? Do we know if they’re interested in doing that?

7

We’ll be happy to switch RPM macros to this standard! (I think I can speak for Fedora’s Python maintainers here, and I believe other RPM distros will join/follow.)

cc @hroncok

8

It’s not just macros – if we take what @ncoghlan said literally, it means that when you’re installing, say, the python-requests rpm, then rpm would need to first check for /usr/lib/python*/site-packages/requests*.dist-info/MANAGED-BY, and if it exists and contains some string besides rpm then it should refuse to install that rpm. So I think this would require patching the rpm tool itself to add a special case check? (Or are macros powerful enough to do that? I thought they were just for setting up the package, not for controlling whether it’s installed at all, but I’m definitely not an rpm expert.)

9

Yes, there will be resistance if we need to add a special-case to RPM itself :​(
At least on Fedora, though, pip installs into /usr/local/lib/ while RPM installs to /usr/lib/.

10

FWIW, it does degrade gracefully for these tools since the main issue for us has been pip modifying packages being managed by a different tool.

As long as this file as added by the distros, it’s an improvement on status quo, regardless of whether other package managers get updated for this scheme (since pip would stop fiddling with packages that are managed by those tools). :slight_smile:

1 Like
11

The in-person discussion included Matthias Klose on the Debian side, and Kale Franz on the conda side, and then I gave Petr a heads up for Fedora when I distributed the original set of notes.

Aside from conda, where my understanding is that the developers actively want folks to be able to install non-conda packages directly from PyPI without risking inadvertent upgrades of conda-managed packages, we’re not really expecting platform installers to respect MANAGED-BY - if the platform package manager provided the Python installation, then it can reasonably assume it has full control over that installation.

Instead, we’re mainly offering the platform tool developers the opportunity to populate the file and have the Python level platform independent tools refrain from breaking people’s Linux installations, even if a user does run ye olde “sudo pip install break-my-distro-please” command. (And even for conda, I’d expect them to end up offering their users a way to check their environment for installations that overlap with projects available from their conda channels and replace the PyPI version with the conda-managed version).

12

I don’t expect RPM and such to pay attention to the MANAGED-BY file. We basically have two classes of installs from the POV of Python packaging:

  • Managed by a tool that uses the relevant PEPs as their primary database of installed packages and related metadata.
  • Managed by an external tool that might happen to also emit the files that the relevant PEP databases uses because the Python level tooling needs it to function properly (not just packaging tools, but other runtime tooling as well).

For the first case, we want all of these tools to largely be interoptable. If you install something with pip you should be able to then uninstall it with totally-not-pip. As long as all of the tools in this category are using the same database for what they consider installed or not, and they all implement the relevant specs, then these tools should largely be interchangeable.

For the second class, these tools are largely NOT interoptable, and it doesn’t even make sense for them to be. apt and rpm are unlikely to ever be in a situation where they’re both installed on the same system and trying to install into the same set of directories. The closest thing to “crossing the streams” in this world would be installing something like conda or Linuxbrew on a system that already has apt or rpm or similar, but in every case of those I can think of, those tools are installing to an entirely different location and don’t attempt to touch each other files at all.

So interactions between two tools within the same category is already largely a solved problem, through one mechanism or another. What we care about really is interactions between tools in different categories.

Within those interactions between categories we have two “directions” the interaction can go, those interactions are roughly a “type 1” (e.g. pip) installed thing being overwritten by a “type 2” (e.g. apt, rpm, etc) install and the reverse, a “type 2” installed thing being overwritten by a “type 1” installed thing.

MANAGED-BYis largely solving the second case, and it does it by teaching the type 1 tools how to understand a special marker that type 2 tools can easily be modified to write.

Of course, we could envision a world where this same system could be used for the inverse, and keep rpm/deb/etc from clobbering something installed by something like pip. However it’s unlikely that such a thing will ever gain traction, because these tools work with far more things than just Python (and in most cases it doesn’t even know something is Python, it’s just dropping files in predetermined locations).

That does mean we’ll need a different solution for the two directions that these conflicts can happen in. However I think that is inevitable given the realities of the capabilities of the two different “types” of tools, and the politics surrounding them that control what kind of changes are possible or not.

2 Likes
13

I didn’t see (maybe I missed) some references above, but didn’t the same apply to pip crossing apt/rpm as well? IIRC sudo pip installs to /usr/local/{bin/lib}, and officially managed apt packages install to /usr/{bin/lib}.

I didn’t see it explicitly distinguished, so just in case: The most likely confusion (including pypa/pip#5605 IIUC) is not that the lib installations gets overwritten, but that pip installs another distribution that takes precedence over the existing (installed by another package manager) distribution. I do not think a MANAGED-BY file (or any other in-distribution marker) would solve this.

14

That’s not something we’re trying to solve with this really. We’re not stomping over the same files, the user is just electing to install something that takes precedence. It might break their system but it doesn’t have the same “two systems fighting over who owns what files” problem.

15

I likely didn’t express my concern clearly. I cannot think of a widespread example of pip actually stumping over another package installer. I also have not seen any real examples above that demonstrates this (unless I missed something). The worry I’m having is that we are solving a phantom problem that does not actually exist.

Edit: I double-checked nad realised that EPEL’s python-pip does install pakages to the same location as yum itself, so there indeed are examples that actual overwrites happen. Sorry for the false alarm.

16

apt-get install python-requests && pip install --upgrade requests

pip will uninstall the version installed by apt-get, then install the version pulled down from PyPI. The desired outcome is pip will do nothing with the version installed from apt-get, and will install the version pulled down from PyPI to /usr/local (the /usr/local thing still requires a patch on the distro side though, making that official is a future enhancement In cases where that patch doesn’t exist and we’re trying to install to the same place we should just fail).

17

Actually, pip already does exactly what you say in this situation, which is what prompted my initial misunderstanding:

$ apt-get install -qqy python3-requests && pip3 install --upgrade requests
[a bunch of messages from dpkg installing python3-requests]
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests)
Installing collected packages: idna, certifi, urllib3, requests
  Found existing installation: idna 2.6
    Not uninstalling idna at /usr/lib/python3/dist-packages, outside environment /usr
  Found existing installation: certifi 2018.1.18
    Not uninstalling certifi at /usr/lib/python3/dist-packages, outside environment /usr
  Found existing installation: urllib3 1.22
    Not uninstalling urllib3 at /usr/lib/python3/dist-packages, outside environment /usr
  Found existing installation: requests 2.18.4
    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed certifi-2019.6.16 idna-2.8 requests-2.22.0 urllib3-1.25.3

Notice that pip skips uninstalling the requests distribution from apt.

Uninstalling pip’s copy of requests also does not break APT’s. There are some dependency problems, but they are caused by pip’s copy taking precedence (and as you said, not our problem), not APT’s copy being broken by pip.

$ python3 -c 'import requests; print(requests.__file__)'
/usr/local/lib/python3.6/dist-packages/requests/__init__.py
$ pip3 uninstall -y requests
Uninstalling requests-2.22.0:
  Successfully uninstalled requests-2.22.0
$ python3 -c 'import requests; print(requests.__file__)'
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.3) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
/usr/lib/python3/dist-packages/requests/__init__.py

So this is not really a problem between APT and pip, at least in recent Ubuntu versions. I’m not sure who takes the credit though :slight_smile:

That said, RPM users are not as fortunate (as I reaslised in the previous edit). So (un)fortunately the issue discussed here is still very valid.

18

Pip doesn’t do that, Debian’s patched version of pip does that.

19

Maybe the “right fix” here is to have upstream pip adopt that patch? That is, make upstream pip install to /usr/local [under the circumstances when it would currently install to /usr].

20

Sorry there’s some confusion here.

On Debian, all versions of pip will install to /usr/local, becasue Debian has patched Python to tell any Python level installation tool the correct place to install things to is /usr/local. Where pip itself installs things to is not what this proposal is about fixing though.

On Debian, the debian supplied version of pip will refuse to touch /usr, because they’ve patched it to do that. That patch cannot be applied to upstream pip, because it only currently makes sense in the context of Debian’s other patches to other software (namely Python). The Debian patch to pip to not touch /usr only works because they’ve also patched Python to do the above. We could upstream the patch to Python, but that would only solve the problem for Python 3.9+ whereas MANAGED-BY solves it right now, for all versions of Python.

1 Like