Is there a neat way to detect that a set of packages don't support the latest of every package?

[14:27:10] is there a neat way to detect that a set of packages don’t support the latest of every package?

[14:27:56] eg here github.com/pytest-dev/pytest/pull/7721/files#diff-b91f3d5bd63fcd17221b267e851608e8R120-R128

[14:28:09] I want to fail the test if we don’t get the latest version of all these packages

[14:34:02] In fact is there a way to list all the packages you have installed with available updates, and those you can upgrade to and those you can’t?

[15:02:30] <McSinyx[m]> I think there’s a way, but not solely using pip

[15:02:49] <McSinyx[m]> wait I was wrong

[15:04:24] <McSinyx[m]> or sort of wrong

[15:05:32] <McSinyx[m]> the following should work: [ $(pip list -o | wc -l) -eq 2 ]

[15:06:54] <McSinyx[m]> but it’s kind of a hack because pip may print warning, so maybe a less trivial check than wc or giving pip -qqq may help

[15:49:29] McSinyx[m]: is there a way to find out what’s holding back an upgrade?

[15:50:17] <McSinyx[m]> AFAIK, other than reading the backtracking process, then no

http://kafka.dcpython.org/day/pypa/2020-09-13#14.27.10.graingert

I’d like a command that shows me what packages I can/can’t upgrade and why, that has an exit code that can be used in CI for checking that all the packages are the latest version with no conflicts

I don’t think there is anything like this available, but it should be possible to be done with importlib.metadata.

In addition to importlib.metadata, you also need a PEP 503 client to detect whether a package is latest. There are two libraries I know of:

I’ve personally only used mousebender in internal applications, but the client logic is so straightforward I’m quite confident both libraries got it right.

1 Like

to detect whether a package is latest.

is there a way to get the incompatible packages?

Incompatibility means multiple things in Python packaging. A package that does not satisfy dependencies of other currently installed packages can be simply downloaded and inspected without installation. Binary distributions (wheels) have static metadata that can be read directly. Source distributions are more problematic; PEP 517 has a hook to generate only metadata without compiling the whole package (prepare_metadata_for_build_wheel), but not every package implement the compilation well to support that. I believe importlib.metadata supports reading from .whl out of the box, and has privisional support for prepare_metadata_for_build_wheel.

A couple of my projects might help here:
https://github.com/jumptrading/luddite (checks whether package(s) are latest)
https://github.com/wimglenn/johnnydep (pretty-prints dep trees)
Neither is giving you exactly what you want, but they might give you some pieces that you need. Hope it helps.

1 Like

I’d use the pip-tools package pip-compile command to get a resolved list of transitive dependencies and their versions, and then check that each of those is the latest version available.

Note though that this is going to be fairly slow, and you might have problems where e.g. IPython has dropped Python 3.6 support (good on them!) but you still have some CI jobs running on those older versions.