Is there an official stance on the installation of circular dependencies? Presently it seems that pip does permit this, although it’s unclear if this is to support distributions with bad metadata or by design.
Any insights are appreciated.
Python and Python packaging tools both support circular dependencies just fine, as long as you’re careful with how you do your imports.
What is wrong with circular dependencies between packages?
It’s totally valid that a part of package A depends on a part of package B, and a different part of package B depends on a different part of package A.
As long as they are both not build dependencies of each other there should be no issue with installation.
My question relates specifically to installation of distributions into an environment. So wheels (or sdists), not the importing of packages.
Maybe an example would help me explain:
If there is a clean environment, and pip begins to install into site-packages, won’t circular dependencies between wheels result in overwriting folders during installation?
Maybe an even simpler way to describe my question.
Are circular dependencies of Requires-Dist metadata valid? What is the expected behaviour during resolution and installation if this is valid.
Yes.
No because I believe pip doesn’t do installation until it’s done resolving what it needs to install. And since pip caches the wheels it downloads it already avoids excess downloads.
2 Likes
Please remember that Python packages aren’t purely used via native Python tooling but are also included in Linux distributions. While they inevitably have to support circular dependencies, they’re never a good idea and usually result in bad UX.
In Gentoo, the biggest part of the problem is that circular runtime dependencies generally imply circular test dependencies. This effectively means that it’s impossible to test either package before installing both. Effectively, you can’t trivially protect a production system from breakage due to bad upgrade because you can only test after it’s already upgraded.
1 Like
Yes.
The expectation is that you’ll end up with an installed set that includes all of the resolved versions. The only limitation really is that the set has to be resolve-able, but we can resolve circular dependencies. This works because we don’t need to care about interim states being useful and we don’t need to use the packages to install them. If foo depends on bar and bar depends on foo, we can just (behind the scenes) install foo (creating a broken environment) and then install bar (fixing the broken environment).
Build dependencies are different, because we need the interim states to be useful and we’re actually going to use the packages we’re installing to do the build. So if there’s a circular build dependency then that fails.
1 Like
Thanks @dstufft that makes sense.
So cycles between packages defined in project.dependencies and project.optional-dependencies can be circular.
Packages defined in build-system.requires can’t be circular.
Is that the correct way to think about it?
Possibly. I’m not 100% convinced that build dependencies can’t be circular, it’s just that if they are, you’d better have one of them available in pre-built (wheel) format to break the circularity.
What I’m not clear about is why you need a hard and fast rule. The reality here is much more practical - if the installer can make it work, it will.
2 Likes
Certain graph-based build systems such as bazel don’t appreciate cycles in their DAG of dependencies. They’re evaluated lazily on demand, so there is no ahead of time installation. Nix is another one that works like this I believe.
I suspect some of the Gentoo problems mentioned above are similar.
So it’s just a good thing to know if it should be expected for those building tools that process python packages to know that they need to handle metadata cycles in the graph.
I wonder if conda handles them…
I believe Conda is similar: run-time dependencies can be cyclic, but build dependencies cannot.
1 Like
To be technically precise, build dependencies can be circular, as long as there’s a wheel somewhere in the cycle to break the chain. So if you’re writing a tool that assumes that there’s no cycles in the dependency graph, it might fail in some pathological cases.
1 Like