Specify conditional dependencies in pyproject.toml

Packaging
1

I have a private package (in GitLab Registry) for my work, which has two dependencies: pandas and SQLAlchemy among others. Now due to recent updates, it can work when pandas<2 and SQLAlchemy<2, or when pandas>=2 and SQLAlchemy>=2. v1 of any one package is not compatible with the v2 of the other.

My question how do I specify such a dependency condition in dependencies section of pyproject.toml. Other dependencies of my package, or other dependencies of other codebases for which my package is a dependency can lead to pinning of either of the two packages. Based on the aforementioned constraints, I want to restrict the final solved version for other package.

I’ve tried specifying both as <2, or specifying both as >=2. Of course that works, but I’d like to use the new format whenever possible, and sometime pinnings caused by other dependencies are forcing the conflict. Can someone please suggest on how to define dependencies constrained on each other?

2

Dependency configuration isn’t dependent on whether it is specified in pyproject.toml, setup.py, or any other file. It is primarily dependent on the ability of the eventual installer (pip or any other tool) to understand the dependencies and arrive a compatible combination of package versions.

I don’t think pip has the ability to understand cross-package dependencies in this way. If pandas doesn’t express a dependency on SQLAlchemy itself, you’ll have to pin all of the dependencies more tightly and disallow usage of versions <2.

3

Well, “normally” this would be done by one package having an optional dependency on the other, and you depending on that optional. That said, I’m a bit surprised by your problem.

FWICS pandas 2.0.1 specifies the following optional dependencies:

postgresql = ['SQLAlchemy>=1.4.16', 'psycopg2>=2.8.6']
mysql = ['SQLAlchemy>=1.4.16', 'pymysql>=1.0.2']
sql-other = ['SQLAlchemy>=1.4.16']

So supposedly they should work with SQLAlchemy 1.x — are you sure the problem lies in pandas and not your code?

Unfortunately, Python dependency specifications do not support the kind of dependencies you’re looking for. The closest equivalent I can think of is defining two optionals like:

v1 = ["pandas<2", "SQLAlchemy<2"]
v2 = ["pandas>=2", "SQLAlchemy>=2"]

and then installing your package as either .[v1] or .[v2], depending on what you need. Ofc it’s only an ugly hack and requires you to make a conscious choice.

1 Like
4

I am quite certain. The issue happens when pandas get resolved to <2, >=2 is fine. pandas<2 did not have a specific bound on SQLAlchemy (see below), so pip install does the latest by default and then it fails because of this.

5

I was actually asking about the other way around — that you can’t use SQLAlchemy-1 with pandas-2.

6

I misunderstood, sorry. Actually the conflicts I faced always forced pandas<2 (and SQLAlchemy remained unrestricted), so never faced the opposite type of conflict.

7

Would it work to make an empty package with v 1.0 depending on 'pandas <2', 'SQLAlchemy <2', and v 2.0 depending on 'pandas >=2', 'SQLAlchemy >=2', and then have your real package depend on this metapackage? You might also want to specify >= 2, <3 while you’re at it.

That’s clearly not very convenient, but if you really want pip’s dependency resolver to be able to select either of those pairings, I think it should achieve that.

2 Likes