Hardware-dependent dependencies

Version v1 of package A depends on a certain version range of package B. However, there are several ways to install the same version of package B, depending on the hardware of the target system (e.g. is there a GPU installed, which cuda version is needed etc).

Now everytime A gets installed, some default version of B gets installed automatically, but this is often the wrong version. So one has to uninstall and manually install the correct version for the system.

Is there any approach to solve this problem in a more automatic way?

What do you have in mind? No, seriously, what do you think a potential solution could look like?

The ‘proper’ solution, to my mind, is for the package with many variants not to have many variants, but to test for hardware and for optional dependencies at runtime. Provide an additional A_cuda package which nvidia users can install, and make A check for A_cuda (and import/use it if available).

Alternatively, don’t provide binary wheels and configure the package depending on the available hardware at build time. (Don’t do this, making everyone compile the code is a usability nightmare)

1 Like

OK, I am talking about Pytorch as package B, so it will be hard to convince them to do things differently :slight_smile:
Pytorch depends on the CUDA GPU libraries if a GPU is installed and packages them. When installing with pip install torch a CUDA version that depends on the torch version is installed by default. But this version needs to match the GPU driver installed on the system to work.
So it is possible to also install versions like torch==1.9.0+cu111 where cu111 is for a specific CUDA library version.

I am not sure if it would be feasible for torch to include all CUDA library versions (which would be huge) and if it would be feasible to determine which of them to install automatically and I have no influence on how Pytorch does things anyways. But I can influence how A does things.

So assuming I know the cuda version required, and I am pip install package A, would there be a mechanism that could automatically update the dependency to something like “latest version of torch that comes with cuda version x”? Is there a (recommended) mechanism at all for dynamic selection of dependency versions during installation?

Just listing something like torch>=x.y.z in the requirements.txt and installing dependencies from there is not enough for this.