PEP 725: Specifying external dependencies in pyproject.toml

[Lecris (Cristian Le)]

Let me try to rephrase some of the ideas.

Naming convetions

The main issue is that if you rely on something like pkgs:generic/spglib you are guaranteed to encounter ambiguities with different naming conventions, different packaging policies, etc. My main argument is to discourage a format like pkgs:generic/spglib as much as possible, with either:

• Specifying specific variants for each distro debian:libsymspg2-dev, arch:spglib, etc.
• Support the non-pkgs variant, i.e. cmake(spglib) which automatically translates to spglib-devel or any naming convention that are present on rhel, open-suse or any rpm-based packaging systems. If the distro cannot expand it, 1) it should, 2) it could use distro-specific labels, 3) how much do we want it to be actually supported

Another issue related to the naming convention is who distributes the name maps? Do the distros have to maintain such a name-map for their packages? Would it be the build-system’s responsibility to maintain them for all distros? Would it be on the consumer side to define for each variant?

On the other hand the cmake(Spglib) artifact does not require maintaining such a map, it can simply use

$ dnf whatprovides "cmake(Spglib)"
Last metadata expiration check: 0:00:08 ago on 2024年08月29日 18時30分39秒.
spglib-devel-2.2.0-2.fc40.i686 : Development files for spglib
Repo        : fedora
Matched from:
Provide    : cmake(Spglib) = 2.2.0

Than even when the naming changes e.g. a better spglib-ng comes along with compatible API, than we don’t need to update anything.

Another approach would be to rely on the known paths and ask dpkg, spack etc. to search for the package that provides that file.

$ dnf whatprovides "*/SpglibConfig.cmake"
Last metadata expiration check: 0:04:59 ago on 2024年08月29日 18時30分39秒.
spglib-devel-2.2.0-2.fc40.i686 : Development files for spglib
Repo        : fedora
Matched from:
Filename    : /usr/lib/cmake/Spglib/SpglibConfig.cmake

Ambiguous request

Let’s say we are requesting spglib, package, then what exactly do we need from it? Do we need the CMake files, then how do we know that the other packages provide the CMake files? Do we need the runtime files (more commonly this is the case for pre-processors: swig, fypp, etc.)?

This would again be resolved if we don’t request a specific package, but instead the precise artifact that we want cmake(Spglib), pkg-config(spglib), /usr/bin/swig, etc. with some syntactic sugar to make a PURL.

Python libraries can advertise what they depend on

At this point let’s say we have a successful build, a lot of the times we do not need to include the runtime dependencies. @tiran mentioned a tool that can be used within python which other build-systems can call. The idea is that after you build, you can introspect the libraries that it needs to run, and if you go with the artifact approach instead of the package mapping, than you can simply ask the system what package gives the relevant library

$ dnf whatprovides "libsymspg.so.2"
Last metadata expiration check: 0:17:40 ago on 2024年08月29日 18時30分39秒.
spglib-2.2.0-2.fc40.i686 : C library for finding and handling crystal symmetries
Repo        : fedora
Matched from:
Provide    : libsymspg.so.2

This is especially useful because there can be various compatibility packages, different versions of libraries, etc. which would not be apparent from the package name alone.

And in the case of runtime dependency, both dep and rpm systems allow to introspect this information, and the consuming project does not have to maintain these, you just need the build-system to run something like elfdeps and populate the relevant Python metadata files in dist-info

Beware of how each package builds

Many projects would have fall-back build processes, e.g. in CMake you have FetchContent(FIND_PACKAGE_ARGS) which runs a find_package and if it fails it downloads the dependency from FetchContent, or they would have bundled sources for the dependencies etc. This should be taken into account since you may not want to use those fallbacks, and just because a project was built after external-deps-build injected a dependency, that does not guarantee that it was used properly.

---

@rgommers

Not quite. Shared libraries and .cmake/.pc files inside a Python package are not usable at build time by another package at all, since they’re not on the search paths of the relevant tools.

They actually are. This is what I’ve been working on recently in spglib and scikit-build-core is creating 1st-class support for that. cython-cmake shows partially this integration.

@rgommers

If a Python package needs the C library of spglib, that’s an external dependency that cannot be obtained from PyPI. So it’s actually not that ambiguous: if a Python package has spglib in build-system.requires or project.dependencies, it’s the Python package only (so in Fedora always python3-spglib).

I don’t think it is that easy. Take rapidfuzz for example. It supports both a compiled C library and python-only implementation. How do we tell it that we want the python library rapidfuzz with or without the compiled library?

@rgommers

I’m not sure I understand this point. Do you mean

1. Link to a shared library at build time and ensure it’s then available at runtime. Or,
2. Don’t link to it at build time, but access it at runtime through something like ctypes?

For (1), the answer would be to put the PURL for the package that contains the shared library in external.host-requires. For (2), put that PURL in external.dependencies.

This is about “Python libraries can advertise what they depend on”.

@rgommers

Now the remaining ambiguousness is this: what if the other Python package needs the Fortran interface? This is something that can’t be cleanly expressed, and I don’t see a great way of taking care of such corner cases.

Similarly with requesting the artifact instead of the package

$ dnf whatprovides "pkgconfig(spglib_f08)"
Last metadata expiration check: 0:43:01 ago on 2024年08月29日 18時30分39秒.
spglib-fortran-devel-2.2.0-2.fc40.i686 : Development files for spglib with Fortran bindings
Repo        : fedora
Matched from:
Provide    : pkgconfig(spglib_f08) = 2.2.0

The provides does not support CMake components because those are quite more tricky to introspect statically, but there are plenty of other artifacts that can be used.