`pkgconfig` specification as an alternative to `ctypes.util.find_library`

Packaging
17

There are several reasons to do this:

  • Keeping proposals scoped tightly and on a single topic makes it easier to move them forward,
  • There are a number of types of users of metadata on native dependencies; things like static analysis tools will only need the metadata itself, not the dependency discovery at build time or runtime,
  • There are multiple ways of doing dependency discovery. pkg-config is the most prominent one, but certainly not the only one. I think it should be better supported, but I wouldn’t want to venture into making its use mandatory at this point. As a package author, I should be able to write something like dependency('mylib') in my build config files, and have the build system use pkg-config, CMake, configtool, or its own custom code to act as the “dependency provider” to discover mylib.

A separate backend isn’t needed for using pkg-config at build time. Any good build system/backend dealing with compiled code should be able to do this. If it’s about using pkg-config with ctypes.util.find_library at runtime specifically, then no build backend should be involved at all. find_library already has platform-specific behavior, and its semantics are pretty vague, basically “do your best to find this library”. pkg-config could be added to that directly as one of the ways to try and locate the library in case pkg-config is already installed. That would be backwards compatible and a sensible thing to do.

I agree, with the note that if the patch tries to use pkg-config when installed, it should be more generic and work for Nix and a number of other distros/scenarios.

I had a look at the list of patches @domenkozar linked to. The first one I recognized is psycopg, and that’s a great example here. It ships only a pure Python package/wheel, does not declare any dependencies in its pyproject.toml, but in its README has sudo apt install libpq5 and does this inside the package:

libname = find_libpq_full_path()  # uses ctypes.util.find_library
if not libname:
    raise ImportError("libpq library not found")

pq = ctypes.cdll.LoadLibrary(libname)

That is indeed pretty fragile. However, I think we indeed have all the ingredients to fix the issue:

  • PostgresQL (which provides the needed libpq) has to be declared as an external dependency in pyproject.toml[1],
  • distros can then use that info to have their packaging tooling add postgresql as a runtime dependency of psycopg (or at least, the packager has an easier time when reading the metadata and handling it manually),
  • this may already be enough for many distros (and indeed, on Arch Linux, ctypes.util.find_library('pq') returns a path to libpq.so.5)
  • distros with non-standard path can implement a rule to automatically add pkg-config as a runtime dependency too if there are any external dependencies in pyproject.toml,
  • that will fix the issue for Nix when find_library is actually trying to use pkg-config in addition to its current methods.

I think it’d be feasible to extend find_library now. Where it hits return None because it can’t find any library, the code could check if pkg-config is installed and if so, run pkg-config --libs libname in a subprocess call, then if that return a -L/path/to/some/libdir check that dir for the library. It’s pretty straightforward and should be backwards compatible, so would that even require a PEP?

  1. I’m not very familiar with PostgresQL, it’s possible libql is its own thing in all distros and that should be the name of the dependency instead. I see that in Ubuntu, it’s libpq5. ↩︎

4 Likes