Soft-deprecating `ctypes.util.find_library`

Core Development
1

tl;dr: I propose to soft-deprecate ctypes.util.find_library in 3.15. It’ll stay where it is, but it won’t get developed further.

The purpose of the find_library() function, according to the docs is to locate a library in a way similar to what the compiler does. The word similar does a lot of heavy lifting.

Unfortunately, platforms don’t provide an API for this task, so the function ended up implemented as a bunch of heuristics, and so it has the typical problems of a heuristic-based function in stdlib:

  • any change has a real chance of breaking working code
  • it can practically never be complete (especially on unsupported OSes and “new” Linux configurations – containers, Nix, AppImage…)

I recently added an explicit warning that’s been implied by (old & stable) docs on Linux behaviour:

Note that if the output of [the programs /sbin/ldconfig, gcc, objdump, and ld] does not correspond to the dynamic linker used by Python, the result of this function may be misleading.

But, that’s far from the only case when it’s misleading.

It is time to officially tell users to consider something else – and possibly fork the function and build something else on PyPI, free from CPython’s platform support policy, release cadence, and problems with pinning a known-good version.
If find_library is blocking you from running some app on, say, musl [1], perhaps the app’s maintainers will be more likely to accept a PR if you show them that it’s is officially nailed to the perch.

  1. or a system that mixes musl & glibc, which is a real user need ↩︎

3 Likes
2

It’s been a while since I used it, but isn’t find_library used on Windows, maybe more reliably than on Unix systems? I’m sure @steve.dower would know better than me on this.

If there’s an alternative that we can recommend, let’s do so. If there isn’t, and the advice is “you’re on your own”, then I’m less comfortable - have we done any code scans to find out how often this function is used “in the wild”?

1 Like
3

Haven’t refreshed my memory, but I’m 99% sure it’s meaningless on Windows and only works on Unix. Windows doesn’t have a concept of “locate library like the compiler does” - the answer to that is “search the paths provided specifically to the compiler for that purpose”, whereas Unix comes with a variety of tricks and implied names.

load_library is the important one on Windows (and I’m also pretty sure it supports the flags that would let you use it to locate a library without actually executing it, which is the “correct” way, but rarely of any value).

4 Likes
4

As a data point, grepping through the Meta monorepo, I found approx 100 instances of first-party code using this API, and approx 1,000 hits from third-party libraries.

5

First, thanks for doing that doc update. People often don’t realize how complicated dynamic library finding can be or how specific build time tool chain dependent it can actually be. Until they’re bitten by it and don’t realize they’re debugging using non-matching tools and scratching their head. So calling it out is a good help to the world!

Playing devils advocate here: Is this really a soft deprecation? Are we trying to say that we’ll never change the function’s behavior again in the future? Or just that we can never guarantee it does what people need and will not do the wrong thing for them.

Because one reading of soft-deprecation could unfortunately be “oh good, this behavior will never change again, I can rely on it being exactly what it is today forever” meaning when find a reason to update it for our own needs it might come as even more of a surprise.

We do use find_library within the standard library in several places. So I could see us “enhancing” it for our own needs, which a soft-deprecation suggests we won’t do. We could create an internal private function to use instead internally where we do behavior changes but that’d feel like over engineering.

When telling people why not to use it, it’d be good to be able to advise people what they should do instead rather than leaving it as a “:shrug: we don’t know, every environment is unique, good luck!” - which if we were actually able to answer that, we could improve our implementation.

4 Likes
6

find_library emulates how the compiler and linker do things (as hand-wavy as that is). On windows, yes, it’s searching PATH and looking at file extensions, though there’s an ominous comment suggesting that here, too, the rabbit hole goes deeper:

# See MSDN for the REAL search order.

On libc, we ask a helper program, praying that it’s in sync with the linker, and parse the human-readable output. Other platforms do one and/or the other, each with its own spelling.
On musl, there’s no helper program, so it would be back to searching *_PATHs, except you parse the ELF header instead if looking at extensions. (And after getting the path – a musl-specific detail of the kind that musl devs don’t like people relying on.)
This topic is me rejecting the musl PR.

I think soft deprecation is the best label to apply.
We can fix bugs. We can un-deprecate. I think something like supporting a new platform wouln’t really be off-limits, if it’s a two-line change.
But keeping existing environments/use cases working should be a priority.

Umm, not really. It’s used in ctypes, ctypes tests, and in _ios_support.py.
In the stdlib it’s easy to switch to a C extension, and find_library is too unreliable to use outside specific platforms.

For all supported platforms, and a few unsupported ones, the best answer we can give is find_library.
But this doesn’t scale: yes, each platform does need a special case, and anyone can add a new “platform" (as musl did). And since we don’t on test those platforms, CPython is not a good home for the code.
Like platform.linux_distribution or urllib.request, this use case is better served by a package that you can update/pin independently of CPython.

What they should do is stop using find_library – which in practice means that the musl/nix/AppImage/… integrators send PRs to projects that use find_library, adding config setting or another workaround – and failing that, they write a PyPI package and patches/PRs to switch to that. As a ctypes maintainer; I figured that the best way I can help them is bless their efforts. (I can help in other ways too, but reviewing musl-specific code isn’t really it.)

From the CPython side, yes, that’s saying “good luck!”. But the alternative is either we say “no”, or we keep building a Jenga tower where people expect a ladder. Or keep the issues & PRs open without a way forward.

2 Likes
7

Right, but the compiler and linker don’t do this on Windows - the loader does it at load time. The linker has its own variables (either LIB or LIBPATH, one is for .NET and the other native, but I never remember which is which), though these are usually not set in any way that Python could expect to make use of. What gets stored by the linker is the same library name that you’d pass to this function, and it doesn’t try to resolve things any more than that.

But overall, I agree that this belongs with distutils, which means we shouldn’t expect the stdlib to keep pace with the range of platforms out there and everyone (apart from the stdlib, of course) would be better off relying on a separate library that can be updated more frequently.

1 Like
8

Ah, right. Compiler, linker, and loader. Wherever the complexity is. Part of the problem with platforms not providing this functionality is that it’s not well defined.
Remember GH-67794?

9

You can have “well defined” as long as you’re also okay with “trivial escalation of privilege through search path hijacking” :upside_down_face: (or at least okay with “everything that came before is totally broken”).

1 Like
10

Thanks for entertaining my thought process. I’m in favor of this soft deprecation.

1 Like