The build and runtime tools already understand the thing I’m proposing. Let me see if I can sketch this out in a bit more detail. Suppose you have native library libfoo, and a native library libbar that depends on libfoo.
The install process for libfoo (e.g. make install, or the equivalent for cmake or whatever else you may use) will create, in the target lib directory, a structure like
lrwxrwxrwx 1 root root 15 Jan 1 00:00 libfoo.so -> libfoo.so.1.2.3
lrwxrwxrwx 1 root root 15 Jan 1 00:00 libfoo.so.1 -> libfoo.so.1.2.3
-rwxr-xr-x 1 root root 123456 Jan 1 00:00 libfoo.so.1.2.3
The build process for libbar will expect a libfoo.so to exist. The linker will read that file, find an entry in the binary file headers stating that its soname is libfoo.so.2 (e.g. DT_SONAME for ELF), and produce an output file libbar.so.4.5.6 with a dependency on libfoo.so.2 (e.g. DT_NEEDED for ELF).
I’m proposing two changes to this process.
First, at no point in this process is the name libfoo.so.1.2.3 actually needed. It’s present for human information, and I think it’s used in the case where you have multiple libfoo.so.1.* files in the sense that ldconfig will find the newest one and update the libfoo.so.1 symlink. But we aren’t calling ldconfig here because we’re not installing anything systemwide. So, it is safe to get rid of the filename libfoo.so.1.2.3 and make libfoo.so.1 an actual file, not a symlink.
Second, libfoo.so isn’t actually loaded as a dynamic library[1]. It’s just used as input to the (compile-time) linker, and the purpose is to cause the linker to figure out which actual name to use in the dependency (the soname) and keep track of which symbols are defined. For all the platforms that PyPI accepts binary wheels for, their linker supports an equivalent mechanism where libfoo.so is a text file. For GNU ld and things compatible with it, this is a linker script (with the same filename). For Apple ld, this is a .tbd file (with a slightly different filename). So, it is safe to get rid of the libfoo.so -> libfoo.so.1 symlink and replace it with this text file.
Both of these replacements can be done as a postprocessing step after the build of libfoo is complete, and does not require knowing anything about which build tool libfoo uses or how it works. You build the library via whatever means, and then you look at the lib directory, and you take any file whose filename does not match its soname but where there’s a symlink from the soname, and you rename it to clobber the symlink. You also take any symlink that has no version number (e.g. .endswith(".so")) and replace it with a linker script or equivalent. And hopefully there are no symlinks left.
And the transformed lib directory is transparently compatible to libbar. The build process of libbar will call ld -lfoo, which will find the linker script and behave just as if it had found a symlink, The runtime of libbar will use the dependency on libfoo.so.2 and open that file and not mind that this is a real file and not a symlink. So you don’t need to care about the build tooling or implementation of libbar either. It also doesn’t matter if libbar does a dlopen("libfoo.so.2") instead of having a declared dependency, or if it’s a Python extension module instead of a generic C library, or if it’s a binary, or even if it’s Python code using something like ctypes.CDLL("libfoo.so.2") instead of native code; all of these cases just open the filename libfoo.so.2 and work whether or not it’s a symlink.
In other words I’m not proposing doing anything in the general space of pkg-config. If your library happens to use pkg-config, the exact same .pc file will work for consumers before or after the transformation.
Ugh, that’s a good point. Thanks.
Would it be untenable to rework the wheel-building process to do the packaging and the auditwheel-ing in a single step? This would require one fix per Python build backend, since the mechanism is build backends directly produce wheels, but it still would not require one fix per C build tool.
Another option - accept this PEP as written, and then teach auditwheel to take a wheel 2.0 as input, see if it can get rid of all the symlinks, and if so produce a wheel 1.0 as output. That would resolve my concern about the practical impact of getting people to upgrade pip.
I’m not sure I agree with that, in that while the packages are few and advanced, I think they will be widely installed by people. Examples given so far are CUDA, MKL, cupy, and arrow, which from at least my perspective (people doing scientific computing but not using Conda) are very common packages. Very few packagers will be doing something like this, but many users will be consuming their packages. And, also from my perspective, running an out-of-date version of pip and not being totally aware that upgrading pip is a reasonable thing to do is also very common.
This is also something of a dependency for moving away from statically linking common dependencies like OpenSSL, and there have been several discussions about how we ought to get to that world.
Apple ld supports text-based stubs as I mentioned in my comment, which can accomplish the goal, see man tapi. Is this insufficient? I learned about these while writing that comment so it is entirely possible I’m missing something. 
And aren’t symlinks only relevant for shared linking anyway? For static linking, my impression is people just have a single normal file libfoo.a and no libfoo.a.1 names or symlinks or anything.
There’s also a discussion to be had somewhere about having separate wheels for development libraries/headers and runtime libraries, and if the development wheels are 2.0, that seems fine. I have no objection to building wheels requiring the latest version of pip, as long as the average user trying to make some tensors flow has a good default experience.
For the same reason I think it’s also fine if we say that AIX users need to upgrade to the latest version of pip—that’s strictly better than requiring that everyone upgrade.
So I think this is convincing me of the merits of accepting this PEP to have the format well-defined and standardized, and least as an interim measure, having auditwheel try to generate 1.0 wheels as best as it can. In a couple of years the need to support old pip versions will be less relevant and we can drop this compatibility code and expect everyone to be able to consume wheel 2.0.
(And, yes, this is influenced a bit by what else ends up in wheel 2.0.)
Unless consuming code is doing
dlopen("libfoo.so")without the version suffix, perhaps viactypes.CDLL("libfoo.so"). But this is technically incorrect and should be discouraged. Precisely because it doesn’t encode the soname (the compatibility version, equivalent to a semver major version), usinglibfoo.soat runtime is unsound. The same function name can change its ABI betweenlibfoo.so.1andlibfoo.so.2, perhaps because a struct changed definition or some#defines changed. Many times this is done in a way that is backwards-compatible in API, i.e. recompiling (including CFFI’s API modes) would work and pick up the binary changes. But if you’re doing stuff withdlopenor equivalent (ctypes, CFFI’s ABI modes), there’s no compilation step and you’re expected to match the ABI, and you can’t do that. Moreover, because this is a development symlink, for system-wide libraries, it’s usually not installed by default by the system package manager, and the symlink is packaged separately. The practical effect is that runtime use ofdlopen("libfoo.so")will return file-not-found unless the user installs a package named something like libfoo-devel, which will include header files, build dependencies, and all sorts of other things not needed at runtime. So your distro packager and their users will be happier if youdlopen("libfoo.so.1")instead. If you want to support multiple sonames and you are dealing with the ABI incompatibilities (the easy case is if some other function you’re not calling is the incompatible one), it’s totally okay to loop over all the sonames you know you can handle. ↩︎