Packaging C/CPP libraries as wheels

Python Help
1

I am trying to package pure C/CPP library as wheel and I could do that by referring some samples available on PyPi.

However, I need to set LD_LIBRARY_PATH for the libraries that are supplied in this wheel, to consume the libraries. Is there a way by which the package can set the LD_LIBRAY_PATH on its own at installation, so that the user does not need to set it.

Please advise.

1 Like
2

How is it intended that these wheels are used? Installing with pip, and then setting an environment variable to the installed location is a bit unusual. Isn’t it possible to just install them somewhere that’s already on LD_LIBRARY_PATH?

It’s straightforward for a Python wrapper to deduce the path to the .so files, and use that to import the compiled libraries with ctypes.

If someone’s making their own C / CPP application that uses these libraries, they need to set LD_LIBRARY_PATH (along with everything else) in the builder or have their installer do so.

1 Like
3

This would be my recommendation as well. Include the bare minimum Python code in there in order to find it (you can look at certifi and pybind11 packages as examples of this, as they do similar things, though not for dynamic libraries).

Or at worst, your Python wrapper can modify os.environ["LD_LIBRARY_PATH"] when called (or whatever API is needed for the current platform, since that environment variable is not universal).

1 Like
4

You should never use LD_LIBRARY_PATH. It’s a hack that can and will cause all sorts of problems. The blog post https://www.hpc.dtu.dk/?page_id=1180 explains the issue with LD_LIBRARY_PATH very well.

Instead you should use new rpath (DT_RUNPATH) in combination with dynamic string tokens like $ORIGIN, e.g. link your extension with -Wl,--enable-new-dtags -Wl,-rpath=$ORIGIN/../somepackage/lib to load shared libraries from ../somepackage/libs. man ld.so(8) explains how dynamic string tokens work.

2 Likes
5

A common idiom for this is to expose Python APIs that return the directory(or directories) containing the C libraries, and let users do what they want with it (including setting LD_LIBRARY_PATH if that’s their preference):

1 Like
8

That does not allow you to distribute the extension, because it’s dependent on installation specifics (well, you can distribute it, but it will not work if the user’s installation layout differs from yours; so it’s ok for controlled internal setups, not for public distribution).

9

That does not allow you to distribute the extension, because it’s dependent on installation specifics

Actually, it would. The $ORIGIN is not a shell variable that is resolved into a path at the moment the library is compiled (if it is properly quoted at compilation time).

It is resolved by the linker at runtime to the path of the folder containing the .so that is trying to link or dynload another .so.

Of course, the example provided by Christian only works if the .so that is trying to link or dynload another .so is in the same virtualenv or python installation so that the relative path works.

10

Yes, but it implies that the relative path is still correct. There is no guarantee that it will be.

11

you can add a post-installation step in the package’s setup.py

This will only work for an sdist, the original question was specifically about wheels which are only unpacked into place and have no install-time scripting hooks whatsoever.

12

I was looking at the numpy wheel available on PyPi for x86 - https://files.pythonhosted.org/packages/b0/d9/7c338b923c53d431bc837b5b787052fef9ae68a56fe91e325aac0d48226e/numpy-2.2.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

this wheel bundles the below libraries
libgfortran-040039e1-0352e75f.so.5.0.0 libquadmath-96973f99-934c22de.so.0.0.0 libscipy_openblas64_-6bb31eeb.so

this way it does not require them to set LD_LIBRARY_PATH.

Does this approach of bundling the library within a parent package that needs the library at runtime common? I have seen this kind of bundling of required libraries using auditwheel in some other packages too.

13

Does this approach of bundling the library within a parent package that needs the library at runtime common?

I should think the majority of C (and even other) extensions are compiled to dynamic shared libraries (.so or .dll files). Even if they use a framework like scikit build, meson or PyO3/Maturin etc., instead of the good old, plain gcc way:

A C extension for CPython is a shared library (e.g. a .so file on Linux, .pyd on Windows), which exports an initialization function .

14

Found Need auditwheel-like utility for other platforms - #4 by njs for Windows. So I think similar approach can be used on Linux for providing libraries that are needed at runtime but not built via C/CPP extensions in Python packaging script. Please advise.

15

Importing the Python C API machinery isn’t strictly necessary. Any .so or .dll can be imported using ctypes. But that can be a bit tricky, possibly involving casting of args correctly, and void pointers. And handling the differences between platforms yourself.

It’s a little bit you say tomayto, I say tomarto. But the extension frameworks these days are very well developed, and straightforward to use for C/CPP programmers used to Make and CMake etc.

16

Okay, let me come to specific question.

How can I provide hdf5 libs in a package for use by h5py wheel. The default packaging mechanism of h5py does not bundle the hdf5 libs. It relies on system supplied libraries. However, I am using a newer version of hdf5 lib which is not available on system, so pls suggest how I can supply hdf5 libraries so that h5py wheel can function well.

17

I don’t think using the wrong version of its dependencies via any sort of custom package is remotely worth it. You can’t change how its devs have designed h5py (without distributing your own third party fork or rebuild of h5py).

Alternatively, you could:

  • Try the h5py Conda release.
  • Run h5py in a docker container, that has the version of the hdf5 libs installed, that h5py supports.
  • uninstall the hdf5 libraries, and install ones with a version that h5py actually supports. E.g. using your system package manager.

From the docs:

although naturally new HDF5 versions released after this version of h5py may not work. Odd-numbered minor versions of HDF5 (e.g. 1.13) are experimental, and may not be supported.

Or:

  • Ask on their forums if they have any ideas.
  • Wait for the next release of h5py.
1 Like
18

So does that mean that either one should create a fork and package compatible hdf5 within h5py wheel by using cpp extensions, or use the system library.

19

The system package manager. Depending how urgent it is, I would pick whichever of those two (and my four other suggestions, Conda, Docker, asking for more specialised help, or simply waiting) you think will require the least amount of work.

20

Or build your own wheel and use that. It doesn’t require forking, it just requires that you install from an index other than PyPI, potentially just a location on your own disk.

If you really want your improvement to be available to the entire world, then contribute it upstream, or at least get their permission to publish a second package (probably “h5py-bundled” or something) to PyPI. The upstream team may be willing to do it and just haven’t been asked.

1 Like
21

I have built hdf5 wheel and I am using it. It installs the libs in the virtual env. However, I need to set LD_LIBRARY_PATH to point to the virtual env site packages path, so that the hdf5 lib from this location can be detected with package that needs it, like h5py.

Is this correct? I am looking for ways to avoid setting LD_LIBRARY_PATH.