# Native-lib-loader: Documentation and best practices on using native libraries in Python wheels **URL:** \ **Category:** Packaging\ **Tags:** documentation\ **Created:** [July 8, 2025, 5:39pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111 "2025-07-08T17:39:57Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [July 8, 2025, 5:39pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/1 "2025-07-08T17:39:57Z")
Hi everyone, as part of the [wheelnext](https://wheelnext.dev/) effort I’d like to share [Sharing libraries between wheels documentation](https://native-lib-loader.readthedocs.io/en/latest/) with you. It’s inspired by @rgommers’s [pypackaging-native](https://pypackaging-native.github.io/) with a more narrow and specific focus on the mechanics of how the underlying system libraries (namely the dynamic loader) handle loading libraries, as well as discussing the various approaches that projects have taken over the years to handle these libraries (auditwheel-style mangling, `ctypes.CDLL`, etc). There are a lot of subtle details that are discussed in comments in various setup.py files or similar places, but are otherwise not captured anywhere more permanent and in a form that can be easily used by new packagers. The new page aims to fill that gap. In addition to these docs, I have a current reference implementation up at [GitHub - wheelnext/native\_lib\_loader](https://github.com/wheelnext/native_lib_loader/), and I am happy to see development proceed there as we can discuss and converge on solutions. I am hoping that this project can serve as a way for us to collect the best practices available to us and better understand the edge cases where they don’t work. In an ideal world I would love for us to work towards proposing standards (PEPs), but given the immense variety of systems on which people run Python extension modules I think that it’s valuable to not let the perfect be the enemy of the good and at least document practices that are the best we can do across a wide range of platforms. There has been plenty of preexisting discussion around this project. Some of the places I’ve found the most comprehensive discussions include @njs’s [pynativelib proposal](https://github.com/njsmith/wheel-builders/blob/pynativelib-proposal/pynativelib-proposal.rst) and [this previous DPO thread](https://discuss.python.org/t/native-dependencies-in-other-wheels-how-i-do-it-but-maybe-we-can-standardize-something/23913) from @virtuald, and @FFY00’s [dynamic-library](https://github.com/pypackaging-native/dynamic-library) project aims to provide another very similar implementation. Happy to see other links as well if others can enlighten me. For those of you who want to continue this discussion, also feel free to join the `dynamic-library` channel on the PyPA Discord. Thank you to everyone who has already provided any contribution to this project so far, and thanks in advance to everyone who helps participate as we go forward! ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [July 9, 2025, 6:58am UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/2 "2025-07-09T06:58:28Z")
> [@vyasr](#): > > Hi everyone, as part of the [wheelnext](https://wheelnext.dev/) effort I’d like to share [Sharing libraries between wheels documentation](https://native-lib-loader.readthedocs.io/en/latest/) with you. That’s a very nice write-up. Quick comment: it lacks a link to the repository where the source code lives. ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [July 11, 2025, 8:04pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/3 "2025-07-11T20:04:27Z")
Very nice indeed, thanks for sharing @vyasr. Something that we found very useful when working on `scipy_openblas32|64` wheels was doing both the symbol and library renaming that you describe, and then providing some utility functions that can easily handle that rename easily at build time for the package linking against the renamed library. `get_pkg_config` and how it supports all of preloading, `auditwheel` style mangling, and using it as a regular shared library for local development in particular may be of interest: > Perhaps something like that could be generalized and included in `dynamic-library`? The `libdir` and RPATH handling in particular seem generally applicable. ---
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [July 13, 2025, 8:50pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/4 "2025-07-13T20:50:12Z")
Thanks for pointing that out, I wasn’t very clear that when I mentioned a reference implementation it lived in the same place as the documentation: [GitHub - wheelnext/native\_lib\_loader](https://github.com/wheelnext/native_lib_loader/) ---
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [July 13, 2025, 9:12pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/5 "2025-07-13T21:12:17Z")
This looks interesting. I see that OpenBLAS supports [a SYMBOLPREFIX](https://github.com/OpenMathLib/OpenBLAS/blob/develop/Makefile.system#L1578). Is the way this is currently working predicated on setting that when building BLAS then using `BLAS_SYMBOL_PREFIX` inside SciPy to map the prefix used when SciPy builds against BLAS? AFAICT there used to be a generated header `blas64-prefix-defines.h` produced [by this build utils file](https://github.com/scipy/scipy/blob/maintenance/1.11.x/scipy/_build_utils/_fortran.py) in order to provide the mappings, but I’m guessing that changed during the migration to Meson for build and I haven’t gone digging for the new approach yet. Producing a pkgconfig (or CMake config for CMake projects) to handle this would certainly be nice. It does simplify the handshake between the producer and the consumer to agree on what the final symbol names look like without having to rewrite code. ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [July 14, 2025, 7:28am UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/6 "2025-07-14T07:28:34Z")
Yes, that’s how it is working indeed. The relevant pieces are: - `BLAS_SYMBOL_PREFIX` in [npy\_cblas.h](https://github.com/numpy/numpy/blob/08916dc1f34de32ccce6b1018613de98c3cc68ff/numpy/_core/src/common/npy_cblas.h#L49-L74) for using the correct symbol names at source level - including that define in the return value from `pkg-config --cflags openblas`: [`scipy_openblas64/ __init__.py`](https://github.com/MacPython/openblas-libs/blob/24557b4f39d4bef98e4f71c9f85858fd80abb6c7/local/scipy_openblas64/ __init__.py#L83) - Ensuring pkg-config is set up for BLAS detection during wheel builds in CI in [`cibw_before_build.sh`](https://github.com/numpy/numpy/blob/08916dc1f34de32ccce6b1018613de98c3cc68ff/tools/wheels/cibw_before_build.sh#L49-L50) ---
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [July 18, 2025, 3:03pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/7 "2025-07-18T15:03:21Z")
Got it, so this approach is viable but it does require that the underlying library support this form of symbol mangling or that we implement something like [machomangler](https://github.com/njsmith/machomachomangler) that also works for ELF and PE. The latter seems like we’d be biting off more than we can chew. I don’t know that we want to rely on any tool that requires knowing the full specifications for these. Even patchelf has bugs and when we run into them they are challenging to diagnose for all but a vanishingly small percentage of users. Conversely, modifying the original libraries to support symbol prefixing at build time is pretty straightforward but requires manual intervention for each library. Maybe we should advocate that approach and if it becomes too unscalable we revisit building a tool for mangling after the fact? ---
**Author:** ![rgommers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rgommers/32/837_2.png) [@rgommers](https://discuss.python.org/u/rgommers)\ **Post date:** [July 20, 2025, 7:47pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/8 "2025-07-20T19:47:21Z")
I agree with your assessment @vyasr, modifying the code to support symbol renames is cleaner. Another advantage over a tool doing the mangling is that it does allow migrating to separate wheels, with say both packages B and C depending on the same package A with a symbol prefix or suffix. With random mangling at build time that isn’t possible. ---
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [July 24, 2025, 6:02pm UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/9 "2025-07-24T18:02:25Z")
I think the other approach that could work in principle would be if the build tool itself (Meson, CMake, etc) supported this directly. That would be the hardest solution to implement I expect, but it might be the most powerful if it could be done. ---
**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\ **Post date:** [August 2, 2025, 11:08am UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/10 "2025-08-02T11:08:08Z")
Very useful write-up! In relation to [Option 3b: Environment Variables](https://native-lib-loader.readthedocs.io/en/latest/reference/solutions.html#option-3b-environment-variables), that’s the approach `venvstacks` uses, but it’s _only_ possible because `venvstacks` ensures both of the assumptions in the “If we were always in a virtual environment … and if virtual environments had well-defined entry points for hooking in to set environment variables we could embed the information in the environment.” qualifier hold. As you say, it’s not a general purpose solution, it requires the adoption of a specific deployment and execution model to enable it. ---
**Author:** ![vyasr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vyasr/32/9915_2.png) [@vyasr](https://discuss.python.org/u/vyasr)\ **Post date:** [August 13, 2025, 2:45am UTC](https://discuss.python.org/t/native-lib-loader-documentation-and-best-practices-on-using-native-libraries-in-python-wheels/98111/11 "2025-08-13T02:45:54Z")
Right yeah the challenge is that most of the documented approaches can work in some contexts but they break down in others. It’s very difficult to come up with something that works even “most” of the time.