Loading dependency of python binding manually

Basically I have a snippet that does:

ctypes.cdll.LoadLibrary(Path(bundled_lib))           
import _python_binding

where the _python_binding module is built with link_libraries to the one in bundled_lib. As far as I have tested, it works fine on manylinux, but I got a report that on alpine docker (muslinux) it fails. Anyone got an idea of what is going on differently there?

I do not want to insert RPATH dependence in the packaging by default if there is another way of loading the dependent library dynamically.