In our latest release, we discovered that Numba is producing wheels that vendors in libtbb and libgomp but we would like to stop auditwheel repair from doing so. Is it possible? In our case, Numba is optionally loading the C-extension that are linking to these libraries.
In addition, I have read that wheels are meant to be self-contained (Packaging DLLs on Windows), but for libraries like TBB, it should not try to vendor them. The reason is that no process should have multiple instances of TBB for performance issues (not sharing threadpool defeats the purpose of dynamic thread scheduler) and conflicts (different versions of TBB in the same process may conflict with each other).
Another reason to disable library vendoring is to avoid accidental violation of license agreements (e.g. GPL).
To clarify, you’re talking about manylinux wheels, yes?
In that case, you’ll need a manylinux version that defines libtbb as part of the platform, so that it doesn’t have to be bundled into the wheel itself. (Or you need an index other than PyPI so you can require libtbb to be installed by users and provide non-manylinux Linux wheels.)
Otherwise, the promise made by manylinux/auditwheel is that anything not considered part of the platform will be bundled. It’s not ideal, but it works better than any of the alternative ideas that have been considered (and people were willing to write the tools to make it work, which is a very important part of volunteer projects like ours).
In our case, the C-extension that links to TBB is an optional feature that Numba will only use if a libtbb is available. In the ideal case, the user can have tbb · PyPI installed to enable the feature. So that the Numba wheel is still manylinux compatible.
What should libraries that requires to be a process-wide singleton do? It does not seem practical to define a new platform for every such library. If auditwheel lets us disable vendoring of certain DSO and pip allows packages that share DSO, it gives user the power to define their “platform” that extends manylinux.
If it cannot be done, would a project like tbb · PyPI be violating the spec of pip/wheel since its only purpose is to share DSOs?