I’m a maintainer for the cocotb project. We are finally trying to solve a particular annoying bug for our users regarding our build process.
One of the C libraries we build requires us to pull headers from the user’s system (they are proprietary headers). If those headers are not found, that libraries is not built and a message is printed. Attempts to use the library that failed to build will fail at runtime.
So, we have a non-deterministic build process. However,
pip always caches the built wheel. The user later attempts to load the missing library and it fails. They try reinstalling our package, or even force-reinstalling, to no avail. Most of them are unaware that
pip caches built wheels and there is an option to turn off that behavior that is required to actually rebuild the wheel to include the library they need.
This isn’t the first time our project has had issues with the wheel cache. In that issue, the wheel cache would store wheels with C libraries that had RPATHs hardcoded pointing to a particular Python environment. That wheel and the RPATHs would be reused in successive installs into other environments on the system, but the Python environment the RPATH pointed to could be deleted.
So I have some questions:
- Where are the rules/assumptions of
wheel, or whatever dictates the current PEP517-compliant setuptools-based build process detailed?
- Is having a non-deterministic build against the rules?
- It would be serious effort, but would it preferable to separate out the non-deterministic part into it’s own package that can deterministically build/fail? Thus making the combined build process deterministic?
- Why is there no way to state in the project source that wheels created from that project are not safe to cache?
- Why is caching the default behavior? Any code in setup.py has the potential to make builds non-deterministic and potentially not reusable. Any difference in the user’s shell environment could also affect the build.