Enforcing consistent metadata for packages

[oscarbenjamin]

@kknechtel

@rgommers

It doesn’t, this is the job of auditwheel & co right now.

So in this case, the build would actually be impossible for an end user?

It is not impossible but it might be difficult. The PEP 517 build interface assumes that non-Python dependencies and build tools are just available somehow externally. If a user has those dependencies available (e.g. from apt, homebrew, built from source etc) then they can build a wheel from source and install it. The wheel built in this way is not portable but that is okay if they only want to install it locally.

To make a portable wheel you need to wrap the PEP 517 build interface between steps that first install/build the non-Python dependencies and afterwards bundle them into the wheel. Typically cibuildwheel is used for this and codifies these additional steps:

1. Install non-Python dependencies (CIBW_BEFORE_ALL).
2. Perform per-Python-version setup (CIBW_BEFORE_BUILD).
3. Run the PEP 517 build
4. Repair the wheel (CIBW_REPAIR_WHEEL_COMMAND).

The repair wheel step bundles the libs into the wheel. Some of those libs might be things that the user does not have on their system. Other things might be libs that they do have but that the extension modules in the wheel might not be ABI compatible with so the wheel needs to bundle its own versions anyway.

When installing from source with e.g. pip you only get the PEP 517 build step. That can only work if you first install the dependencies and build tools system wide e.g.:

sudo apt-get install libopenblas-dev python3-dev build-essential
pip install --only-binary numpy numpy

For certain dependencies it would be nice to build them as part of the PEP 517 build step and have them bundled into the wheel but there is not currently an easy way to make this work.

@kknechtel

What I understood from @jamestwebber 's idea is that it should be easy enough to detect at runtime that the dependency is missing (provided neither by the installed wheel nor by the system) and give a suitable message (perhaps with advice like “try using the system package manager to install foo-dep” or “please downgrade Python and/or this package, and wait for a wheel for your platform”).

It is not just about the dependency being present or not but about ensuring ABI compatibility. You need to know that the extension modules were built against the same build of the libraries that will be used at runtime. There is no general way to detect ABI compatibility except by ensuring that the extension modules and the libraries were built together. There are two common ways to ensure this:

• Install the libraries system wide, build against them and only use the resulting wheels on that system.
• Bundle all the libraries that were used when building into a wheel that is portable because it only uses its bundled libraries at runtime.

If you have no information about the provenance of the built library files that are found at runtime then there is no way to detect whether they would be ABI compatible with the extension modules.

This is also why ABI dependency between portable wheels is difficult/impossible: you would need to be able to say “this wheel is only ABI compatible with that exact other wheel” regardless of what the version constraint metadata in any sdist says. There are ways that this could be made to work but it necessarily involves the wheels having much tighter dependency constraints than the sdists from which the wheels are built.