Assistance building pygmo2 wheels with cibuildwheel

Packaging
#1

(Cross posting from Assistance building pygmo2 wheels · Discussion #1368 · pypa/cibuildwheel · GitHub)

I am collaborating with pygmo2, a Python & C++ under the European Space Agency Github, to try to build and publish manylinux wheels. At the moment the project is only publishing conda packages, the existing wheels got outdated, and no source distributions are on PyPI.

I made significant progress on https://github.com/esa/pygmo2/pull/110 creating a custom Docker image based on quay.io/pypa/manylinux2014_x86_64 that includes all the dependencies. However, I’m hitting a wall now.

The project does not use a conventional procedure with a Python metadata file (setup.py, pyproject.toml, or anything else), but rather a custom CMakeLists.txt that shuffles the Python files around:

For this reason, to even start cibuildwheel I’m doing touch pyproject.toml, but of course this ends up producing a hollow, pure-Python wheel:

  cibuildwheel: Build failed because a pure Python wheel was generated.

Trying to compile the code inside CIBW_BEFORE_ALL fails because Python3 is not found:

CMake Error at /opt/_internal/pipx/venvs/cmake/lib/python3.9/site-packages/cmake/data/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES
  Development Development.Module Development.Embed) (found version "3.11.0")

I get that probably I shouldn’t try to compile the code against a specific Python version in CIBW_BEFORE_ALL, and that cibuildwheel will take care of that. But I couldn’t find any technical details in the documentation on how the manylinux Docker images work, and the “how it works” section of the docs is a bare flow chart with not much explanation.

In summary, I’m a bit lost on how to proceed. I’m wondering if it’s technically possible at all to build the wheel with the current architecture (in which case I’d love to have some guidance), or rather if it will be difficult or impossible and I should first try to standardize the build process using something like scikit-build.

#2

Hi @astrojuanlu , first configuring your package with scikit-build will address the issues you are encountering: artifact installation and finding Python build dependencies. cibuildwheel will work like a charm after scikit-build is configured.

1 Like
#3

Thanks @thewtex ! Will bite the bullet and give it a try

1 Like
#4

cc @henryiii

#5

Turns out, introducing scikit-build on the project was much less complex than I expected, and it’s kind of working already :smile: WIP: Build project using scikit-build by astrojuanlu · Pull Request #113 · esa/pygmo2 · GitHub

4 Likes