"Modern" way to build packages with C (Cython) extensions

[rgommers (Ralf Gommers)]

@newkozlukov

For example, in nixpkgs it appears to be extremely difficult to maintain pytorch, which hides cmake behind layers of setuptools.Extension.

I’ve worked on PyTorch for a couple of years, including build & packaging stuff. I doubt that this has to do with the thin shim in setup.py - the problem is simply that PyTorch is an extremely complex package itself and has ~30 submodules in third-party/ with also complex C++ code.

What packaging for other build systems typically does is (a) build a wheel, (b) unpack the wheel, and (c) repack it into the correct package format after doing what the tooling needs to do to make everything relocatable, strip debug info, etc… The wheel step is not very interesting, it’s just a convenient interface to be able to do pip install . --no-build-isolation instead of figuring out for every package what the right setup.py / cmake / meson / ninja invocations are.

@steve.dower

There is definitely a severe lack of tooling that assembles wheels from a disorganised set of files (which is what would be desirable to adapt a regular CMake project’s output into a wheel), and in my opinion an overly strong focus on tooling that assumes you’ve got your own Git repository (which I can only rarely make any use of, since this is not normal for my work).

+1 this is lacking pretty badly.

@henryiii

I’ve been funded to work on Scikit-build for the next three years.

@henryiii congrats! Looking forward to collaborating to make building packages with native code easier:)

@ChrisBarker-NOAA

I’m quite excited to watch scipy’s progress with meson. For meson-python though, I have to wonder if it wouldn’t be, in the long term, a less painful road to have entirely external binary builds,

I’m also excited about meson-python – but not sure about the “entirely external binary builds”.

I’m not even sure what this means. meson-python is a thin shim layer; Meson does all the heavy lifting. There’s a lot of little annoying things one has to do to comply to packaging standards (like wheel tags tend to be a cause of bugs). The most interesting parts of meson-python are probably the UX ones, like “if I use pip or build, how do I pass along custom build options” or “how do I express the ABI compatibility constraints for depending on numpy in pyproject.toml in a sane way”.