"Modern" way to build packages with C (Cython) extensions

Packaging
1

TL;DR

How the heck does one set up and build a complex package with Cython and C++ and all that with the modern tools?

NOTE: I’ve been scouring the net looking for up to date information on this, and have found nada, zip., zilch. This page looked promising:

https://packaging.python.org/en/latest/guides/packaging-binary-extensions/

But:
Packaging binary extensions
Page Status: Incomplete
Last Reviewed: 2013-12-08

So – not so helpful.

Maybe there’s a resource out there, but I haven’t found it.

I’m asking this now because my CI build has just broken with up-to-date Python and setuptools, (I haven’t been able to figure out why yet, it still works on my development machine) and I’ve been seeing messages about setuptools being depreciated, and that I should use build, and I know distutils is going to be taken out of core Python, and …

So I clearly need to modernize my build system – but I have no idea how.

(Side note: “build” is an unfortunate name for a package – it’s impossible to search for information about it!)

Maybe it’s just that I haven’t found the right docs, but I’ve been following the modernization of packaging for a long time, and I’m a bit distressed that my important use cases seem to be continuing to be neglected. That is:

  • Building C extensions
  • “editble” mode.

For a tiny bit of history about why that’s what’s important to me:

  1. I discovered the distutils a LONG time ago-- and was SO HAPPY that it provided a platform independent way to build C extensions – the pure Python stuff was nice, but meh – the C extensions were the killer feature.

  2. Then setuptools came along, and I was not the least bit happy about how much it merged into one package. But it did provide “develop” mode – which is an absolute godsend – I have no idea how anyone can develop a complex package without it.

Anyway – I’ve been trying to pay attention to the progress of the modernization of packaging for a good while, but haven’t done much new for my complex packages because I don’t see the features I need being supported.

Hopefully, someone will be able to point me to a resource that will help.

6 Likes
2

setuptools per se is not deprecated. The CLI interface python setup.py on the other hand is.
In general, you can basically use python -m build[1] as a replacement for python setup.py sdist bdist_wheel. (Instead of build you can search for pypa-build, maybe that helps?).

setuptools will still be able to handle C extensions and the editable mode[2].

In summary what is happening in the ecosystem is the build will work as a frontend and call setuptools[3] behind the scenes.

  1. After pip install-ing build. ↩︎

  2. Current existing editable mode for setuptools requires the existence of either setup.py or setup.cfg file. Also note that “editable” mode does not make C-extensions editable. ↩︎

  3. Or other packaging tools. ↩︎

3

If you are building an extension that requires NumPy then you are essentially stuck with setuptools < 60 for the time being: Status of numpy.distutils and migration advice — NumPy v1.23.dev0 Manual

This makes all the current discussion about moving to more declarative configuration (that requires a recent setuptools) a bit hard to follow, since what parts of this are expected to work.

4

Thanks – the problem is that our current build is not compatible with
python setup.py sdist bdist_wheel – though hopefully we can fix that without too much pain.

And hopefully that will result in a lot fewer deprecation warnings.

But we’ll see – this does seem to be an under-documented issue at the very least.

5

we are indeed highly dependent on numpy – though I don’t think we use numpy.distutils (at least not directly).

Maybe it’s time to check out Meson (cmake has always been a struggle for me) – but I’d sure like a pure-python solution – maybe a front-end to meson?

We’ll see – time to get with the program.

6

I am very sure that is, sadly, the case.
Regarding setuptools docs specifically, if you find any problems/confusion, please feel free to open an issue.
Moreover if you are in the mood/have the time to contribute, PRs are always welcome!

7

In SciPy we (not me but our very able SciPy folks) switched to Meson and NumPy will soon follow.

2 Likes
8

Thanks – scipy is massively more complex than what I"m dealing with, but that may be a good place to start.

9

cibuildwheel is a core pypa project that makes it easier to produce cross-platform wheels in CI. There is a working examples page that links out to some packages that vary in complexity.

cython is closed tied to setuptools as I understand it. SciPy is working to make cython more compatible with meson, but it is still a work in progress.

2 Likes
10

@henryiii and @rgommers would likely have valuable input here…

Declarative config (in the setup.cfg) was originally introduced in Setuptools 30.3.0 (released December 2016), with some specific keys and semantics requiring later versions; however, requiring >=42 should cover basically everything. Tool and metadata configuration in pyproject.toml was only just implemented by @abravalheri , but is still experimental so probably not yet advisable to use it for production quite yet anyway. See the docs for more details.

To note, python setup.py sdist bdist_wheel is what should be replaced with python -m build (though under the hood, the same/similar code gets called). I’m not 100% sure what you mean here—that your package can’t be built into a wheel? Or even into a sdist? (If so, how are you distributing it right now?) You’re not using setuptools at all for your build system? There’s some other issues? If you are looking for some general guidance on this, maybe we can help advise something if you’re able to provide more details on the problem.

11

@ChrisBarker-NOAA my recommendation would be to wait a few months if you can; SciPy 1.9.0 will do a release with Meson (and meson-python · PyPI), and at that point everything should be pretty robust. I think things are pretty much good to go now, but before we’ve done full release cycle there may be some unexpected little issues with sdist/wheel builds or support for less commonly used platforms.

The one thing you want that will be missing at that point is support for editable installs. Those are fairly hard to do, because Meson builds are fully out-of-place by design (but we have a plan to deal with this, it’ll just take a little longer - it’ll be the next step after doing a release).

Second recommendation: consider staying with python setup.py develop for a little longer. Yes everything is deprecated, but it won’t go away tomorrow and there isn’t a good story for a replacement if you’re working on compiled code. As earlier responses pointed out, there is a good replacement with python -m build for producing sdists and wheels - and I recommend build for that, it’ll also work with Meson - but it doesn’t help you with your development workflow today.

It’s not really. Cython itself ships with setuptools integration, but you’re often best off ignoring that and treating Cython as a compiler and invoke it at the start of your build to “cythonize” all your code. See for example https://github.com/scipy/scipy/blob/main/tools/cythonize.py, that is robust and you can safely vendor it; I’m not sure if Cython itself ships the same thing now.

There’s also native support in Meson now that is young but probably already more stable than what Cython itself provides for setuptools once you get beyond simple build scenarios (because of a fundamentally better design, no monkey-patching and competing build_ext extensions).

3 Likes
12

Hello! I apologize in advance for a post that brings in perhaps too many external and biased references.

I’d like, however, to chime in and express an opinion that at least from the perspective of packaging for distributions, it might appear to be considerably easier to build projects where one interacts directly with the actual build tool (meson, cmake, …), than the projects that try to manage the build “from the python side”. For example, in nixpkgs it appears to be extremely difficult to maintain pytorch, which hides cmake behind layers of setuptools.Extension. At the same time, it can be relatively trivial to build naively-packaged projects. For example, have a look at NVIDIA/instant-ngp: it’s a plain cmake build that results in an importable .so, entirely oblivious to python’s packaging ecosystem, and with rather heavy dependencies (cuda). It might look extremely naive and ad hoc, but it turns out to be actually easy to build and deploy. The only thing left is to generate dist-info and pack the result into a wheel.

By the way this last step is actually one that lacks convenient tooling, e.g. nixpkgs’ opencv has to do some very out-of-place stuff to ensure that .dist-info is generated, and even then it would ship only opencv.dist-info, but lack opencv-python, opencv-python-headless, opencv-python-contrib-headless, etc-dist-infos that downstream packages ask for in their setup-pys/pyprojects.

I’m quite excited to watch scipy’s progress with meson. For meson-python though, I have to wonder if it wouldn’t be, in the long term, a less painful road to have entirely external binary builds, and a pep-517 backend just for validating the outputs, generating dist-info(s), and packaging the wheels.

13

Note actually that step builds a wheel and we have a hook for doing that which should be used instead.

14

I’ve not been following in any detail, but I do (in general) like the idea of leaving the problem of building C extensions to tools that are good at it, and wrapping the results (somehow) in a Python package.

That’s certainly something it’s feasible for someone to develop. Like with a lot of things in the packaging ecosystem, though, it’s not likely to happen just because “we”[1] think it’s a good idea. Progress tends to happen the other way round - someone builds something and only then do people realise it’s exactly what they wanted all along…

  1. whoever “we” are :slightly_smiling_face: ↩︎

3 Likes
15

There is definitely a severe lack of tooling that assembles wheels from a disorganised set of files (which is what would be desirable to adapt a regular CMake project’s output into a wheel), and in my opinion an overly strong focus on tooling that assumes you’ve got your own Git repository (which I can only rarely make any use of, since this is not normal for my work).

I made a straightforward build backend that can run an MSBuild build and rearrange the output into a wheel, and though it doesn’t trivially adapt for other build systems, I can attest that it’s really quite simple to construct a wheel if you make the user tell you what files should be in the final package and where they should go. Most tools run into trouble because they try and infer that part, rather than letting the author decide.

Once you figure that out, your next problem is keeping up with metadata and “standard” changes :wink: This is why my backend is woefully out of date already, and probably won’t ever catch up, because I just don’t have time to invest in features that I don’t use myself. But making a wheel is really quite simple.

6 Likes
16

So true!

Sure! Never presumed otherwise

We took this in PM

17

Good point. I’m not aware of a reusable library that helps construct a wheel from API calls.

I’m hoping that this will continue to become easier as more of the packaging specs get into a reusable library like packaging, etc.

18

I’ve done my best to make it super easy to develop build hooks for Hatchling https://ofek.dev/hatch/latest/plugins/build-hook/

Hatchling assumes everything shipped is under the project root, but I’m going to add an option for outside content by PyCon when I release publicly. To be clear, I’m really not a fan of the idea but I keep hearing this request so I have to…

19

If it helps make you feel better, I frequently find myself including files that come from other build systems or steps that must be in other directories (e.g. code signing services or SBOM generation). While I can often induce them to be copied into the project directory before zipping, it’s much easier to list them into the tool as instructions than to have the tool try to somehow find them automatically.

2 Likes
20

I believe the wheel project has a longer-term intent to expose a programmatic API, but it doesn’t have (a supported) one yet.