What should be used to build the CPython of tomorrow?

Committers
1

Hypothetically, what would be your build system of choice if we were to implement a new one for CPython tomorrow? Assume that all the difficult issues have been resolved :magic_wand: We’re just soliciting contenders and core development team skills (at the same time) :slight_smile:

If your favourite build system is missing, post your suggestion and Zach or I will create the poll for it.
To save screen space, each poll is conveniently hidden behind a “details flag”; click the arrow (:arrow_forward:) to expand the polls.

CMake
CMake, https://cmake.org
  • I have read configuration for it and found it easy to read
  • I have read configuration for it and found it hard to read
  • I have contributed to existing build configuration for a project using it
  • I have written build configuration for a project from scratch with it
  • I’m a contributor to or maintainer of it
  • I like it
  • I don’t like it
  • It would be my preferred CPython build system
  • I have no opinion on it
0 voters
Meson
Meson, https://mesonbuild.com
  • I have read configuration for it and found it easy to read
  • I have read configuration for it and found it hard to read
  • I have contributed to existing build configuration for a project using it
  • I have written build configuration for a project from scratch with it
  • I’m a contributor to or maintainer of it
  • I like it
  • I don’t like it
  • It would be my preferred CPython build system
  • I have no opinion on it
0 voters
Bazel
Bazel, https://bazel.build
  • I have read configuration for it and found it easy to read
  • I have read configuration for it and found it hard to read
  • I have contributed to existing build configuration for a project using it
  • I have written build configuration for a project from scratch with it
  • I’m a contributor to or maintainer of it
  • I like it
  • I don’t like it
  • It would be my preferred CPython build system
  • I have no opinion on it
0 voters
Buck2
Buck, https://buck2.build
  • I have read configuration for it and found it easy to read
  • I have read configuration for it and found it hard to read
  • I have contributed to existing build configuration for a project using it
  • I have written build configuration for a project from scratch with it
  • I’m a contributor to or maintainer of it
  • I like it
  • I don’t like it
  • It would be my preferred CPython build system
  • I have no opinion on it
0 voters
3 Likes
2

Expanding on my CMake vote: I’ve contributed minor PRs to CMake and I’m acquainted with their development flow.

1 Like
3

Expanding on my CMake vote: I think it’s the best of a bad lot (and by that I mean every build system that’s ever existed for a project of our size :upside_down_face: ), and so while it’s not my preferred build system, it’s likely to be our best option.

It’s also worth noting that Visual Studio has some level of CMake support, so we could probably completely remove the old project system without overly disrupting those developers. Any other build system is likely going to cause us to keep the current Windows build system around and up to date for development/debugging purposes.

4 Likes
4

If we have to choose new build system, I am eager to learn Meson or Bazel anyway.
But I prefer more simple one. And IIRC Bazel focus on building Mono repo code for multiple languages :wink:

5

I’ve added Buck2 to the list.

6

My perspectives:

CMake

I’m with Steve on this one, it is my preference. Even though I haven’t spent time to learn how to use it “for real” yet, I’m more than happy to do so. It’ll be valuable.

In the past decade I’ve seen an increasing number of Open Source projects using CMake successfully, all sizes from small to huge, include cross compilation, and including ones that build Python applications with a mix of C/C++/Rust extension modules. It seems to be the common community choice. There’s a lot of value in that.

CMake gets used for other C based often cross compiled Python implementations: MicroPython and thus CircuitPython.

Meson

I voted “I don’t like it” on Meson rather than “no opinion” because despite never having seen it or used it, one thing I think we should avoid is for CPython to require a build system written in Python. We don’t need more long term bootstrapping complexity. Even if it is easy to argue that we already have that situation in other forms or for other dependencies I’d like to avoid gaining more of it.

SCons would fall in this bucket as well, being Python based.

If we go with anything Python based - we need to avoid getting into a situation where arbitrary code can ever be written into what should be a pure a build configuration expressing an action graph. We already got rid of setup.py, we don’t want to allow that to recur.

Basically the build system should ideally not allow us “cross the streams” between platform introspection, configuration, and phases of build actions. (ideally… because no matter which system some of that is just a policy we should set for ourselves, actual prevention is hard)

Bazel

I actually really like it as a build system. It has so many right ideal concepts. But I don’t think we’re really complex enough to need it.

I lack knowledge about how, if at all, Bazel can solve any platform configurability issues. Can it? Any build system we move to needs to mean ending our reliance on configure.ac.

My other nit: Bazel is written in Java. I take issue with Java because it the JRE is a heavyweight dependency. That is probably unreasonably bias on my part and should be ignored… From the perspective of a Bazel user you install it and it includes all runtime bits it needs.

But Java is also strongly associated with Larry Ellison and Oracle both of whom have clearly demonstrated their stance as Evil anti open source entities. I prefer to avoid depending on them even if others are willing to ignore the tusks on that elephant in the room.

If we went with Bazel assuming the platform configuration story has an acceptable to us answer, I’d be happy to use it.

Buck2

(edited to add this)

I voted both I like it and I have no opinion. :sweat_smile:

I expect the only people who’ve ever used it are in the Meta sphere. Its concepts should be entirely Bazel like. It also uses Starlark which is Bazel’s python-subset-syntax language and likely still uses BUILD rules resembling those from Bazel. So people familiar with one are likely to be able to navigate the basics of the other pretty readily?

The difference is its implementation is in Rust & Starlark rather than Java & Starlark.

How does Buck2 deal with platform configurability?

3 Likes
7

Bazel is written in Java

I agree that should be a disqualifying dependency right there. Adding build dependencies of complex other ecosystems would be a disservice to the entire Python community and especially core developers. For example, Java and, unfortunately for the foreseeable future, Rust would both be problematic for macOS users (yes, I know there are third-party providers like Homebrew).

5 Likes
8

Does CMake handle cross-compilation smoothly? There are more and more use cases for that, like WebAssembly (Emscripten/WASI) and now iOS.

9

Quite smoothly from my experience with the CircuitPython build.

That one builds for many different architectures, at once if desired, including building native host tools to do their equivalent of py->pyc bytecode compilation.

3 Likes
10

Well, as smoothly as it handles any other configuration option :slight_smile:

We’re still going to end up with extensive documentation listing all the options that may need to be provided when you generate your build scripts. And the CMake script is going to be loaded with conditional checks for various platforms throughout (maybe we can split these into included files, like you would with MSBuild, but I don’t think CMake has a similar “ambient settings” approach and we’d likely end up getting to twomany separate lists of projects and files pretty quickly).

Might also be worth pointing out that CMake is essentially a build script generator (and also knows how to trigger those scripts so that you can pretend CMake is also doing the build). So it replaces autoconf/configure entirely, but we likely still need to carefully consider which build tool it’s actually going to be using behind the scenes on each platform.

3 Likes
11

I voted “I like it” for CMake because, while it’s not extremely pretty, it’s a solid and dependable solution. There are a huge number of projects using it, including projects more complex than CPython, ample documentation and help on the Internet (and also on paper).

2 Likes
12

My, biased, preference would be Meson. The main reason being how powerful it is, and at the same time how difficult it makes to do things wrong. I think it is especially nice for large projects with lots of contributors because it tends to limit the amount of “janky”-ness that can happen, and is IMO nice for new contributors. Additionally, it also has a pretty good cross-compilation support, with a clear distinction between the host and the target, and AFAIK has decent Visual Studio support.

The main issue is that it is written in Python, creating bootstrapping issues. My solution for that would be to use muon in the CI, which is a Meson implementation written in C. I have used muon before and did not have any issues with it, but I guess this comes down to how comfortable others are with using it as part of our workflow.

A couple complex projects that use it are NumPy and SciPy.

Again, biased, but I have had bad experiences with CMake, though I guess that mostly comes down to how careful we are with our build definitions. If we opt for CMake, we should probably have some guidelines in the devguide for how to use it.

I haven’t had any experience with any of the other mentioned build systems.

7 Likes