Revisiting the case for CMake as a primary, cross-major-plats, build config

Core Workflow
1

The Python Windows build manifests everything we non-Windows users expect to see in a Windows build system, and no contact with the reality of how well building on Windows is today in the real world.

Out of the necessity of supporting the most ancient of legacy dos/win/msvc configs, it uses BAT files instead of modern, open-source, cross-platform, POSIX Shell-specification-based Powershell CORE scripts; It is presented as a series of visual studio solution and project files so complex that people are building complex tooling to make simple changes to it like enabling static libraries: python-build-standalone/build.py at 3307e3920cd6cd404629899adadfcf248c5ca992 · indygreg/python-build-standalone (github.com)

As the gap between the oldest and newest supported VS editions grows wider, the PCBuild folder becomes more and more frustrating to everyone, doing proper service to none.

Modern Visual Studios (2017+) treat CMake and Ninja as first-class citizens and ship with them, while CMake Generators have been capable of creating visual studio projects for quite a long time.

I believe there’s a cross-platform advancement opportunity here, to begin migration to a CMake-based spec for building Python.

Modern CMake has come a long way, modern cmake files are a lot more declarative and focused on creating ADGs rather than messes of variables that need regex manipulating.

I still hate cmake, but it does at least do it’s job these days, and it’s MUCH easier to reason about.

Having a modern build-system description, e.g a CMake project, would be a boon to organizations that either want to embed Python, integrate building extensions cross-platform, integrate end-to-end or vendor-testing, etc.

  • Maintain autoconf because people already rely on it and for edge-case e.g embedded hardware devs CMake does not support,
  • Maintain PCBuild for older windows and msvs version below VS17,
  • Introduce an umbrella CMake project for building and/bsd/ios/mac/lin/win etc,
  • Indicate that CMake is primarily targetted at non-vestigial development by selecting a CMake version such as 3.16 or 3.18,
  • Automate externals-fetching via CMake’s FetchContent or CPM,

If Python were to ship with a well-furnished cmake lists, building it - including fetching externals and building them - would be as simple as:

# MacOS build
ssh mac && cd python && cmake -B /share/mac_build . && cmake --build /share/mac_build && cmake --install /share/mac_build
# Linux build
ssh lin && cd python && cmake -B /share/lin_build . && cmake --build /share/lin_build && cmake --install /share/mac_build
# Windows build (ssh -> powershell)
ssh win && cd python && cmake -B s:/win_build . && cmake --build s:/win_build && cmake --install s:/win_build
# Windows cygwin build
ssh win && cd python && cmake -DCMAKE_TOOLCHAIN_FILE=../cygwin.cmake -B s:/cyg_build . && cmake --build s:/cyg_build && cmake --install s:/cyg_build
4 Likes
2

Agreed. Also CMake would be a large improvement on autoconf on Unix.

Of course, the problem is that it’s a large and tedious endeavour that needs a champion.

3 Likes
3

One problem is that it would be yet another build tool to maintain. However, after some years, one could probably get rid of PCBuild, so it would level out (after a while).

3 Likes
4

FTR:

1 Like
5

I believe this might be offset by the fact that people reliant on opposite ends of the spectrum would gain reasonable freedom to contribute changes relevant to themselves. Individuals supporting 20 year old kiosks running DOS can let rip on the .bat and MSVC6 support, while people trying to build with vs2022 can add the necessary tweaks to .cmake files without terrifying the bejeebus out of the former set.

6

(aside: I’d seen both first, I actually thought there would have been more)

Both of these came from a somewhat different angle of “lets just use cmake!” without concern for legacy cases and rightly got shot down because cmake doesn’t cover everyone, there are alternatives that have different overlaps, etc…

If you were to teleport today’s cmake back to the 1990s, Guido would still be right to tell it he would fart in its geenerale de-rektshun.

It’s the transformation in win/mac environments, the “well, you can do it manually” only support for ios/android in autoconf, and the amount of work going into treading water for mac/win builds that I think presents a fresh reason to consider relegating autotools to second place and introducing a modern build system to streamline the vendor/ci/test integration, and I think cmake, having the widest platform coverage, is likely to be the candidate.

7

Actually, this does not ring true.

PCbuild is all generated files - visual studio .vcproj files - full of interpolated variable instances, meaning if you want to change from Static to DLL, you have to hand-edit every project file. (* even microsoft don’t expect you to do something this stupid, so it’s HARD to do thru the ide)

In real-world terms, PCbuild is currently the equivalent to someone having run autoconf, removed the autoconf and autoconf.ac, checked in the generated files and Makefile.

What use case would be jeopardized by replacing PCbuild with a cmake config? Given cmake is available (or can be built) for every version of windows back as far as python itself supports, and has Visual Studio project generators going back all the way to VS6.

2 Likes
8

See also:

1 Like
9

I’ve been made aware that, my aggressive neutrality aside, I caused offense in describing hand-editing 49 generated files as “stupid”.

1 Like
10

As was brought up in some of the linked issues, if there is going to be a big build system switch, I imagine Meson should also be considered. It is written in Python, used by many projects, has a more modern, declarative config and uses Ninja as its backend. While I haven’t yet used it personally myself yet, a number of core scientific Python projects I have contact with are switching to it, e.g. SciPy, NumPy, etc. I’d be curious to hear your thoughts on why CMake and not Meson. Realistically, that would be one reason why this would need a PEP and a formal discussion process and decision.

3 Likes
11

Well, one reason is that requiring Python to build Python may not be practical everywhere.

2 Likes
12

That becomes less and less of a problem as time goes on. Python comes on Macs and Linux machines (or is installable via the preferred system packaging tools). Windows users can always install a binary version to start. I assume most of the core devs who might need to build Python from source probably have an executable or two lying about which would be up to the task. :wink:

1 Like
13

@pitrou raises a good point. The main issue, as I understand it, is not for individual devs but for Linux distros, which usually have a policy of building everything from source rather than relying on prebuilt binaries, so it would make bootstrapping Python more complicated (you’d need a previously built Python to build a new Python, and I’m not sure all their policies would allow that…but they do manage to handle bootstrapping C compilers, so its not impossible). There’s also the question of porting Python to a new architecture or OS—in order to build Python, you’d need to have Python already built, which again creates a bootstrapping problem (unless you cross-compile, and you’d face some of that anyway with getting Python itself working regardless).

1 Like
14

The main issue, as I understand it, is not for individual devs but for Linux distros, which usually have a policy of building everything from source rather than relying on prebuilt binaries…

If that’s the case, I presume the distro maintainers can bootstrap their way into Python for version N+1 using Python for version N as their executable. I’m still not seeing a big problem here. Maybe this is another reason to write a PEP for this, so these issues can be enumerated and addressed, with inputs from the parties you expect to be affected.

Edit: One other thing. If this sort of thing is a problem for distro maintainers, how do they deliver both GNU Make and GCC? You need Make to build GCC, but to build Make, you need some kind of C compiler, presumably GCC in the common case. Chicken, meet egg.

1 Like
15

It’s kind of a moot point now: Issue 45950: Reintroduce bootstrap_python for freezing - Python tracker is implementing bootstrapping for 3.11. :slight_smile:

1 Like
16

Why does it make it a moot point?

1 Like
17

Only that, as things stand, the current plan is for all Python 3.11 builds by default to automatically bootstrap themselves so, in theory, distros and other builders should not have to do anything special to bootstrap their builds for 3.11, unlike as was being discussed above.

18

Note the context of this discussion: this is was in response to the suggestion of using Meson, a build tool implemented in Python, to build Python.

19

I believe that making Python a prerequisite for building Python needs to be carefully discussed.

On platforms, where Python is readily available, this is likely not a problem, but what if you want to port Python to a new platform which is not supported yet ?

You’d have to rely on cross-compiling tools and make sure that the build system knows the target environment thoroughly – not very likely if the build system doesn’t run on the target platform.

Or you port an older autotools based version of Python first, then port the build system, and then port the new Python version. That’s a lot of work for the luxury of using Python to build Python :slight_smile:

The current autotools based build system may not be the most attractive, but it has been around for ages and only requires a basic C compiler to get its tooling ported to a new platform.

CMake needs a C++ compiler, so the tool chain is a little less portable, but not that much these days. CPython doesn’t need a C++ compiler, so it’s still a bit awkward to require C++ just for the build system, though.

1 Like
20

We’ll always need non-Python based build system capable of just building a minimal CPython required to run any Python based build system if we chose to adopt one. Never assume an earlier Python must be available on a platform. Leave that kind of mess for bootstrapping the C/C++ compilers only, or for runtimes that want to declare a forever-stale release that they also maintain for bootstrapping purposes (ex: golang 1.4.x?). We should not aspire to that level of complexity.

This doesn’t rule out Meson, just makes it clear what simpler build pre-requisite would need to exist. We’d also want a tested guarantee that such a Python based build system is satisfied by our minimal boostrapping interpreter build.

There are no platforms in the world that matter for anything the size of CPython without a working C++11 compiler. This is a non-issue. It isn’t awkward. What is awkward is that Python continues to be written only in C.

4 Likes