Revisiting the case for CMake as a primary, cross-major-plats, build config

The Python Windows build manifests everything we non-Windows users expect to see in a Windows build system, and no contact with the reality of how well building on Windows is today in the real world.

Out of the necessity of supporting the most ancient of legacy dos/win/msvc configs, it uses BAT files instead of modern, open-source, cross-platform, POSIX Shell-specification-based Powershell CORE scripts; It is presented as a series of visual studio solution and project files so complex that people are building complex tooling to make simple changes to it like enabling static libraries: python-build-standalone/build.py at 3307e3920cd6cd404629899adadfcf248c5ca992 · indygreg/python-build-standalone (github.com)

As the gap between the oldest and newest supported VS editions grows wider, the PCBuild folder becomes more and more frustrating to everyone, doing proper service to none.

Modern Visual Studios (2017+) treat CMake and Ninja as first-class citizens and ship with them, while CMake Generators have been capable of creating visual studio projects for quite a long time.

I believe there’s a cross-platform advancement opportunity here, to begin migration to a CMake-based spec for building Python.

Modern CMake has come a long way, modern cmake files are a lot more declarative and focused on creating ADGs rather than messes of variables that need regex manipulating.

I still hate cmake, but it does at least do it’s job these days, and it’s MUCH easier to reason about.

Having a modern build-system description, e.g a CMake project, would be a boon to organizations that either want to embed Python, integrate building extensions cross-platform, integrate end-to-end or vendor-testing, etc.

  • Maintain autoconf because people already rely on it and for edge-case e.g embedded hardware devs CMake does not support,
  • Maintain PCBuild for older windows and msvs version below VS17,
  • Introduce an umbrella CMake project for building and/bsd/ios/mac/lin/win etc,
  • Indicate that CMake is primarily targetted at non-vestigial development by selecting a CMake version such as 3.16 or 3.18,
  • Automate externals-fetching via CMake’s FetchContent or CPM,

If Python were to ship with a well-furnished cmake lists, building it - including fetching externals and building them - would be as simple as:

# MacOS build
ssh mac && cd python && cmake -B /share/mac_build . && cmake --build /share/mac_build && cmake --install /share/mac_build
# Linux build
ssh lin && cd python && cmake -B /share/lin_build . && cmake --build /share/lin_build && cmake --install /share/mac_build
# Windows build (ssh -> powershell)
ssh win && cd python && cmake -B s:/win_build . && cmake --build s:/win_build && cmake --install s:/win_build
# Windows cygwin build
ssh win && cd python && cmake -DCMAKE_TOOLCHAIN_FILE=../cygwin.cmake -B s:/cyg_build . && cmake --build s:/cyg_build && cmake --install s:/cyg_build
3 Likes

Agreed. Also CMake would be a large improvement on autoconf on Unix.

Of course, the problem is that it’s a large and tedious endeavour that needs a champion.

3 Likes

One problem is that it would be yet another build tool to maintain. However, after some years, one could probably get rid of PCBuild, so it would level out (after a while).

2 Likes

FTR:

1 Like

I believe this might be offset by the fact that people reliant on opposite ends of the spectrum would gain reasonable freedom to contribute changes relevant to themselves. Individuals supporting 20 year old kiosks running DOS can let rip on the .bat and MSVC6 support, while people trying to build with vs2022 can add the necessary tweaks to .cmake files without terrifying the bejeebus out of the former set.

(aside: I’d seen both first, I actually thought there would have been more)

Both of these came from a somewhat different angle of “lets just use cmake!” without concern for legacy cases and rightly got shot down because cmake doesn’t cover everyone, there are alternatives that have different overlaps, etc…

If you were to teleport today’s cmake back to the 1990s, Guido would still be right to tell it he would fart in its geenerale de-rektshun.

It’s the transformation in win/mac environments, the “well, you can do it manually” only support for ios/android in autoconf, and the amount of work going into treading water for mac/win builds that I think presents a fresh reason to consider relegating autotools to second place and introducing a modern build system to streamline the vendor/ci/test integration, and I think cmake, having the widest platform coverage, is likely to be the candidate.

Actually, this does not ring true.

PCbuild is all generated files - visual studio .vcproj files - full of interpolated variable instances, meaning if you want to change from Static to DLL, you have to hand-edit every project file. (* even microsoft don’t expect you to do something this stupid, so it’s HARD to do thru the ide)

In real-world terms, PCbuild is currently the equivalent to someone having run autoconf, removed the autoconf and autoconf.ac, checked in the generated files and Makefile.

What use case would be jeopardized by replacing PCbuild with a cmake config? Given cmake is available (or can be built) for every version of windows back as far as python itself supports, and has Visual Studio project generators going back all the way to VS6.

2 Likes

See also:

1 Like

I’ve been made aware that, my aggressive neutrality aside, I caused offense in describing hand-editing 49 generated files as “stupid”.

1 Like