Revisiting the case for CMake as a primary, cross-major-plats, build config

kfsone · January 4, 2022, 2:27am

The notion of it installing anything is self-evidently a strawman argument injected into the conversation. It’s nothing more complex or alarming or less expected than what pip, anaconda, homebrew, rust or golang do when they build from source, except that autoconf doesn’t do it.

That’s a link cited earlier on as an example of just what lengths people will/do go to in order to try and be able to create reproducible/embeddable builds of python.

Put another way: You went and looked at what 3rd party developers have to do to try and integrate Python into their projects, and you did not like it.

Because it’s not attached to the python project directly, that repository has all the burden and encumberment of trying to work around multiple versions and flavors of python and across multiple platforms.

Let’s for a moment assume that ‘./configure && make’ is sufficient for the majority of linux, mac and less-mainstream platforms.

Without looking, can you tell me how to build a static library on windows? And maybe then try it and tell me how that went?

A core point of this thread is that we don’t currently have one build system, we have 2 very different ones, three if you consider the scale of the Mac folder, and “many” if you look at the sheer number of user-built wrappers for the autoconf solution.

But let me finish by coming back to what I’m trying to ask here, by pointing to that link you gave again.

There are lots of examples like that out there, people who have had to go to insane lengths to be able to build python distros for embedding/vendorings under another build system.

Thanks to the many pull-requests and posts over the years where someone has tried to crap-out a half-baked cmsavebazeson build script that probably only worked for one version of PC/DOS build from 1987 these folks have put blood sweat and tears into building crazy, unwieldy multi-python-capable build project wrappers instead of contributing something much simpler and less painful to the Python project proper.

From which I draw the conclusion that for this to get any traction, there needs to be some preliminary discussion or decision by the core team: for anyone wishing to make such a proposal, what would merit or nix a proposal, or - sorry but I’m ending with a Python reference - what shall be the number of the counting?

gpshead · January 4, 2022, 3:18am

Thanks for the links. Otherwise you’re fundamentally misunderstanding me.

If you want a convenient “make fetch-and-build-deps” equivalent command to invoke any such thing, fine. I’d even use it. But default build targets should never cause “make fetch-and-build-deps” to happen. It must be explicitly asked for. sudo apt-get build-dep is but one example of a quite explicit method of doing it today. A build missing some dependencies should at most error nicely saying what wasn’t found and bonus, pointing out potential ways to get them.

Digression: cmake’s FetchContent at least looks better than what git submodules provides by not being tied to the version control system and allowing things that are not git repositories and not just only specific git hashes. I’ve written a small program to do exactly that before. Any build system could be made to invoke such a thing. I don’t consider this concept in any way exclusive to cmake. CPM.cmake itself is not cmake. It’s an add on. As could be done with any build system.

Reason I consider that a digression: Mentioning these dependency management tooling just complicates the discussions by making it less clear what is being talked about.

My other broader point is basically “show me the code”: Anyone who creates a reworked build system branch of the latest release or of main and demonstrates its usefulness and what it provides that we’re missing today is in a much better position to say “lets move to Thing, here is what we gain, here is what we lose, here’s how we adapt existing workflows for it, including for release managers, windows visual studio users, new classes of users and use cases it officially enables us to support XY&Z, etc”.

Example: That python-cmake-buildsystem repository, nice that it exists. Kudos to them. It seems very old (3.6 and earlier). Have they ever reached out to the cpython project itself ever or at least recently? Merely existing in a parallel world without consideration for the main project needs isn’t really a “show me the code” response. We already know it’s possible to build CPython with a variety of build systems. That may be fun, but isn’t the hard part. It’s the last mile details to make everything that we need replace everything important in existing system work that matter. Someone needs to do that grungy detail hard work.

kfsone · January 6, 2022, 5:01am

Fair enough, but they were your own interjection.

I’m not sure I’m clear what your objection is. You’ve mentioned ‘should not install’, and I’m not aware of “fetch-and-build-deps” doing that.

If what you are objecting to is the download & build of a package without any user input - as happens when you use pip or anaconda - it can simply be firewalled behind a cmake option, rather than being a reason to reject cmake itself.

option(Python_FETCH_AND_BUILD_DEPS OFF)
if (Python_FETCH_AND_BUILD_DEPS)
  include(cmake/fetch_and_build_dependencies.cmake)
endif()

I’ve worked on projects subject to ITAR, and the government auditors did not object to the use of pip packages such as numpy, cython, scipy, psutil, etc, despite the fact that “pip install psutil” may build from source. They didn’t object to pinned cmake/bazel projects that went off and fetch zlib, openssl, etc, as long as it required an option enabled on our part, or used a sha-pinned version.

I’m not trying to ram instant noodle soup down your throat here, rather I’m pointing out it’s ability most modern dev systems incorporate (e.g pip, et al) and a lot of people expect to happen automatically. I don’t mean to imply that makes it the right choice for how python builds.

kfsone · January 6, 2022, 5:33am

I refer back to the point I’ve made several times:

I’m not saying the core need to sit down and write out a spec, but every past attempt at discussing a switch has resulted in strong negative response.

We already have a build system, autoconf,
Visual Studio IDE generated project files are in the folder, they sort of work,
Supporting build system X means we have to support 2 [sic] build systems,
A handful of embedded developers will be inconvenienced by having one system that supports linux, bsd, darwin, raspbian, mac, android, ios, etc but might not be available on the PDP-11,
We don’t want to use a build system written in <Python|Java|Golang|…>
BuildSystemZ isn’t the current hawtness,
<Google|Facebook|…> doesn’t use BuildSystemP,
“I” don’t like 's syntax,

Please appreciate I am not judging the validity of any of those statements, I’m simply paraphrasing the first handful I found in older threads/discussions.

Allow me a step back:

I started out this thread while I was working on modernizing a game studio’s build system to update to Py3 and resurrect their Windows and Linux builds (they mostly build mobile and mac). After running into a number of issues, I looked around to see what others had done for incorporating py3 into their codebases/vendoring, leading me to former colleagues and friends at a number of large tech companies who’ve had to tackle this themselves and ended up choosing to do their own, siloed, isolated private solution, and one who is actively working on the public bazel python rules.

“That’s weird”, I thought, “why would they do that”, and I looked around at a bunch of other public efforts. Some of them are abominations, but others were serious attempts.

I reached out to numerous authors, and the general feedback was “they weren’t interested”.

My hunch is that they took a stab at setting up a wrapper, read a few old forum posts and got the sense is not an option, and never bothered more than asking ‘is there any interest in’ and having no particular reason to fight to gain that interest. It’s certainly a way to build oneself some bus-value if your employer can’t build python without the 20k lines one calls a build script

Again: the links are examples of amounts of work done, they are by no means representative of what an actual in-tree builds-its-own-revision config would look like.
e.g

Q1. Is there a list of platforms that must be supported?
Q2. Are there any build systems that are explicitly verboten? (e.g autoconf would be a bit pointless, wouldn’t it?)
Q3. Are there any well-known constraints on the choice of build system: (is python-based an option or does the bootstrap issue make that a no? can it be in java and all the dependencies that brings in? can it not use components that have international trade embargoes on them?)

gpshead · January 6, 2022, 6:47am

Q1. CPython has never had an official list, we’ve had core dev conversations suggesting we define tiers but I don’t see that as ever resolving. The best definition is in PEP-11. Rule of thumb: look at the buildbot fleet. Those existing and usually passing on the most recent CPython release branch (3.10) implies importance.

Q2. The build system should not impart licensing complications on the project or developers. I’m not aware of any that anyone would consider that would.

Q3: I’d be against any build system requiring Python itself. I think that rules out SCons? We don’t want a bootstrapping issue. I’d personally be against anything requiring Java as Oracle is actively harmful to open-source and software licensing in general. Java is also a heavy footprint prerequisite that many people do not want installed on their machines. I could be convinced otherwise on that one though.

cmake is rather ideal in the same way that autoconf is. It is light weight with few dependencies of its own. It relies on other tools (make, ninja, etc.) to run the actual build after cmake has handled it’s autoconf ./configure equivalent step.

encukou · January 6, 2022, 12:26pm

You’ll always get negative responses. Python developers don’t have one common mind. (For definitive answers you can ask the Steering Council, but you’ll need a much more focused question to ask.)
IMO, none of the above are reasons to veto the effort. Instead you should see them as considerations that should go into selecting the best build system. (Which could be autotools – if there isn’t a compelling reason to change, status quo wins.)

We try to not block people from contributing support for a platform, and unfortunately there’s no official list of people who’ll yell at you for breaking their platform. That said, Victor has a pretty good list on his personal page.

Not by name, but most have some undesirable property or other.

All of those look like issues that’ll meet with heavy resistance.

But cmake doesn’t have any of those. If you like cmake and want to contribute a cmake-based build system, just go with it. Searching for a slightly better system doesn’t sound like a good way to spend time and enthusiasm.
I haven’t seen anything here that would disqualify cmake. But there are many tedious details to sort out, and trying this out probably is the best way to find what they are. A draft implementation would be helpful, because you already know the theoretical roadblocks we can think of.

Anyway, I think this discussion is too long already, and has been going in circles and bikesheds. If you want to push this forward, I suggest writing a PEP to organize (y)our thoughts. It doesn’t need to be a final spec – a summary of the proposal, the discussion so far, and the alternatives with their pros & cons, should be enough to get the discussion back on track.

Livius90 · May 8, 2022, 10:28am

Hi all!

Is there any news about to develop a CMake build system? Some days ago, I wondered how easy it would have been to use a standardized CMake project for CPython, when I developed and test my time.sleep() improvements in Windows and Linux too.

In 2022, now, the most state-of-the-art platform independent build system can make via CMake. Why I need to say it? Because CMake is a general pre-configurator script platform to generate any kind of supported “build system” (make, ninja, VS project). You can easily detect your operating system in CMake scripts, and you can provide alternative steps (CMake toolchain files) for Windows or Unix build-target.

The most important things, what generators are available in CMake, let’s see it in CMake documents (it is also a highly well documented, as Python docs and really similar in style)

As I see now, CPython uses Unix make in Unix OS like in Linux and macOS, uses VS project in Windows. In the document page we can see that these are all available to use in CMake, it means CMake pre-configurator script can generate automatically the same Unix make or VS project which are in CPython at the moment. Do not need to maintain them in separately anymore, you need to just concentrate to make a multi-platform CMake script to be able to generate the target build-system project (make, ninja, VS project) for any target OS/platform, and you are ready.

More over it is possible to use immediately any kind of other build-system like Ninja, Eclipse CDT, Kate… So, maybe you can get some benefits and in surprise you can support any kind of marginal platform without any extra effort.

In my work, I am maintaining some Yocto project to make custom Linux distribution to any kind of ARM based SoC. As I see now, the CMake is the best and most popular to make an automatic build process for rpm, deb … etc package in Yocto, because easy to implement it to be OS independent. Most of modern open-source SW project likes to use it, always.

Here are a small list of why is CMake can be the future building solution for CPython.

CMake generator available for Unix make, VS project (these are used for CPython, now) and many more, possible to support any other platform quickly in future.
CMake can fetch any kind of 3rd party dependencies, so you can fetch an external git repo, https, nuget … and if they have also CMake you can include them in your build process easily as a .dll, .so or static lib.
CMake can find library dependencies automatically in the dev PCs, so for example find_package(BZip2) will try to find and get the path of BZip2 for your build process.
CMake is able to detect your target platform, you can make alternative config steps for each supported OS.
You can easily configure any kind of supported compiler for CMake, you can force/limit it to be able to build only C, but in normally it is support C, C++, C# easily. Again a new benefit can be, if your CMake scripts are well written, you can use any compiler in any OS.
CMake is available in all Unix OS and Windows, moreover it is easy to use for any ARM SoCs too, which have a Linux system. You can use ARM cross-compilers gcc in CMake too, because gcc is supported in generally. There is a high-level support to use CMake for cross-compiling.
Most of CI systems are fully support CMake to use for daily build and for others.
GoogleTest for unit-test and others are available in CMake, you can make unit-test via CMake, it can be implemented in OS independent also.
Many IDEs can support CMake project, for example Eclipse, VS code, Visual Studio Professional 2019 or later. So for in imagine, It means, I can easily open a CPython CMake project in Windows platform via VS code or VS Professional then if I configure well my ARM cross-compiler GCC and target Linux sysroot in the environment variables, CMake can provide me a full editable/buildable cross-developement project in Windows. So I can edit the CPython code in Windows, and I can change the target platform/compiler in any time then test the build, it would be awesome.

My opinion, autotools is a things of the past, it can not provide platform independent solution, so CMake is the future in C, C++, C# developing because it can provide it, it is doing well the same in Linux and Windows.

Livius90 · May 8, 2022, 1:31pm

There is an open-source project which are gorgeous well in CMake, let’s look it and learn from it.

Let’s see how nice and simple the building commands for all supported OS, moreover, thanks to that you can open it just in one click in a CMake compatible IDE like VS Code, VS Professional and so on:
https://xgboost.readthedocs.io/en/stable/build.html#building-the-shared-library

gpshead · May 8, 2022, 4:03pm

We’re all well aware of cmake. We don’t need a sales pitch. What we need is a core developers willing to actually work on it with a goal in mind of it replacing the existing trifecta of builds we have along with autoconf across all of the platforms we must support. Nobody’s had a motivating reason to replace what already works even if occasionally annoying with something else.

Livius90 · May 8, 2022, 6:47pm

Author and maintainer of this repo as an “supplier” could help you in all, dont you think or plan to take contact with them? As i see they are done in most of things pretty well, and actively developing it. It can be good idea to take over their great work to be it to official in CPython.

gpshead · May 8, 2022, 7:54pm

Please read the history of this discussion before repeating it.

erlendaasland · May 27, 2022, 12:36pm