Is there prior discussion around the build system of CPython itself?

(I hope I’m posting in the right section)

As with all old, large codebases it’s difficult to know the complete picture as an outsider, so pardon my ignorance if I’m bringing something up that was discussed and decided before. I couldn’t find any discussion related to this.

CPython has a build system which is a combination of autotools, a handwritten Makefile template and a ~2500 line setup.py script. On top of that, the Windows/MSVC build duplicates a big portion of this. For context: I’m bringing this up because I build CPython for a lot of different configurations (architectures X operating systems - around 18 total), with custom toolchains, dependencies, flags; and the assumptions in this build system make it more difficult than it should be. Very often this results in having to patch the build to accomplish something. There are build systems that allow customizing all that without modifying the build which is preferable. The volume of the code in the build system makes it very hard to approach and reason about the logic in it.

I think there is definitely room for improvement on this front and I was wondering if there is openness in the CPython project for revamping the build system (perhaps with CMake) - I would be happy to contribute.

2 Likes

CMake is not panacea but it would still be much better than the current setup IMHO.

Another possibility to explore might be the Meson build system, which is written in Python. I have no personal experience with it, but it seems they’ve ported Python as an experiment (perhaps one could actually contact the original author)

1 Like

I brought up the idea of moving to CMake back at PyCon US 2009 and if I remember correctly it was dismissed as not having enough people who knew CMake and whether CMake supported enough platforms as Make itself is.

And I believe Meson uses Ninja to do the actual building, so one question would be how portable Ninja is.

How many people actually know autotools? :wink: I probably edited our autoconf file a few times but I have no actual idea how the whole thing works under the hood (I’m not even able to tell the difference between “autoconf”, “autotools”, “autoreconf”… or whether there is a difference at all). Having worked with CMake for two years, I must say that I find it much more approachable - even if it’s still not high-level enough for my tastes.

Interesting. Ninja is a simple tool, so it should be quite portable. But of course the devil’s in the details.

3 Likes

How would you bootstrap and build Python on a CPU architecture or platform that does have a working interpreter yet?

I suppose that could be done through cross-compiling, though it’s an open question.

CMake and Ninja are used for Chromium and LLVM/Clang. They’re plenty capable.

What is really needed is for someone to untangle what actually matters from our autoconf and setup.py pile of code. Turn that subset of needed checks into something other build systems can use to make pyconfig.h and module building decisions.

An implementation of a new CPython build system that works on non-current macOS/xcode (whatever version our actual binaries use), a few Linuxes on notably different architectures, and a couple of pesky not-dead-yet BSDs would go a long way toward convincing folks of the benefits of such change. Until that work is done, we’re all just pontificating.

For Windows it was “easy” in that the platform is so constrained that a pre-made pyconfig.h or three makes sense.

We could ship pre-made pyconfig.h files for supported platforms, but that gets annoying fast when the number expands – and our number is large with esoteric platforms and architectures. I suspect there isn’t a large actual set of generated pyconfig values, but they’d need to be gathered, considered and managed. Code is going to be needed to detect and decide which to use regardless. A hardcoded mapping of os-platform tuples to configs would be a major source of maintenance pain.

2 Likes

I think in the 10 years that passed since then, CMake has gained a lot of ground in terms of people knowing it, and as it was pointed out in the thread, the cumulative knowledge of Autotools is most likely a fraction of CMake’s.

In terms of portability, both CMake and Ninja are pretty good. But CMake also doesn’t require ninja, it’s just nice to have (though I’m sure that’s not going to be the culprit). Is there a document listing the current officially supported platforms?

1 Like

Meson is an interesting choice. I’ve heard good things about it, but frankly, as a grumpy build engineer I would strongly prefer the boring choice, which is CMake today. CMake is more likely to be already installed in an environment and it’s something that more people are comfortable with. Don’t get me wrong, it’s a pretty terrible language, but as a build system it’s proven to be very successful. LLVM tooling & major IDEs support it out of the box as well.

Apparently someone already started a CMake build: https://github.com/python-cmake-buildsystem/python-cmake-buildsystem (no idea about the quality of implementation, but I’ll check it out).

Agreed.

An implementation of a new CPython build system that works on non-current macOS/xcode (whatever version our actual binaries use), a few Linuxes on notably different architectures, and a couple of pesky not-dead-yet BSDs would go a long way toward convincing folks of the benefits of such change. Until that work is done, we’re all just pontificating.

Apparently there is cmake build in the works here:

It seems pretty extensive; I’ll play around with it a bit and try to contact the author.

2 Likes

Here: https://pythondev.readthedocs.io/platforms.html

More generally, Python is expected to build with little hassle (perhaps minor porting work) on platforms with a C99 compiler and a POSIX stack (including POSIX threads). “Supported” platforms are a subset of that because that also implies passing the test suite, which is not something the build system is normally concerned with.

1 Like

Also, a reasonable course of action could be to keep the CMake and autoconf-based builds in parallel for a couple versions, until all issues are solved on the CMake side.

Does that mean bootstrapping the build tools as well? CMake is written in C++, so that would be a no-go in that case.

That’s a good question. I would hope the answer is “no”, but that would have to be discussed, I guess :slight_smile:

Easy: We shouldn’t waste our time supporting platforms that do not have a modern C++ compiler. AFAICT: g++ and clang are available for anything that matters.

2 Likes