I believe we have a general interest in at least finding out if Muon &| Meson can be a viable replacement for our configure.ac/Makefile.pre.in mess. If it were not destined to replace it so that the old autoconf+make based system could be deleted, I doubt we’d convince enough fellow of our core team to want to adopt it as that would just increase our build system maintenance burden.
If it could handle all of our PEP-11 Tier non-Windows build systems that’d be impressive, including our various configure flag enabled builds. That’d leaves (1) understanding the needs of release builds from our release managers perspectives and (2) seeing if it would be feasible to get it to also work on similar non-tier platforms.
Yes, this is my experience as well. Maintaining multiple build systems is not a good idea. However for projects as large as cpython there will have to be a multi-year transition period to give users time to update their ci/scripts/etc..
I also fear that there will be some obscure operating system that meson doesn’t (and cannot for whatever reason) support, however the PEP-11 document you’ve linked is a lot more reasonable than I’ve expected, given that the compiler/system evaluation has checks for weird stuff like AIX, MINIX or SunOS/Solaris.
Is there any particular reason why it should just handle the non-Windows systems? It obv. won’t work out of the box, but with some adjusting (also checking msvc compiler flags, not unconditionally including all of the “Modules with some UNIX dependencies”, …) I do believe it could be made to work.
Mostly, despite my hopes and dreams, I wouldn’t block on a new build system also supporting Windows. If it does work there, great, but lets not presume that we can actually get rid of the need to maintain Visual Studio build system support as I expect that is central to various workflows on Windows. Getting off of autoconf+make onto something we perceive as more maintainable and performant is the desirable carrot to chase.
I’m far less worried about the transition period for configure+make users. We should not consider those to be public APIs so it IMNSHO shouldn’t need to drag out for years in our main branch once we’ve decided to switch. (we’d decide that as part of the replacement build system PEP that discussions and trials like this should turn into)
There might be reason to have some simple workalike shims that invoke meson under the hood while waiting for some build invocation things to be updated, but even that feels questionable. Updating calls to a build system should not be hard.
Our existing autoconf and make build setup has been carried along for many decades through different eras of computing platforms, so conditionals for all sorts of less relevant or even obsolete things do still exist within even though we’ve occasionally attempted to remove some.
I think it reasonable that - if cpython were to adopt meson - there ought to be some CI job that ensures its compatible with muon, as muon is generally more broken and less tested (In order to get my cpython meson script to build using muon I had to patch its source that it looks for lib64 instead of lib for example).
But muon itself is not necessarily very well-tested or platform-agnostic. According to their README:
Additionally, muon is not fully supported on all platforms yet. The current status may be viewed on muon’s ci dashboard. Platforms with some or all tests disabled are currently WIP and platforms not tested in CI have unknown status. In general, posix systems should work fine. Windows support is improving slowly.
(and, practically, it seems to have a single maintainer)
It only needs to work on one OS. We can use the python it built to run meson to cross-compile cpython for other systems and architectures. Cross-compiling with meson is usually significantly less of an issue than it is with autotools, esp. if you have an ‘exe_wrapper’ (wine/qemu) available, as this allows meson to do compiler/system evaluations by actually running a built binary.
But what I’ve noticed about bootstrapping is, is that at least the openssl subproject (taken from mesons wrapdb) uses python to generate a vs module definition file (also for non microsoft targets), which isn’t ideal. Not sure about the other required subprojects (bzip2, lzma, zlib, libffi), but if they do I didn’t encounter them during me building for Android and having PYTHONPATH set to Lib/:<builddir>/Modules/Dynamic (due to the beforementioned getpath.py issue) and this breaking my build machine python. This should probably be changed that there isn’t a requirement for a preinstalled openssl when bootstrapping python directly on a system that doesn’t have a cpython yet.
I’m very enthusiastic about the prospect of moving to meson/muon, and (from having gone through this with SciPy and elsewhere – incl. on windows), I’m convinced any eventual issues can be worked out. Thanks for exploring this further!
It’s possible to build openssl without python (I’d say: the default). Same for the compression libraries you mention. I know because we do so in conda-forge.
In the context of OpenSSL discussing meson as a build system (where the same bootstrapping concern for exotic architectures exists), one of the meson developers said
Well, but the person who said that isn’t a muon developer. At best, it can be read as an educated guess, not a promise of anything. Supporting a new platform is not trivial, especially in C.
That is still generally much more cumbersome than building directly on the target architecture. While cross-compiling is useful, my opinion is that it can not be used to hand-wave away the desire for running the build system natively on the target platform.
Yeah, basically this. So many Windows developers expect to be able to open directly in Visual Studio (and will often do the “git clone” from within Visual Studio), and we don’t really want to ask them to leave their IDE to download and build other tools with other random tools to generate project files.
If there’s a solution for generating projects that can be checked in and are fully relocatable, that could be considered. I doubt it would be able to cover the full range of functionality we currently have, or that it would make builds faster than the hand-written projects (or more reliable to patch at build time, as many do), but if it could meet all those criteria then it wouldn’t be impossible for core devs to use the tool when we modify the projects and check in updated ones.
I know it is (vs studio module definitions are kinda useless for any non-msvc target after all), its just that patched openssl with meson support (from wrapdb) doesn’t.
Sure, but you will obviously be able to use meson itself to build python on Windows. This is only about the hypothetical bootstrapping case “what if all python binaries where to be wiped from existence”. It also wouldn’t matter if the microsoft visual studio specific build system were to be retained.
I didn’t know about that being possible. While meson does have a backend that generates visual studio project files, I don’t think they’re made to be checked into VCS.
Yeah, they’re generally not (quite reasonably). CMake certainly doesn’t (I just rewrote the generated zlib-ng project into something that can be checked in for us), and even my own pymsbuild doesn’t.
Nonetheless, I think it’s a reasonable requirement for us. In general, we generate most files ahead of time and check them in.
(Another aside - we are currently generating pyconfig.h on Windows to try and properly handle free-threading, and it’s breaking VS users. So we’d want to find a way around that kind of issue, too.)
I was thinking more about the case “Python is being ported to a new platform which muon does not support, or not well enough”. I agree that Windows should probably not be a problem.
I don’t think this it’s in any shape realistic that you already have a preexisting OS, C compiler and a POSIX shell but no python. (There are obv. the systems that are classed as no longer supported in PEP-11 that fulfill this, but you wouldn’t want to port cpython to them).
If you already have to cross-compile a C compiler and a shell, you might aswell cross compile cpython too. If they don’t want to, requiring them to fix muon for their plattform doesn’t seem that major of an issue.
The vast majority of FOSS projects don’t have VS project files, and that hasn’t stopped people from developing with those on windows. I think the checked-in files are really bad[1]. They should be generated for the requested build orchestrator (whether ninja, visual studio, or whatever) on demand.
Note: this doesn’t mean that I think current functionality on windows should have to regress, just that generating the build files should be table stakes for a next-generation build system (and of course: visual studio can learn to use meson, just like it can use CMake).
Source: eternal suffering from patching checked-in visual studio files for conda-forge ↩︎
This just seems to be getting a little too nitpicky. I help maintain a
buildsystem that’s not even in the discussion here (scons), but if there
was an interest in using it for a major project like cpython, we’d
figure out, collectively, how to make that work. I’d be really surprised
if meson, cmake, ninja, bazel, or most of the other zillion buildsystems
wouldn’t have the same attitude. Yeah, we generate VS Project files
too, and they’re not perfect, and what can you do? I think those need
to be manually maintained for complex projects, autogenerating is
unlikely to be reliable enough.
I agree, hence my point that these problems can be fixed. At least aspirationally, getting away from these checked-in files should be an uncontroversial wish. I’d go a step further that these checked-in files really should go away concretely rather than aspirationally, but I also need to emphasise that I have no relation to meson except as a user and distributor[1], so don’t take what I say as representative of them.
suffering from way too much exposure to build systems already, including scons BTW ↩︎
Perhaps you need some other ideas about how to override them, then? My own build replaces all the externals with other sources/pre-built binaries, and overrides certain compiler settings without actually modifying the checked in files at all. (Though to be fair, I did make changes to the build files to allow or simplify this in some cases.)
The only thing I dislike about the current set of build projects is that they need to have each file listed individually. In part, that’s because we don’t have logical groupings of source files in our repo, but it’s also due to the C++ support in VS not handling wildcards nicely in projects.
Though ultimately, we’ll never get away from having a hand-maintained list of source files somewhere, and those lists are inherently different between Windows and other platforms anyway, so we likely get two lists.
You’d better put in that feature request with the Visual Studio team now, because it’s minimum 12 months from releasing, and another 3-5 years before we can assume that contributors will have it. And that’s assuming the team is motivated enough to support Meson (they’re not) or have heard of it (probably not). You’re severely underestimating the cost of “just” updating tooling here.
You’d better put in that feature request with the Visual Studio team now, because it’s minimum 12 months from releasing, and another 3-5 years before we can assume that contributors will have it.
It can’t be done as an extension? We already have Meson extension for VSCode.
VS extensions are significantly more work, due to the deeper integration and older platform (I used to own the Python extension for VS).
It could be done, but I don’t think we’d want to require our contributors to have to install an extension in order to be able to contribute. Productivity and success rate for first-time Windows contributors (e.g. at sprints) is quite high, but drops off quickly once things need to be installed. And developing in VS is currently a nice and smooth experience, where incremental build and code completions work reliably, and debugging has all the information it needs.
As I’ve said, if the experience can be preserved, it can be considered. Most responses to that have been “don’t check in [tool’s] generated files” or “make people install other things”, neither of which bode well for preserving the experience.
