With PEP 517 it is possible to declare a build system for your project. This has lead to the development of new build systems that pip can now utilize to perform builds. They typically are terrible when it comes to extension modules. Should we recommend meson as build system for those cases and perhaps in general?
The existing systems have various issues. Setuptools setup.py often has a lot of imperative code and carries a lot of historical baggage. Flit is generally great for pure Python projects, but you can’t use it with extension modules. I can’t say much about Poetry as a build-system, but as far as I can tell it does not support extension modules either, and it’s a pain to bootstrap considering the amount of dependencies it has.
Some years ago the Meson build system was released. It’s typically used for projects where one may otherwise use cmake. Behind the scenes, it uses Ninja to perform the actual build. Compared to other build systems Meson is very pleasant to use. The syntax looks like Python and as a build system it contains already the routines you want for finding dependencies.
Obviously there is already support for building against Python. Furthermore, it’s great for cross-compilation.
In Nixpkgs we’ve noticed that an increasing amount of packages are switching to Meson, most notably Gnome and there is also a push from a part of the community to use it for the actual package manager so we could support Windows. I saw there is a package that allows one to use Meson as a PEP517 build system and the author opened a PR for pygobject to adopt it.
Meson is implemented in Python and wheels are available for it on PyPI. Also, wheels are available for the dependency Ninja for Linux/MacOSX/Windows. For an example of a project, I suggest looking at the source of the Python package providing support for Meson.
Edit:
I realized I should have mentioned Bazel as well. This build system is getting an increasing amount of traction. It is a far more complex system though and a heavy dependency so unless you really need it (say you have a project like tensorflow), we should not bother with it.
It’s hard to know what to say to a question as general as this. What build system to use is really a question for individual projects, and if you’re interested (for whatever reason) in having more projects use meson, then you’d need to start by persuading some projects to use it, likely by offering PRs and demonstrating the benefits to them individually. As more projects use it, there’ll be a critical mass where it’s easier for new projects to see the benefits. But I’m not sure what “Python packaging” as a community could do here.
Maybe if you have concrete actions in mind that you want groups like the PyPA, or the developers of Python packaging tools, to do, you could describe them explicitly?
I think the suggestion here is that that meson could be a tool that’s one step above flit and a decent alternative to setuptools. Not sure what’s actionable here TBH, so, I’ll second @pf_moore’s question.
For actually deciding what system that would be, I agree with you it is best to begin with getting some projects to use it. Would it be useful to create an overview comparing the various build systems? I think it may, considering the various topics that touch upon the build systems.
I think it might. But I’m not sure where it would be best to publish it. The point I didn’t make very well in my previous post is that I think the people on this list (what’s the way of describing “the group of people who participate in a given category on Discourse”?) are more packaging tool builders than people who create and build real end user projects (with a few exceptions and some people who cover both roles), so promoting meson here could end up missing a substantial part of your actual target audience.
The key thing here is that packaging.python.org is intended to describe current best practices, rather than impose or promote change. For a good example of that, look at the discussions over the use of a src layout for projects - there is no commonly agreed best practice, so there’s no community standard for the packaging guide to describe. Similarly here, the approach should be to get community consensus that meson is “the preferred best practice”, and then switch the packaging guide to describe it (it’s fine having meson in an “alternative tools” section before that happens, but for it to be the recommended approach it needs to have already gained community consensus).
These groups appear to be the gatekeepers, so you’re being asked for permission to promote it as a backend.
And I know we don’t actually think we’re gatekeepers here, which is why the question doesn’t make much sense
But a concrete action would be to mention Meson in our other discussion threads that typically only mention flit and setuptools. By regularly only talking about those two, we give the impression that only those two matter/count/are “approved”.
Of course this doesn’t scale. Neither does repeating the explanation that we’re not actual gatekeepers here.
So probably the right response is to say what you already said: thank you for bringing it up, feel free to promote it yourself, we’ll add it to the list of commonly used tool when it’s a tool that is commonly used.
Well, OK then - if someone wants to promote it as a backend, they have my permission, for what it’s worth. And no, I’m not being sarcastic, it genuinely never occurred to me that anyone would feel the need to ask permission, but if they do, I’m happy to give it.
That’s a fair point. But for myself at least, I use setuptools and flit as examples because they are the only ones I know enough about to use as an example. I’d happily use meson as an example, particularly if it illustrates points that using setuptools and flit can’t, but I’ve got no idea how it works so I really can’t.
I did a quick search through the references linked in the original post, and got basically nowhere working out how I’d write a “hello world” level C extension that used meson rather than setuptools to build, and then build a wheel with it (on Windows and Linux - and MacOS, although I have no means of testing MacOS instructions).
I’m OK with adding meson to the list of things I can mention as example backends (along with enscons, which I also keep forgetting to mention) without going into details, though.
Does meson have support for running as a PEP 517 backend and producing sdists or wheels? I can’t actually tell from this thread.
There’s also scikit-build, which is CMake-based.
I think everyone would be happy for packaging.python.org to host a big list of known build backends. Maybe one of the maintainers of these systems would like to get that started?
There was a link to this package in the original post. But following that link and skimming some of the docs left me not much wiser. My assumption is that if you know how to build your project using Meson, and have set that up already, then mesonpep517 lets you trigger that build through the PEP 517 APIs. But how you set things up so that your project builds via meson seems to need more meson knowledge than I could get from a quick look at the meson docs - the project seems to focus on far more general build situations, and there’s no “quick start” for a Python module build that I could find.
It looks like mesonpep517 is a lot like enscons in that it adds wheel support to an existing build system. Very cool. We don’t know how to create a build system that everyone will prefer, so we let pip invoke any build system instead.
Actually, I have a question: what is so hard about building Python extension modules that setuptools must exist to build those .so/.pyd files?
And this is a serious question because I’m not sure of anything that unique to how to compile a C source file into a Python extension module that would suggest a more C-specific solution wouldn’t necessarily be better, and so I want to make sure I’m not missing anything. Now if there’s nothing special, the question should be asked as to why we are sticking with setuptools as our solution. I can understand historical precedent from the early 2000s of needing a cross-platform build tool to help build C code and package up Python code, but we have come quite a way since setuptools was created over 15 years ago.
Basically I’m asking if we should take a step back and ask ourselves whether we would still write a build tool like setuptools today? Like why doesn’t setuptools emit ninja build files and lets ninja manage all of that now instead of managing the entire build process itself? If the answer to all of this is “we wouldn’t do this that way today, but there’s history”, then maybe it’s time to stop and take a hard look at what we truly need to support in a custom fashion compared to what we can leverage from outside of PyPA. That way we can lighten our load and potentially get ourselves into a better position to really focus on the stuff that’s truly Python-specific. Because otherwise it seems to me that for extension modules the problem we should be how to know where to grab some .so file to include in an eventual wheel then how to execute clang.
As you know setuptools / distutils have been criticized for conflating. It is everyone’s least favorite kind of flating. That means it is less useful to talk about them as a single thing and more useful to talk about specific things that they do. In enscons we use the part of distutils that tell us compiler and linker flags including those related to the interpreter configuration, where the C compiler is, and what file extension extension modules should have. Over the years some of this information has become available in other modules not called distutils like https://docs.python.org/3/library/sysconfig.html . This information gets fed into the underlying build system to do the actual shelling out.
There isn’t anything really hard AFAIK, but setuptools / distutils automate a couple things:
choosing the right compiler and compile flags for the current platform and Python install
telling the compiler where to find the Python headers and shared library (if applicable, for the latter)
devising the target filenames and location for compiled extensions, depending on the Python install (ABI tags, etc.) and the build options
I’m probably forgetting a couple more. But much of the complication on the C extension side (the build_ext command and its internal dependencies) seems either legacy or self-inflicted.
On Windows, distutils locates the C compiler without needing the environment to be set up. This is hugely useful, as it means you can pip install from source at a “normal” command prompt rather than needing a “Visual Studio Compiler” prompt (and if you use Powershell as your shell, like I do, you can’t even use the vsvars batch files to activate the compiler toolchain). I would imagine most build tools could do this, though (although whether they actually do is something I haven’t checked).
CMake does this, and I would be surprised if Meson or Bazel didn’t. However, the subtlety here is that you want the right compiler for the current Python version (or is that not important anymore with the new CRT thing?).
But couldn’t we simply expose those in sysconfig or something (if they are not already there) so they can be passed down to a C compiler as appropriate?
As long as this is static information, then yes. But perhaps there are things (mostly destination file location, I guess) that are invocation-specific, such as in-place install vs. regular install.
Meson itself not. Meson can build extension modules and be used to put files in designated places. The Python package mesonpep517 however provides support for using Meson as build system, and it includes functions for building a wheel and sdist.
Interesting, I was not aware enscons did the same but for scons, thanks!
The Python module in meson uses sysconfig for this as well.
I gave it a try. The setuptools instructions are wrong, you need to removepyproject.toml as otherwise you get an error that you can’t use --no-use-pep517 because pyproject.toml specifies a build system that isn’t setuptools. After doing that, the setuptools build fails with an unresolved external PyInit_greetpkg/greet.
The Meson build failed with:
ERROR: Command errored out with exit status 1:
command: 'C:\Users\UK03306\AppData\Local\Programs\Python\Python37\python.exe' 'C:\Users\xxxx\AppData\Local\Programs\Python\Python37\lib\site-packages\pip\_vendor\pep517\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\xxxx\AppData\Local\Temp\tmpqfjdlofd'
cwd: C:\Users\xxxx\AppData\Local\Temp\pip-req-build-j2iznc90
Complete output (63 lines):
Could not run meson: The Meson build system
Version: 0.52.0
Source dir: C:\Users\xxxx\AppData\Local\Temp\pip-req-build-j2iznc90
Build dir: C:\Users\xxxx\AppData\Local\Temp\tmpv1mkmrcu
Build type: native build
Project name: greetpkg
Project version: 0.1
meson.build:2:0: ERROR: Unknown compiler(s): ['icl', 'cl', 'cc', 'gcc', 'clang', 'clang-cl', 'pgcc']
The follow exceptions were encountered:
Running "icl " gave "[WinError 2] The system cannot find the file specified"
Running "cl /?" gave "[WinError 2] The system cannot find the file specified"
Running "cc --version" gave "[WinError 2] The system cannot find the file specified"
Running "gcc --version" gave "[WinError 2] The system cannot find the file specified"
Running "clang --version" gave "[WinError 2] The system cannot find the file specified"
Running "clang-cl /?" gave "[WinError 2] The system cannot find the file specified"
Running "pgcc --version" gave "[WinError 2] The system cannot find the file specified"
Looks like Meson can’t find Visual C. I didn’t run vcvars.bat, so I tried again in a Visual Studion 2019 “x86 build tools” prompt, and got:
command: 'C:\Users\xxx\AppData\Local\Programs\Python\Python37\python.exe' 'C:\Users\xxx\AppData\Local\Programs\Python\Python37\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\xxx\AppData\Local\Temp\tmpposwuqd4'
cwd: C:\Users\xxx\AppData\Local\Temp\pip-req-build-j5aowfis
Complete output (95 lines):
Could not run meson: The Meson build system
Version: 0.52.0
Source dir: C:\Users\xxx\AppData\Local\Temp\pip-req-build-j5aowfis
Build dir: C:\Users\xxx\AppData\Local\Temp\tmpwsycwb8q
Build type: native build
Project name: greetpkg
Project version: 0.1
meson.build:2:0: ERROR: Compiler cl can not compile programs.
At this point I gave up. I’m not sure how relevant any of this is, as from the discussion it seems clear to me that the right way of looking at this is that if you find Meson a good fit for your project, and you’ve set the build up that way, using the Meson backend is a fine idea. But it should remain a project choice.
Now if there’s nothing special, the question should be asked as to why we are sticking with setuptools as our solution. I can understand historical precedent from the early 2000s of needing a cross-platform build tool to help build C code and package up Python code, but we have come quite a way since setuptools was created over 15 years ago.