Till now the sure way to obtain the compilation flags required to compile Python extensions has been to look at what distutils does and do the same. For example, to check whether the extension module should be linked or not with libpython Meson uses this probe:
Python 3.12 removes distutils thus this strategy is not available anymore (unless setuptools is installed to get distutils back from the side door, but it is my understanding that eventually distutils will not be exported anymore by setuptools) and another solution needs to be found.
It is always necessary to link to libpython on Windows. For Python 3.8 and later, on POSIX platforms, in principle, sysconfig.get_config_var('LIBPYTHON') is the name of the library to be linked or an empty string for no linking to libpython.
However, on Cygwin and Android sysconfig disagrees with distutils when Python is not built shared. In this case, distutils never links to libpython, while the former act the same as a shared build.
IMO, this is precisely what sysconfig should be exposing - information about the Python interpreter, how it was built and configured, etc. It may not cover everything yet, and its API may not be ideal, but those are issues that can be solved, and @FFY00 is working on them. It’s just been a little slow, because until now people were using distutils and so not reporting any issues for sysconfig.
Yeah, Paul’s right. sysconfig should be able to provide the information that was used to build the current runtime, so that other tools can use it or derive it for building other compatible binaries. If it’s not doing that today, that’s a bug that should get fixed (once someone figures out how to fix it).
What changes without distutils is that we are no longer promising to return working command lines for particular compilers from the standard library. Those things change over time, and can vary by platform independently from Python version, so it just makes more sense for a separately versioned tool to produce them.
For now, setuptools is your best bet, but I hope we see other helper libraries develop over time (probably in the vein of “helper library for invoking a particular compiler” rather than a complete build backend, but we’ll see what people need badly enough to develop!)
The issue has been raised many times already but it seems impossible to find someone with the expertise and knowledge required to provide the information that would need to be encoded in sysconfig or at least review patches. The pull requests I linked to in the previous message have been filed in December and one is still waiting for a core developer able to review it.
I’m among those that is trying to create a better alternative to distutils and setuptools to compile python extensions (by the way, I am sure that “helper library for invoking a particular compiler” is not remotely close to the right way forward, but it doesn’t matter). However, finding the information required to create such an alternative so far has been impossible. Every time the issue that sysconfig is not remotely enough has been brought up, the answer is that distutils is the best source of information. I wonder why then it has been decided to remove it from the standard library, if an equivalent does not exit.
Building Python extensions without relying on distutils requires understanding the pile of hacks in distutils, doing a lot of archeology to understand which ones are still relevant, which ones are plain wrongs and worked around downstream, and still come up with a solution that most likely is not going to work in all cases.
I didn’t realize this was the case. Because distutils has specific provisions for these systems, I thought that they were supported. Unfortunately the intersection of people with interest on alternatives to setuptools and expertise on for building python extensions on these systems is vanishingly small.
I think it matters - maybe not to this immediate discussion, but in general. Would love to hear more of your thoughts on it.
Yes, but at least it won’t be frozen at 3.12 beta. So at least as new issues are fixed, they can be made available immediately.
This is a big reason we wanted to get rid of distutils - it would lead people into a false sense of security on stuff like this!
Don’t I know it! The intersection of people with this interest and expertise on Windows is also vanishingly small Hopefully with distutils out of the way, and projects having a more sustainable way of choosing a different build backend, we’ll be able to grow it.
I’ve merged one of the PRs, and approved the other, which I’d like someone else to also have a look at.
I agree, it’s very tricky though. Just exposing compiler flags like proposed on the issue linked seems like a temporary solution to me, as it’s compiler specific, and users would need to translate them to other compilers (this includes other C compilers, or compilers for other languages, like Rust). Ideally, we’d have a way of exposing details of the build, instead of relying on exposing compiler flags.
I can review sysconfig patches, the issue is that currently a lot of the information people rely on comes from the makefile instead. It’s difficult to keep up with the CPython repo, so I semi-regularly search for issue/PRs that might involve sysconfig, but it’s still very easy to miss stuff like this. Feel free to ping me in those cases. PRs on sysconfig itself already ping me automatically, but not the rest.
Perhaps we could try to contact those projects to see if they’d like to have a point of contact?
You can ping me. I already reviewed the PR and pinged people internally to see if anyone is available to have a look, if it no-one merges it in a week or so I’ll merge it myself.
I agree. Actually, like I mentioned above, I’d like for sysconfig to get a way to expose build details without it being compiler-specific flags.
I think that Meson does a great job at abstracting away the peculiarities of different compilers (and for what is possible even different languages). What it needs is just a way to know where to find headers, which libraries it needa to link, and where these libraries are. Some large projects distributing Python extensions are migrating to Meson and they seem happy. meson-python has the story of taking a Meson project and packing it up into a wheel covered. I find these two tools much nicer to work with than the alternatives.
I think the request in the linked issue didn’t come across clearly. The issue asks for a way to access basic information without the compiler flags. The required information is: where the headers are, which libraries to link, and where these libraries are.
Traditionally, on POSIX systems, at least these information have been exposed via pkg-config .pc files which indeed contain compiler flags, and this works fine because all compiler available on these platforms accept the same set of basic options to describe these three things. On other platforms, if required, it is easy enough to parse these compiler flags to extract the required information and construct command line argument for other compilers.
What the issue asks is: if Python distributes pkg-config .pc files, can we please have them work in the same way on all platforms? .pc files are not the best solution, but it is a solution we have now and that can be easily made to work at least on some platforms.
I haven’t looked in detail but one of Eli’s complains is that python-config exposes way too many compiler options (IIUC it reports all compiler options that were used to compile Python itself) making it very hard to extract only the options that are required to compile extension modules. I think sysconfig is the way to go. There are a few cleanups to the Meson heuristic around comping extensions modules in the pipeline. Once these land I’ll see which parts of that could make sense to incorporate into sysconfig.
Hopefully with