That’s what the build system behind say flit would do. So e.g. meson or scons or cmake. Of course, you could write something that directly invokes a compiler, up to you.
I’m sorry, I don’t follow. How is a generic build system, like meson or cmake, to know these details of the Python interpreter’s build configuration, if Python is no longer in the business of communicating this via the distutils.* modules?
E.g. from the sysconfig module.
1 Like
That could work. sysconfig.get_config_vars() on the computer where I’m typing this (Linux, Python 3.7) dumps out 670 variables and it’s not at all obvious which ones are relevant, but that’s a simple matter of writing documentation.
I just remembered that a long time ago I wrote a script that works out how to build extension modules using a boring old-fashioned Makefile: https://github.com/zackw/tbbscraper/blob/master/analysis/get-module-compile-cmds.py If there were a documented way to do what that does, cross-platform, using only sysconfig, I would be happy.
1 Like
I’ve been successfully using this simple Makefile (originally by @tiran) and it’s seemed to be okay: https://github.com/zooba/spython/blob/master/syslog/Makefile
I have a more complex Cmake script for the same thing that I could extract and share this part of it once I’m back at work.
1 Like
OK, now that the discussion has died down, my question is how serious are people about this idea? What are next steps? Is there anyone else we want to bring into this discussion (e.g. @takluyver)?
I’m interested, but I don’t think I have a lot of useful input at this point.
As @uranusjr mentioned, the interface I’ve toyed with for Flit is to run a build script on a copy of the source files (those that go in the sdist), and then package up the result as a wheel. This would cover platform-independent build steps (e.g. minifying bundled Javascript) as well as platform specific things like compiling C code.
The tricky bit is that something then needs to decide what tag to apply to the resulting wheel, which depends on the build steps. So the build script can’t be a totally opaque ‘run this and package the results’ interface. I can think of ways to pass this information back, but it messes up the neat idea of the build script as a simple, reliable thing. As far as I know - which is not far - cargo doesn’t have this problem, since Rust code is always built for a specific target platform, so there’s nothing like a py3-none-any wheel.
Maybe rather than making a separate layer for build tools behind the packaging backends, we should instead:
- Define a standard table for static metadata in pyproject.toml, so each backend doesn’t need to specify its own
[tool.flit.metadata]or[tool.mesonpep517.metadata]table. - Better encourage sharing low-level functions for things like putting a collection of files in a wheel, so build backends focus on the building parts.
As a backend author, though, I’m less keen on point 2 - it means outsourcing critical functionality to somewhere where it’s harder to control. Part of what’s nice about working on Flit is that I can easily change almost anything above the level of the zipfile module.
How does setuptools decide what tag it is building for? Maybe we can take ideas from it?
With mesonpep517 the meson tool generates a list of files that are to be installed during configuration time. mesonpep517 then checks if there are any files that look like extension modules based on their suffix, and uses that to decide whether it is pure or not.
So in the case of meson it can be done by a relatively low computation introspection step. Apparently cmake also has such functionality (https://stackoverflow.com/questions/25904074/how-to-get-the-list-of-files-that-will-be-installed-when-installing-a-cmake-comp).
I suppose one could require a command that is used to generate the to-be-installed files. If one then wants to offer a Makefile-backend, that is still possible, but the Makefile would have to offer some target for constructing that list of files.
This has been brought up in the last week and the immediate thing people object to is being told how to specify the version; some like Flit’s __version__ and some want a VSC-driven solution. Personally I would love to standardize this stuff, but we will have to be very careful and clear about what is going to be tool-specific and what will be universally supported (along with what is required and what is optional, i.e. if you leave the version out can the tool then just handle that detail?).