So I’m starting to view this as a wheel-building front-end (which we have an API for in PEP 517) with an optional compiling back-end (whose API doesn’t exist but @uranusjr gave us a strawman for). That way we start to minimize repetition in metadata specifying, wheel-building, etc. and have a very clear separation of concerns for how to prep the files for a wheel and producing the wheel.
IOW an example would be the call chain of pip -> flit -> meson and all we are talking about is an API to make it standardized on how flit can call meson so that people can more easily switch in the compiling back-end they want instead of having the metadata/wheel-building aspect also be tied into the compiling part. Pip wouldn’t even need to care about this new API unless there was a reason we found for it to care (e.g. maybe editable installs would play into it?). So to pip it’s just calling flit, it just so happens that flit is calling meson to compile extension modules to end up in the final wheel.
This would also potentially let people keep configuration for setuptools while letting setuptools slowly get out of the compiler game by guiding people to use a different compilation back-end.
So the thing that comes to mind for me on this is ee are talking about compiling C code, so the cost of spawning a process is going to be greatly overwhelmed by the cost of the compiler. But otherwise we could say it’s up to the tools choose whether it must be done in a separate process or it should be done that way. If I remember correctly PEP 517 has that subprocess specification because we didn’t trust tools to not mess up. But if we are willing to be a bit looser on that requirement in this instance then people can just do what they think is best.
I will also say I hope people are not invoking their compilers that frequently in a row with no substantial change to their code as the pip test suite.