How do we get out of the business of driving C compilers?

Packaging
6

Aside: I found this issue and it seems that Flit was entertaining a similar idea inspired by Cargo. Maybe @takluyver has thoughts on this?

The API should likely be specified parallel to PEP 517, so a PEP 517 backend can interface to it (to build wheels), but a frontend can also access needed metadata (for e.g. editable/develop install) without proxying them in PEP 517.

Continuing the Cargo analogy, here’s how it handles the build script:

  • Locate the build script (defaults to build.rs and can be configured through. This can be left to implementations (using the tool field in pyproject.toml) for now.
  • Compile it with build dependencies.
    • This is sort of analogous to the build environment set up by PEP 517 frontend. We can use PEP 518’s build-requires to declare them. Backend doesn’t need to worry about this, and can invoke the build script directly.
  • Run to build.
    • Cargo does not make any efforts ensuring whether the build tool (e.g. CMake) actually exists. I guess that’s the most reasonable thing to do, like Setuptools also makes no effort ensuring compilers exist.
    • It might be worthwhile to emit better error messages, like how Setuptools improved from the cryptic Unable to find vcvarsall.bat. That’d mostly involve creating wrapper Python modules to call those tools (instead of straight subprocess). Still this is not necessary for now, either.
  • Interact with the build script.
    • This can be done in either like Cargo or PEP 517.
    • Cargo way: One entry point with results categorised via specially prefixed stdout
    • PEP 517-like: Specially-named functions for each information kind.

A PEP 517-like interface might look like this (illustrative only, likely missing pieces):

  • get_paths_triggering_build(config_settings=None)

    Return a list of strings specifying items on the filesystem, relative to the project root. Frontend is expected to call build_for_dev if any of them is modified later than the last build.

  • build_ext(build_directory, config_settings=None)

    The hook is expected to write files into build_directory, and return a list of 2-tuples. Each 2-tuple represents a file to include in the package:

    • The first item is the path to include this file in the wheel (same as the first column in RECORD).
    • The second item indicates where to find that file, relative to build_directory.

    The frontend is expected to pass a consistent value of build_directory across each call to this function. The hook should expect the directory already containing previously-built files, and may choose to reuse them if it determines they do not need to be rebuilt.

1 Like