Packaging a compiled executable

I’m trying to write a setup.py (or pyproject.toml/setup.cfg equivalent) for a project that contains an executable compiled for the target archicture. I’m hoping to support:

  • Creation of target-dependent wheels
  • pip installing the project to trigger the executable to be compiled when wheels are unavailable

More details on the setup I’m trying to achieve:

  • setup.py runs a compiler to create my-executable
  • My project depends on this executable, running it via subprocess.run(os.path.join(some_path, "my-executable"), where I’m unsure what some_path should be (determined by pkg_resources perhaps?)

I believe this is effectively what’s being asked in Compiling & installing C executable using python's setuptools/setup.py? - Stack Overflow, but I’m not convinced by the solution there (and installing the executable to a venv’s bin directory isn’t particularly the goal for me - I’m expecting it to be internal to the project’s package).

The executable is written in Zig in my case, and I plan on using ziglang · PyPI as a build-time dependency to fetch the Zig compiler, although I believe my question would be effectively the same if the executable were written in C/Rust/… .

I was wondering if the expectation for things like this is to declare them as ‘extension modules’, but I couldn’t see an obvious way to do that in this case.

If anyone could provide any pointers for how to go about this it would be much appreciated :slight_smile:

Yeah, you don’t want to customize the install command, that generates eggs, which are deprecated, and is not compatible with PEP 517. I would customize the build command instead. You can generate your executable and place it inside your package (in the build directory) so that it is shipped as a package resource.

Eg.

my_package/
    __init__.py
    my_executable

You can then access it via the importlib.resources API:

import importlib.resources

my_executable = importlib.resources.files('my_package') / 'my_executable'

See importlib — The implementation of import — Python 3.9.6 documentation
For Python versions older than 3.9, there is also a backport: GitHub - python/importlib_resources

Thanks, that makes sense.

I would customize the build command instead. You can generate your executable and place it inside your package (in the build directory) so that it is shipped as a package resource.

Are there any official docs on customising the build command? I couldn’t find any on a quick look - the closest I found is python - Hook to add commands to distutils build? - Stack Overflow, although this doesn’t make it clear how to control where the executable is placed.

Basically no, customising distutils has always been a matter of reading the sources and trying to work out what to do from that - and I assume the same remains true of setuptools. (Offtopic, but this is one of the reasons distutils was so hard to maintain - without a documented API, changing anything could be a compatibility problem…)

It should be the same as the install command – subclass the default build command and override run.

I think I’ve got this working by subclassing distutils.command.build.build, but I thought distutils was [in the process of being] deprecated? I couldn’t see an equivalent in setuptools.

This also doesn’t result in a custom platform tag being applied to wheels - I find myself manually changing e.g. from zig_minesolver-0.1.0-py3-none-any.whl to zig_minesolver-0.1.0-py3-none-win_amd64.whl before uploading.

I’ve decided I’m probably better off just making the project installable only with wheels, partly because the ziglang build dependency seems to be kind of slow to install (no fault of Python), but also this dependency is only available in wheels itself. I can then just write a script to iterate over target platforms performing the zig build (using cross-compilation) independently from setup.py, package up the compiled file as ‘package data’, and then rename the wheels.

A couple of related links I came across:

The latest setuptools versions have their own copy of distutils. It’s not fully integrated yet, because they want to be able to support code that keeps importing from distutils, but eventually their copy will be The copy.

The distutils in CPython (as of 3.10) will report a warning whenever it is imported. The fix for that warning is to make sure you have an up to date setuptools so that you get theirs instead.

1 Like