How to make setuptools work correctly with HPy

Hi,

We currently have two issues to solve with setuptools when porting to HPy.

Issue 1
We have ported a bunch of packages to HPy, which usually have “ext_modules” defined and thus the auto-discovery of setuptools>=60 doesn’t apply. One example is kiwisolver. When porting to HPy, ext_modules is replaced with hpy_ext_modules. This means that the auto-discovery is not disabled, causing setuptools to refuse to continue building the package with

error: Multiple top-level packages discovered in a flat-layout: ['py', 'kiwi'].

My current workaround is to specify an empty ext_modules list.

Question 1: what should be the way forward? Should packages migrating to HPy make sure they work correctly with the auto-discovery mechanism at the same time? Should we also special case hpy_ext_modules in setuptools?

Issue 2
When creating a HPy universal binary, we create a stub loader for the binary, since it is not a standard CPython extension module, and thus needs to be loaded through HPy. We cannot use an importlib hook, because that would still require import hpy before e.g. import numpy could work with the ported version. Also, HPy can compile to the universal HPy ABI or the Python ABI, the latter of which does not require HPy to be involved in the loading. At the moment, we monkeypatch setuptools to write our stub loader here. This has broken before while updating setuptools, and I suspect it is not a supported way to hook into this process.

Question 2: Is there or can we create an official hook for writing a stub loader?

2 Likes

Hi @timfelgentreff.

If the intention is to disable auto-discovery, the most straight forward way is to specify packages=[], which explicitly tells setuptools there is no package to be discovered.

Setuptools provides an entry-point mechanism for extensions to hook into the build process. This requires the developer to opt-in, by adding the “plugin” to the build dependencies (pyproject.toml > [build-system] > requires).

There are many ways of doing that, one of them is to add a custom build sub-command. There is some discussion and examples in setuptools #2591.
You can use an setuptools.finalize_distribution_options entry-point to install a sub-command

def install(dist: Distribution):
    if should_activate(dist):  # pseudo-code
        dist.cmdclass.update(custom_build_step=CustomBuildStep)
        build = dist.get_command_obj("build")
        build.sub_commands = [*build.sub_commands, ("custom_build_step", None)]

I am not exactly sure I understood this, import hooks can be surprisingly flexible… A dynamic .pth file (i.e. a line starting with import) can be used to install a import hook during the Python interpreter startup. (But I understand that a stub might be the best solution anyway).

For how to add a pth file to your installation with setuptools, see

2 Likes

Thanks for those pointers, those are helpful already. I fear failed to mention some more context around what we are doing and why. Anto Cuni has created a more thorough explanation on the HPy github: Improve the integration with setuptools and the packaging ecosystem · Issue #370 · hpyproject/hpy · GitHub

One issue e.g. with creating our own subcommand I think is that we are actually piggybacking on the build_ext command and injecting the hpy_ext_modules into the ext_modules with some hacky modifications.

If you are replacing build_ext, then I don’t see a problem with using the same class to write the stubs.

Please just be mindful of consistently extending the methods that are part of the API (e.g., get_outputs(), get_source_files(), get_output_mapping(), etc …) for full compatibility.