Packaging platform-dependent binaries in wheels

Packaging
1

I want to optimise certain performance-critical parts of my application using C/Zig/Rust code and package the compiled code up in platform-dependent wheels.

I know that wheels support being platform-dependent (and am aware of the concept of manylinux). What I’m struggling with is the recommended workflow for creating these wheels.

I am aware of setuptools: Describing extension modules, but my understanding is that this specifically applies to python extensions written in C?

The project I’m trying to solve this in is minesolver, where my current approach is:

  • use a make-wheels script
  • this performs the build (cross-compile with zig build in my case)
  • it then runs python3 setup.py bdist_wheel --plat-name <platform-tag> to generate a wheel with the appropriate platform tag
  • setup.py includes the binary via the package_data

Concerns I have with this approach:

  • I’m trying to keep with the times, PEP 517, PEP 621, etc., and I know invoking setup.py directly is no longer officially supported
  • I have to come up with the platform tags myself and I’m worried I’m getting them wrong (e.g. I’m not sure it’s correct to be using manylinux1 given I’m not doing it in a manylinux container…)
  • Lack of clean developer workflow - I have to manually build the binary, editable installs don’t work properly?

Can anyone suggest anything better? I’m wondering if there’s some kind of hook for performing the build that can be implemented while remaining compatible with tools such as the python build package and non-setuptools backends.

1 Like
2

Hi Lewis, I am assuming you are already familiar with GitHub - PyO3/setuptools-rust: Setuptools plugin for Rust support right?
Does that not help with your use case?

3

Setuptools does custom support build steps: Support for custom build steps · Issue #2591 · pypa/setuptools · GitHub

Regarding a universal non-setuptools approach, I don’t think that is currently possible.

4

Thanks for the response :slight_smile:

I’m assuming this is Rust-specific - I’m more interested in the general approach, where I’m currently using Zig, but I’m also keen to improve my understanding of what’s possible in Python packaging.

It appears I had previously liked that issue! I hadn’t properly understood what it proposed until rereading and following through to your " Add ‘setuptools.sub_command’ entry-points" PR - this does look like it might be a reasonable solution for my request. Until it’s merged though, the support for custom build steps seems somewhat implicit, and must be done in a setup.py?

That’s not a concern to me really, I guess I was more interested in how it fits in with the pyproject.toml declarative config, which seems to be touched on in [FR] Proposal/Discussion: Minimising asymmetry between plugin and in-tree customisation/extension · Issue #2900 · pypa/setuptools · GitHub.

2 Likes
5

Thanks, this is the first time I am receiving feedback on those proposals :hugs:

Yes, right now you need to use setup.py. Which is fine to be honest…
Running python setup.py directly is deprecated, but the existence of the file itself is not.

The approach with subcommands will work with python -m build.

1 Like
6

As far as building wheels, cross-compiling them for different OSes and ensuring you get the right tags, cibuildwheel seems to be a solution to a lot of your issues on that front. Maybe @joerick or @henryiii , who help maintain it, could share a little more on that as my expertise in this particular area is rather limited.

1 Like
7

Indeed, cibuildwheel could fit your use case. I know that some users are using it to build rust-based wheels. I had a quick browse of our Working Examples list and Polaroid looks to be a reasonably simple example.

3 Likes