Specifying build variants for cython-enabled packages

Packaging
1

In my project I’m using the zeroconf package. The maintainer has been adding cython implementations of the functions for speed improvements.

I’m not in the business where I require the cython speed. I measured the difference for me and for some operations it’s not measurable and for others it’s less than 2%. The cython implementation does have a couple of downsides:

  1. The cython implementation generates rather big .so files, and I’m space constrained. I understand this does not apply to everybody. The wheels are now smaller than in previous releases of the project but it’s still 3.9M versus 743K for me on the last release of the project, which is a significant size difference (and which got me started investigating this in the first place)
  2. if there is no wheel for your platform, installation takes significantly longer: on my system it takes over three minutes! (up until the last release, there were no wheels for Windows)

The problem is, the way to not get the Cython-implementation to build is to set an environment variable SKIP_CYTHON=1. But I can not express this in a pyproject.toml or a requirements.txt file. And also, if there are wheels available, I’ll get those installed no matter the environment variable set and then I still have the large installation size. And if anybody else builds the project on a different python version or on a different architecture, or maybe without a compiler present, I’m not sure if they get cython or not, I can’t be sure. And this is not great for reproducibility.

I discussed with this with the maintainer here and he stated that

While isolated benchmarks may show modest improvements (~2%), the real benefits become apparent in production environments with heavy mDNS traffic. (…) The library is specifically designed to handle environments where incoming traffic processing would otherwise overwhelm the system (…)
I understand the concerns about wheel size and the desire for flexibility. We would be open to accepting a PR that adds pure-Python wheels as an installation option, allowing users to choose based on their specific requirements - whether they prioritize smaller package size or need the performance benefits for high-traffic environments.

But I think currently in python packaging, we can not have wheels for different installation types, and making an ‘extra’ that is pure-python is also not an option. I only see two possibilities:

  • publish a separate module that supports the cython implementation, which could be an ‘extra’, and then let the ‘regular’ zeroconf module load that module if available
  • publish a separate dist that is called zeroconf-purepython which does not come with the cython improvements

But I feel that Python packaging should support this use case in a more ‘native’ way, to make it easier for package maintainers to provide cython speedups.

Am I missing something? What would be your suggestions?

4 Likes
2

Hypothetically speaking: if there was a pure-python (-py3-none-any) wheel available:

  1. This wheel would only be installed on platforms where the installer cannot find a more specific + compatible wheel. In the case where python-zeroconf provides compiled wheels for all supported platforms, the -py3-none-any wheel would never be chosen by default.
  2. This wheel could still be installed/forced by passing e.g. --only-binary --abi=none to pip install (--no-deps and --target=... would probably also be needed).
  3. An added bonus for python-zeroconf is that new/future unsupported platforms would be automatically supported by this hypothetical pure-python wheel.

Point #2 means that you would no longer have to add SKIP_CYTHON=1 to the environment and perform the build from source yourself. However, instead you now need to pass custom options to the installation process, in order select the correct wheel. I don’t know if this is possible to express in a pyproject.toml or requirements.txt file. At least I don’t see a way to specify ABI tags here: Dependency specifiers - Python Packaging User Guide

So even if what you’re asking for should be possible within the current design of wheels and their tags, I’m not sure how you can automatically force that behavior when your own package installs python-zeroconf as a dependency…

Also, this depends on python-zeroconf being willing to support and distribute this -py3-none-any wheel in the first place.

1 Like