Optional C Extension handling

Packaging
1

In aiohttp and family, we provide optional C Extensions with Pure Python fallback.

Now we have a setuptools custom build_ext command which detects compilation errors and silently switched to pure-python installation.

The problem is:

  1. Performance without C Extensions is significantly worse than in normal case (e.g. C written HTTP parse is much faster than pure-python one).
  2. We provide MacOSX, Windows, and manylinux binary wheels but many people use Alpine docker images to install aiohttp. manylinux is not Alpine compatible because Alpine uses musl instead of glibc.
  3. People do pip install aiohttp on a platform that has no compatible binary wheel published on PyPI and Compilation Toolset is not available (gcc is missing, Python.h is absent etc.)
  4. aiohttp install succeeds but it switches to pure-python fallback version.
  5. it can hurt the performance of the production server easy without any notice, people should pay for more powerful servers but they can avoid it if notified.

I’d like to fail loudly if installation on CPython implementation, PyPy works fine with pure python.
On the other hand, I’d like to provide a way to explicitly say: “Ok, I know that my environment cannot compile C Extension but I can live with it”.
For aiohttp, C Extension compilation should be default but opt-out.

pip supports --install-option <options> for pip install command, I can utilize it and parse in setup.py.
I can raise an error with the suggestion to use --install-option --no-extensions to enforce pure python version.

The problem is: --install-option is not supported by *requirements files`.
Would pip maintainers consider adding it?

Another solution is using environment variables but it looks ugly: currently, we have aiohttp, yarl and multidict libraries in our stack, all three have optional C Accellerators:

NO_AIOHTTP_EXTENSIONS=1 NO_MULTIDICT_EXTENSIONS=1 NO_YARL_EXTENSIONS=1 pip install -r requirement.txt

Sorry, too many variables and again: no way to pin this information inside requirements file.

Providing two packages like aiohttp and aiohttp-slow is not an option too:
a user should decide what version to use, not an author or third-party library.
The third-party should have just aiohttp in dependencies. Recall psycopg2 vs psycopg2-binary famous problem.

It would be nice if pip has an easy and obvious solution to this.

Any recommendations?
Did I miss something?

2

The opt-out part is hard. You can provide a pure Python wheel and then use --no-binary on e.g. Alpine, but I don’t think that fits with your constraints.

1 Like
3

This is very related to the concept of “Opportunistic dependencies” in general. See this topic suggestion for the packaging mini-summit, which links to this packaging-problems issue.

As I mentioned in that thread, I think this a more general problem, where there are other possible “nice to have” style dependencies that you want to be either opt-out and/or “soft failing” other than extension modules.

2 Likes
4

I talked to Andrew about this a bit yesterday. One idea that came up: this is actually a pretty generic problem – there are a fair number of packages out there that have accelerator extensions + pure python fallbacks. Since it’s a generic problem, maybe there should be a generic solution.

For example: we could add a --allow-slow=<package list> option to pip. If this is given, then whenever compiling a package from source that is given in the list, pip would pass "pypa.allow-slow": True through the PEP 517 config settings dict. For legacy setup.py-based packages, it would pass … some other thing, I’m not sure what setup.py accepts here. --allow-slow=:all: would cause this option to be passed to all source builds.

1 Like
5

pganssle I’m asking not for dependency but an option to pass from requirements.txt to setup.py or build-system in general.

I have C Accelerator in the same code base alongside with pyre-python fallback. It simplifies development and testing a lot.

Extracting accelerators into another package is technically possible but makes supporting such approach very hard.

1 Like
6

To provide a possible pointer and in case people are curious, here’s an example of a PR I recently wrote that adds support for a different option in requirements.txt: Support --use-pep517 and --no-use-pep517 inside requirements files by cjerdonek · Pull Request #6447 · pypa/pip · GitHub

At least in this case, it was a tiny change. The file req/req_file.py contains an allow-list of options (called SUPPORTED_OPTIONS) that can be used inside a requirements.txt file. (There is also a separate SUPPORTED_OPTIONS_REQ variable, which differs in some way.) Does anyone know the reason for limiting what options can be used inside a requirements file?

7

It’s a common problem, but is it really generic? I would think that the different requirements for “I need/want/suggest the faster version of X package” vary quite a it among different projects.

If anything, reified extras seem to make more sense an (even more) generic solution for this that could accomodate far more use cases. Being able to install aiohttp[allow-slow] as some kind of Provides: aiohttp package could give a lot greater granularity to maintainers to make the decisions about what kind of fallback rules are permitted for different contexts, provided the extras information gets passed to the build backend.

8

aiohttp[allow-slow] looks great but there is no aiohttp-slow library.
IFAIK pip doesn’t call a specific hook in setuptools but uses extras_require to install additional libraries.
What I need is a flag available somehow in setup.py (or another build system) to opt-out disabling C Extensions compilation