# Options to build the same package in different ways (with different build dependencies)? **URL:** \ **Category:** Packaging\ **Created:** [June 17, 2020, 9:32am UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458 "2020-06-17T09:32:01Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 17, 2020, 9:32am UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/1 "2020-06-17T09:32:01Z")
For some packages, we really need to have different builds (producing different wheels), with different build dependencies. My motivation is in particular about the fluidfft and fluidsim packages and we need to be able to build with and without Pythran and with and without MPI and mpi4py. I don’t enter in the details of why we need this because it’s going to be long to explain (I can do it if it is useful). The solution we choose was to build differently depending of the result of `import mpi4py` and `import pythran` with try/except statements. It is far from optimal but it works. Of course, it forbids the use of isolated build and therefore pyproject.toml. Note that it is similar to how conda works (the package installed by a command depends on what was installed in the environment). However, I feel that soon having a pyproject.toml will be kind of mandatory to stay mainstream and get good behavior with many Python tools. Unfortunately, all examples I see with pyproject.toml are very simple and I don’t see any mechanisms to specify options to build the same package in different ways (potentially with different build dependencies) so that it produce different wheels. For example, in conda the fftw package ([https://anaconda.org/conda-forge/fftw/files](https://anaconda.org/conda-forge/fftw/files)) is build as fftw-3.3.8-mpi\_openmpi, fftw-3.3.8-mpi\_mpich, fftw-3.3.8-nompi, etc. Could it be possible to support things like ```auto pip install fluidsim # build with pythran as build dependency (or get the wheel if available) pip install fluidsim[purepy] # wheel without extension and with no MPI support pip install fluidsim[pythran, mpi] # pythran & MPI ``` It seems to me that currently the brackets in `pip install` commands are not at all usable for this usage. I tried to read PEP 517/518 but I don’t understand how they apply to our needs. Note also that mpi4py does not upload wheels on PyPI (as far as I understand it is for good reasons because mpi4py needs to be recompiled for different MPI implementations). I know that it was a problem few months ago to add it in `[build-system] requires`, but maybe it’s no longer a problem? I hope someone here could give me some good advice. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [June 17, 2020, 10:35am UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/2 "2020-06-17T10:35:16Z")
My first question would be how you expect the different wheels (with different dependencies) to be named. When installing a package, installers should be able to look at the wheel filename and determine if the wheel is the correct one to install (that’s the point of the filename standard and the packaging tags in the wheel spec). If you need different wheel filenames, you need to argue for a revision to the wheel spec. If you can explain how the package index would store these variant wheels _without_ a filename spec change, then that would probably clarify how installers like pip could be told which variety to choose. ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [June 17, 2020, 11:17am UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/3 "2020-06-17T11:17:41Z")
Could you misappropriate the local version identification? eg, `3.14.15+mpi12`, `3.14.15+mpi13`, `3.14.15+nompi`, then have exact version specifiers. My suggestion would be to have multiple subdirectories, each with their own “pyproject.toml” etc (setup.cfg, setup.py, libs), then sym-link (or copy, for peace of mind) in the source. This wouldn’t work if you have different package source code between the different builds. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [June 17, 2020, 3:53pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/4 "2020-06-17T15:53:57Z")
I just posted the seed of an idea that may help you (one day, if/when it ever happens): [Idea: selector packages](https://discuss.python.org/t/idea-selector-packages/4463) Would be interested in your thoughts. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [June 17, 2020, 4:49pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/5 "2020-06-17T16:49:42Z")
> [@EpicWink](#): > > Could you misappropriate the local version identification? Installers wouldn’t be able to do anything other than treat these as part of the version, so they only really work well if you pin exact versions. Christoph Gohlke’s wheel repository for Windows used to use this scheme (I don’t know if it still does) and it worked OK, but not perfectly. And that was for relatively casual use. So maybe, but probably only if you’re looking at a well-constrained environment, not for general use. ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [June 17, 2020, 6:55pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/6 "2020-06-17T18:55:32Z")
Wouldn’t that be a good use case for the `Provides-Dist` metadata? For example one could publish both projects `MyProjectPure` (pure Python) and `MyProjectC` (built with C extensions). And to link them somehow on the _metadata-level_ they could both announce as `Provides-Dist: MyProject`. This way tools (installers) could react appropriately and for example make sure only one or the other is installed. ---
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 17, 2020, 7:24pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/7 "2020-06-17T19:24:12Z")
Thank you for you answers and suggestions. I understand that there is no proper way to do that (even though `Provides-Dist` could be useful). A very hacky way would be to modify `pyproject.toml` dynamically to be able to publish different packages (with different package names and different build dependencies) from one unique source. It is strange (and unfortunate) to have to do this by hand but it could work. For example `fluidsim-purepy` could depend on `fluidfft-purepy` and `fluidsim[mpi]` (here, just an `extras_require` since there is no extension using MPI in fluidsim) could depend on `fluidfft-mpi`. ---
**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\ **Post date:** [June 25, 2020, 5:24pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/8 "2020-06-25T17:24:54Z")
This feels like a desire for some sort of “variants” feature that I’ve seen pop up before. Maybe it’s something we should be trying to push on in some way to get something figured out. ---
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 26, 2020, 1:42pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/9 "2020-06-26T13:42:07Z")
It seems to me that it could be useful if tools like `pip`, `poetry`, `pipenv`, etc… could use a `pyproject.toml` dynamically produced from one string. Of course, the production of the `pyproject.toml` file has to be done without external packages but it is not really a bad limitation. Just pure python and the assurance that the python code is run from a root directory of the project. In my case, I can use different `pyproject.toml` files to produce different variants of a package. These variants can be uploaded to PyPI under different names. It’s very hacky but so far so good. However, there is only one `pyproject.toml` in the root directory of a repository, so only one variant of the package can be installed directly from the repository (with something like `pip install git+https://github.com/serge-sans-paille/pythran#egg=pythran`). ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [June 26, 2020, 7:16pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/10 "2020-06-26T19:16:04Z")
I don’t think you need any sort of dynamically produced `pyproject.toml` file. If you are building from source, what you want to do is available _today_, because PEP 517 has a mechanism for backends to dynamically provide build dependencies: `get_requires_for_bdist_wheel`. You can already use `setuptools` to achieve this; here’s an example `setup.py`: ```python from setuptools import setup import os import sys GREP_TARGET = "x11234" prefix = f"{GREP_TARGET} ({sys.argv[1]})" try: import attr print(f"{prefix}: Attrs imported!") except ImportError: print(f"{prefix}: Attrs not imported.") if os.environ.get("BUILD_FLAG", None) == "1": setup_requirements = ["attrs"] else: setup_requirements = [] setup( name = "mypkg", version = "0.0.1", setup_requires = setup_requirements ) ``` Pair it with an appropriate `pyproject.toml`, and you can use an environment variable to determine whether `attrs` is present while the wheel is built: ```bash (venv) $ pip install . -v 2>&1| grep "x11234" x11234 (egg_info): Attrs not imported. x11234 (dist_info): Attrs not imported. x11234 (bdist_wheel): Attrs not imported. (venv) $ BUILD_FLAG=1 pip install . -v 2>&1| grep "x11234" x11234 (egg_info): Attrs not imported. x11234 (dist_info): Attrs imported! x11234 (bdist_wheel): Attrs imported! ``` The real problems here are not about providing a mechanism to do this (as you can see, such a mechanism already exists), but more about recording metadata about it. How do you tag the wheels? How do you tell `pip` which version you want? How do you declare dependencies on this? I don’t think there’s any good solution to the metadata/labeling problem that exists _today_. I would definitely love to see support for something equivalent to `Recommends-Dist` (optional dependencies where the default is some flavor of “install it if you can”), which would make it easier to have a bunch of build variants that are selected either with extras or as a chain of fallbacks, but that’s just my own personal wish list. 😃 ---
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 26, 2020, 10:33pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/11 "2020-06-26T22:33:57Z")
I understood pyproject.toml and the isolated builds were used in particular to avoid `setup_requires` and having to specify build dependencies in the file `setup.py`. I also read that there are potential problems with `setup_requires`. I would prefer something cleaner than using both `[build-system] requires` plus `setup_requires`. You mention `get_requires_for_bdist_wheel` and PEP 517. It would be good to provide somewhere simple but realistic examples of how to use these things. I find only the PEP on the web (maybe I was just not able to find the good resource) and this part [Build backend interface](https://www.python.org/dev/peps/pep-0517/#id7) is not simple. For example, I guess it should not be very difficult to reproduce your example (build dependency depending on an environment variable) with `get_requires_for_bdist_wheel` but how? > How do you tag the wheels? How do you tell `pip` which version you want? How do you declare dependencies on this? Yes, an environment variable is not sufficient. Something like `pip install my-super-package[a-string-about-the-variant]` would be nice. Of course the name of the wheel should somehow contain `a-string-about-the-variant`. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [June 27, 2020, 12:35am UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/12 "2020-06-27T00:35:48Z")
> [@paugier](#): > > I understood pyproject.toml and the isolated builds were used in particular to avoid `setup_requires` and having to specify build dependencies in the file `setup.py` . I also read that there are potential problems with `setup_requires` . I would prefer something cleaner than using both `[build-system] requires` plus `setup_requires` . `pyproject.toml` (specifically the `build-system.requires` array) is created to avoid `setup_requires` being the _only_ way to specify build-time dependencies, and thus `setuptools` being the only possible build back-end (because you need it to read the value), PEP 517 uses a two-step process to solve this: First, `pyproject.toml` specifies what back-end to install and run; once you’re in the context of a given back-end, two hooks `get_requires_for_build_wheel` and `get_requires_for_build_sdist` are used to specify additional build-time requirements. For `setuptools`, they simply provide whatever is specified in `setup_requires`. There is no point to avoid `setup_requires` here since it provides the functionalities that you want—a way to specify build-time dependencies dynamically determined at build-time. If you’re inclined to avoid `setuptools` entirely, indeed you can also quite easily write your own PEP 517 back-end that uses those hooks to specify what you want dynamically. But you don’t really need to do that if you’re calling `setuptools` under the hood—`setup_requires` provides the exact same functionality anyway. ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [June 28, 2020, 3:39pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/13 "2020-06-28T15:39:28Z")
To add to what @uranusjr said, the primary reasons you should generally not use `setup_requires`: 1. Anything specified only `setup_requires` cannot be required dependencies for invoking `setup.py`. If you have `import somepackage` in your `setup.py` not guarded under a `try/catch` or an `if` statement, the build machinery that installs your dependency won’t ever get invoked, because `seutp.py` will fail before it calls `setup()`. 2. When invoking a legacy build (i.e. _not_ a PEP 517 build), `setuptools` will do the installation with `easy_install`, which can mess up your Python environment and has many other major flaws. In your case, neither of these applies, because you are necessarily doing a conditional import, since your intention is to be able to do a _full build_, even if the package isn’t installed, and as @uranusjr says, `setuptools` will just add anything in `setup_requires` to the requirements when `get_requires_for_build_wheel` is invoked, and `easy_install` will never be invoked. In any case, my point wasn’t really “here’s how you can accomplish what you want to accomplish”, because I think that outside of very limited circumstances, build flags of this sort tend to circumvent the expectations of a lot of tools. My point is that in the development of any “variant builds” functionality, the focus should be on the metadata problem, because it’s already technically possible for backends to “dynamically” modify the build requirements (and indeed for `setuptools`, already implemented). ---
**Author:** ![pganssle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pganssle/32/245_2.png) [@pganssle](https://discuss.python.org/u/pganssle)\ **Post date:** [June 28, 2020, 3:48pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/14 "2020-06-28T15:48:44Z")
To follow up on my earlier recommendation of `Recommend-Dist`: This may just be my preferences and biases speaking, but I think this whole issue _really_ comes down to the fact that we don’t have a way to specify a `Recommends-Dist` [“opportunistic” dependency](https://github.com/pypa/packaging-problems/issues/214), nor any good way to say, “here are 5 things that can satisfy this dependency”. If you could do that, then with a little extra work, you should be able to refactor your packages to allow for “build variants” in most cases by structuring them as a core package with a bunch of “plugins”. Right now, you can already achieve the “build variants” _with_ metadata, by splitting your package up into four packages: ```auto fluidsim fluidsim-purepy-backend fluidsim-pythran-backend fluidsim-pythran-backend-mpi-support ``` Then you would declare that `fluidsim` depends on `fluidsim-purepy-backend` (or just have `fluidsim-purepy-backend` ship with `fluidsim` unconditionally), and have `pythran` and `pythran-with-mpi` extras that add dependencies on the other two. The problem is that in my experience most people who provide multiple packages that do the exact same thing are doing it because they have one package that provides wide compatibility but is slow (e.g. a pure Python version), and one that is faster or better in some way, but is otherwise hard to build, only works on some platforms, etc. Generally you prefer the _default_ to be to install the faster/better one _if possible_, and if not, fall back to the compatibility mode. But there’s no way to do that, because extras can only add rather than replace existing dependencies. If we had `Recommends-Dist` and a system for saying what groups of packages can satisfy a given dependency, you could get the default values you want with the correct fallbacks. For example, you could add a dummy package `fluidsim.backend`, which can be satisfied by either `fluidsim-pythran-backend` or `fluidsim-purepy`, and a `recommends-dist` on `fluidsim-pythran-backend`, so that if `fluidsim-pythran-backend` is unavailable or hard to install for whatever reason, you fall back to `fluidsim-purepy`. The extras `pythran` and `pythran-with-mpi` would add _hard_ dependencies. Obviously this is a bit more constraining than a situation where you invoke different build options on the core package itself, but it has the advantages that it’s _basically_ how feature flags are implemented with extras _now_ and it has some advantages: 1. It discourages excessive build-time complexity, because having N tightly-coupled feature flags would require creating and maintaining N _separate packages_ (I see this as a benefit rather than a downside). 2. It would make the true dependency graph (including feature flags) more amenable to static analysis. 3. Nothing would need to change about wheel tagging, because the variant features are encoded into the package name. 4. It would likely play “nicely” with other packaging systems, because it doesn’t really introduce a new “package variant” abstraction — the structure of these packages and their dependencies wouldn’t need to change much or at all when translating them into package management systems that only have the concept of a single “package” that has dependencies. ---
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 28, 2020, 8:03pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/15 "2020-06-28T20:03:48Z")
> as @uranusjr says, `setuptools` will just add anything in `setup_requires` to the requirements when `get_requires_for_build_wheel` is invoked, and `easy_install` will never be invoked Thank you for this explanation. I didn’t understand that and I thought `setup_requires` would still use `easy_install`. ---
**Author:** ![paugier](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/paugier/32/2353_2.png) [@paugier](https://discuss.python.org/u/paugier)\ **Post date:** [June 28, 2020, 9:22pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/16 "2020-06-28T21:22:08Z")
@pganssle thank you for your long and interesting answer. I now approximately see how with pyproject.toml, setuptools and environment variables, the same package could be build in different variants (for example `fluidsim-purepy-backend`, `fluidsim-pythran-backend`). I’m still not sure that it would be supported to change the name of the package using pyproject.toml, setuptools and environment variables (the name should not be written in pyproject.toml but computed dynamically in setup.py) but I guess it should work. (In practice in my case, I wouldn’t use a package `fluidsim.backend` and `fluidsim-purepy-backend` and `fluidsim-pythran-backend` would both contain the whole fluidsim code, but anyway, it is not important). But then for the dummy package `fluidsim` (which would only be used to depend on `fluidsim-purepy-backend` or `fluidsim-pythran-backend`), I would need to compute the runtime dependencies from the list of extras (and maybe from other information, like the availability of wheels for one platform, like “Is `fluidsim-pythran-backend` available or not?”). Even without considering this extra-complication (availability of wheels), is it possible to get the list of extras in setup.py and compute runtime dependencies from it? I’ve never managed to do that… ---
**Author:** ![takluyver](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/takluyver/32/513_2.png) [@takluyver](https://discuss.python.org/u/takluyver)\ **Post date:** [August 4, 2020, 5:53pm UTC](https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/17 "2020-08-04T17:53:56Z")
I’m [trying](https://github.com/h5py/h5py/pull/1619) to apply the recommendation to use `setup_requires` for dynamically determined build dependencies in h5py (specifically, optional MPI support). Unfortunately, it’s not quite the case that with PEP 517, `setup_requires` are only passed through to the frontend. They are exposed through the `get_requires_for_build_*` hooks, but if they’re not satisfied when the other hooks are called (as they may not be if you use `pip install --no-build-isolation`), setuptools cheerfully starts trying to install them itself. I believe this is a bug in setuptools - when it’s being a PEP 517 backend, it shouldn’t be trying to install anything itself. I’ve [filed an issue](https://github.com/pypa/setuptools/issues/2303) about it.