Provide optional packages using extras

Hi, We have a Python project, a restful API client for a web service. Alongside the API client, we provided a mock package for users to test their project with the service without actually connecting to the external service.

The mock package contains some test data, so users may want to leave them out of their production environment. So we want to provide the mock package using extras.

We have the following pyproject.toml file:

[project]
name = "gcp"
dependencies = [
   ...
]
dynamic = ["version"]

[project.optional-dependencies]
dev = [
   ...
]
mock = []

[options]
packages = ["gcp"]

[options.extras_require]
mock = ["gcpmock"]

[build-system]
requires = ["setuptools>=45", "setuptools_scm>=7.0.0"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.package-data]
gcp = ["py.typed"]
gcpmock = ["py.typed"]

We hope pip install gcp will leave out the gcpmock package. If users want to install gcpmock for testing, they can do pip install "gcp[mock]". If we use python -m build to build the package and then upload it to a PyPI server, pip install gcp will also install the gcpmock package.

I think one way to do this is: during the build, we generate the gcpmock as a separate package and then add it to the optional dependencies of gcp. So we will end up with two separate packages on the PyPI server. But those two packages can share a single repository and, more importantly, a single pyproject.toml. In this way, we can easily ensure the API methods’ signatures in gcpmock are matched with those in the real gcp package. And we can share a lot of tool configs in the same pyproject.toml file.

There is an old topic discussing about the similar needs. I don’t know if I should bring up an old topic so I created a new one.

I think this can also help this topic. If the WebSocket support related package is not installed without the extra, Unicorn won’t need to worry about the user accidentally installing an unmatched webscockets package.

PEP 517 doesn’t really support building more than one wheel from a pyproject.toml, and the wheel format doesn’t really support “optional” packages. So, yes, you need to publish two packages, and ideally you should also use two pyproject.toml files. After all, these are two packages with different dependencies, etc. There will be some duplication but it will be really minimal.

You could try hacking around some way to build two packages from a single pyproject.toml. You could e.g. use config_settings to determine which package is being built, and then users would have to do e.g. python -m build --config-setting=--build-package=gcpmock but that would be really inconvenient and confusing, so please don’t do that.

I don’t think options is a valid pyproject.toml key - where did you get the suggestion to use that?

Also, as @mgorny noted, each wheel should really have its own pyproject.toml.

Sorry, the options indeed have no effect at all. I didn’t notice that it’s for setup.cfg not for pyproject.toml.

Currently, many packages use extras to provide optional functionalities. So I thought it was a natural way to provide optional packages or event sub-packages using extras. Perhaps in the distant future this will be possible? Is this a feasible direction?

I think having a separate pyproject.toml per project/package/wheel is pretty much the standard at this point. Nothing’s impossible, of course, but I don’t think it’s likely.