Optional dependency groups omitting package requirements

Packaging
1

It would be great if there was a way to remove/omit default requirements using optional dependency groups.

Currently the docs on optional dependencies doesn’t mention this, so I assume it’s not possible at the moment?

pseudo code pyproject.toml example of what I’m suggestion

[project]
name = "Package-A"
dependencies = [
    "foobar>=1",
]

[project.optional-dependencies]
compiled = {"omit" = ["foobar"], "add": ["foobar-compiled>=1"]}

There are a number of use cases for this:

Switching a dependency

Switching from a default dependency to a substitute package with slightly different characteristics, e.g. compiled/binary, GPU implementation, smaller etc.

The main package at runtime can then do try: import foobar; except ImportError; import foobar_compiled as foobar.

My own use case is to make pydantic smaller, I’d like to compile and upload a pydantic-core-lite package which omits certain dependencies at compile time in order to minimise binary size.

The problem is that since pydantic has (/will have) pydantic-core (with slightly larger binaries) as a default dependency, adding a pydantic[lite] optional dependency doesn’t help - pydantic-core will still be installed.

I’d like to be able to have an optional dependency group with:

[project.optional-dependencies]
lite = {"omit" = ["pydantic-core"], "add": ["pydantic-core-lite"]}

Production Mode

Creating a lite or production group which removes dependencies which are installed by default to aid development.

For example uvicorn has a standard group which installs some “cython-based dependencies” and other “optional extras” to aid in development.

It would be much easier to get started with if pip install uvicorn installed everything you needed for development, then pip install uvicorn[prod] installed the things you most likely need for production.

(I haven’t spoken to the maintainers of uvicorn about this, I’ll reach out to them)

[none] group

You could also imagine a pip install the-package[none] mode where no dependencies are installed and users can decide exactly what they want to install and how manually.

To be clear: packages listed under omit wouldn’t be uninstalled, and having them installed (e.g. manually, or as a dependency of another package) wouldn’t be an error, it would just tell pip not to bother installing that package as a dependency of the package in question.

I think adding this feature would be a real win for python packing, pip and the whole community.

I also imagine it could be achieved with minimal compatibility issues, since the only change visible to users and installers is a change in the schema for the optional-dependencies bit of pyprojec.toml/setup.cfg/setup.py.

3 Likes
2

Not ideal, but I’ve done this in the past with “dummy” packages that
are mainly just lists of dependencies. You can upload a new
foo-minimal package which is identical to foo but without additional
requirements, then replace the old foo package with one which
depends on foo-minimal plus the original requirements.

I agree, though, it would in theory be possible to accomplish this
with a single package if the dependency declaration syntax were
sufficiently expressive.

2 Likes
3

There have been variations of this request made before, so it’s certainly something a number of people are interested in. So far, though, no-one has been sufficiently interested to take this to the level of creating the necessary PEP to propose a standard for this. (It’s not something pip would implement without a standard defining the feature, to be clear).

1 Like
4

Thanks for the reply.

Yes, I assumed someone had asked for this before. But in a brief search I couldn’t find anything. Any idea where previous discussions took place?

5

The one I was thinking of (which is distinct, but related) was Adding a default extra_require environment. Apart from that, I recall there being general comments about people wanting to push extras to do more than they currently can, but nothing specific.

I think the key thing here is that people wanting more from extras should get together and come up with a unified proposal. That stands more chance of making progress than individual requests for features with more limited applicability.

6

Thanks, I guess having thought about this a lot, and wanted more from extras for a bit; I think what I’m suggesting here would have a large applicability.

I’m sure there are other requests, but I think this would be a big enough easy win to be a PEP on it’s own.

7

Personally, I find the proposal a bit over-complex. Having “omit” be package names whereas “add” is requirements is odd, IMO, and while I’m sure it would be well-defined, I’d find it unintuitive to work out whether foobar should be installed if I ask for project[compiled,base] when we have

[project.optional-dependencies]
compiled = {"omit": ["foobar"], "add": ["foobar-compiled>=1"]}
base = ["foobar > 1.0"]

I assume, by the way, that extra = {"add": "something"} would be the same as extra = ["something"]. Having to ways to say exactly the same thing is unfortunate, but I guess necessary because the existing format isn’t extensible in the direction you’re after…

But I’m a very infrequent user of extras, so take my concerns with that in mind…

8

Ye, I guess the other option is to just define what dependencies should be installed with a given option. So the usage would be:

[project.optional-dependencies]
compiled = {"override": ["foobar-compiled>=1"]}

Where override was the full list of dependencies to install with this option checked.

This would simplified project[compiled,base] since it would clearly mean installing both sets of dependencies, as it does now.

I think this would be simpler and would cover all the same use cases, though it might required some duplication.

The other change would be to add an all new key instead of extending optional-dependencies, so we could have the following.

[project.override-dependencies]
compiled = ["foobar-compiled>=1"]
9

One of the things I like about how tox handles dep declaration is
that every testenv section can supply its own complete list of
dependencies, but when supplying the list you can also expand
another testenv’s dependencies:

[testenv:foo]
deps =
    xyzzy
    plugh

[testenv:bar]
deps =
    {[testenv:foo]|deps}
    quux

[testenv:baz]
deps =
    {[testenv:foo]|deps}
    gralpy

Something along those lines could give you sufficient flexibility to
both have extras which don’t share any of the same dependencies, but
also extras which build on top of other extras in order to avoid
having to repeat common sets. I guess the main issue to contend with
there is detecting recursion loops (I haven’t looked to see how tox
deals with that).

10

Well, that still doesn’t provide a way to accomplish the following without repeats.

[project]
name = "Package-A"
dependencies = [
    "a>=1",
    "b>=2",
    "c>=3",
    "d>=4",
    "e>=5",
    "foobar>=1",
]

[project.optional-dependencies]
compiled = {"omit" = ["foobar"], "add": ["foobar-compiled>=1"]}

Which is what I was talking about with duplication.

But really, I wouldn’t care which of these options was adopted if one of them was.

This change would be so helpful as a package maintainer.

1 Like
11

You implement it in an additive manner rather than subtractive:

[project]
name = "Package-A"
dependencies = [
    "Package-A[base-deps]",
    "foobar>=1",
]

[project.optional-dependencies]
base-deps = [
    "a>=1",
    "b>=2",
    "c>=3",
    "d>=4",
    "e>=5",
]
compiled =[
    "Package-A[base-deps]",
    "foobar-compiled>=1",
]

The problem to overcome there, I think, is some way of not
installing the package’s own default dependencies when installing
extras. I guess a toggle would suffice for that change in resolution
logic, or a separate project.overridden-dependencies list for extras
treated that way (this possibility came up in another context
recently too). Also, per other discussions, the ability to “hide”
that base-deps extra from users might be necessary for some
projects, perhaps by extending the syntax to allow some extras to be
flagged as “abstract” and therefore only usable via references from
other dependency sets within that same package.

1 Like
12

I’d go with something inspired by Arch PKGBUILD’s provides:

[project]
name = "Package-A"
dependencies = [
    "a>=1",
    "foobar>=1",
    "spameggs>=3",
]

[project.optional-dependencies]
compiled = [
    {"foobar": "foobar-binary"},
    {"spameggs": {name: "spameggs-binary", extras: ["somelib-bin"]},
]
13

This is the way imo, and is what Rust does

14

Somewhat related is Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml if there was a way to remove all default dependencies.

1 Like
15

I agree with @fungi. You can create two packages with different dependencies. Every package is equal and if this becomes a popular thing it becomes a problem when two packages will “fight” for priorities.

16

Two packages is the only solution right now, but it would require some ugly hacks and would be more confusing for end users.

Two packages is an ugly work around for a problem that seems pretty easy to fix elegantly IMHO.

17

I just came to chime in and say that I agree this would be useful to me for FastAPI and Typer at least.

For example, pip install typer should include Rich and Shellingham by default, and if someone explicitly wants to strip that out they would add some additional syntax (e.g. pip install typer[minimal]).

The point is, I would like to be able to declare default extra dependencies that could be removed by advanced users.

3 Likes
18

Why don’t you use namespacing? core and core-lite with same namespace

On your code you will not need any trick, just

import core

And you install it like that.

dependencies = ["core-lite"]

[tool.pdm.dev-dependencies]
dev = ["core"]

In that case, core will override completely core-lite in dev-mode

19

Because I don’t want to install core, and core-lite.

Three reasons this would be a problem:

  1. Ideally they would both have the same package name, with this you couldn’t be sure which package would be imported.
  2. Let’s say core is 1.5mb and core-lite is 700kb - it would add significant completely unnecessary overhead to install core-lite to never use it. Make those numbers 15mb and 7mb if it helps the argument.
  3. This solution doesn’t help at all in the foobar[prod] or foobar[lite] case where you specifically want to remove packages with a dependency group.
20

I think you’re missing my point. Any set can be constructed through
purely additive rules with no need to implement a subtractive
mechanism. Yes it does imply having some packages or extras which
exist only to install other extras and dependencies, but a purely
additive solution doesn’t require you to end up with two different
versions of the same library installed simultaneously.

This is much easier to reason about if you ignore the existence of
extras for a bit, and just look at doing it purely with separate
packages: install one package if you want the light build, a
different package if you want the full-featured heavy build. The
idea is to be able to replicate that functionality, but with
something like extras under one package name instead of using
separate packages, and having one of those options be the “default”
when no extra is specified. Doing that would require that whatever
extras-like mechanism handles this allows installing fully disjoint
sets of packages, in particular the ability to not necessarily
install things that would be included in the default set.

You can almost do this today with an “empty” package that has no
dependencies by default and puts them all in extras, you just don’t
get the convenience of the user having an importable module by
default, forcing them to specify one of the extras at install time.