Optional dependency groups omitting package requirements

[samuelcolvin]

Well, that still doesn’t provide a way to accomplish the following without repeats.

[project]
name = "Package-A"
dependencies = [
    "a>=1",
    "b>=2",
    "c>=3",
    "d>=4",
    "e>=5",
    "foobar>=1",
]

[project.optional-dependencies]
compiled = {"omit" = ["foobar"], "add": ["foobar-compiled>=1"]}

Which is what I was talking about with duplication.

But really, I wouldn’t care which of these options was adopted if one of them was.

This change would be so helpful as a package maintainer.

[fungi]

You implement it in an additive manner rather than subtractive:

[project]
name = "Package-A"
dependencies = [
    "Package-A[base-deps]",
    "foobar>=1",
]

[project.optional-dependencies]
base-deps = [
    "a>=1",
    "b>=2",
    "c>=3",
    "d>=4",
    "e>=5",
]
compiled =[
    "Package-A[base-deps]",
    "foobar-compiled>=1",
]

The problem to overcome there, I think, is some way of not
installing the package’s own default dependencies when installing
extras. I guess a toggle would suffice for that change in resolution
logic, or a separate project.overridden-dependencies list for extras
treated that way (this possibility came up in another context
recently too). Also, per other discussions, the ability to “hide”
that base-deps extra from users might be necessary for some
projects, perhaps by extending the syntax to allow some extras to be
flagged as “abstract” and therefore only usable via references from
other dependency sets within that same package.

[dekoza (Dominik Kozaczko)]

I’d go with something inspired by Arch PKGBUILD’s provides:

[project]
name = "Package-A"
dependencies = [
    "a>=1",
    "foobar>=1",
    "spameggs>=3",
]

[project.optional-dependencies]
compiled = [
    {"foobar": "foobar-binary"},
    {"spameggs": {name: "spameggs-binary", extras: ["somelib-bin"]},
]

[ofek (Ofek Lev)]

@pf_moore

The one I was thinking of (which is distinct, but related) was Adding a default extra_require environment.

This is the way imo, and is what Rust does

[brettcannon]

Somewhat related is Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml if there was a way to remove all default dependencies.

[xen (Mikhail Kashkin)]

I agree with @fungi. You can create two packages with different dependencies. Every package is equal and if this becomes a popular thing it becomes a problem when two packages will “fight” for priorities.

[samuelcolvin]

Two packages is the only solution right now, but it would require some ugly hacks and would be more confusing for end users.

Two packages is an ugly work around for a problem that seems pretty easy to fix elegantly IMHO.

[tiangolo (Sebastián Ramírez)]

I just came to chime in and say that I agree this would be useful to me for FastAPI and Typer at least.

For example, pip install typer should include Rich and Shellingham by default, and if someone explicitly wants to strip that out they would add some additional syntax (e.g. pip install typer[minimal]).

The point is, I would like to be able to declare default extra dependencies that could be removed by advanced users.

[micaelmalta]

Why don’t you use namespacing? core and core-lite with same namespace

On your code you will not need any trick, just

import core

And you install it like that.

dependencies = ["core-lite"]

[tool.pdm.dev-dependencies]
dev = ["core"]

In that case, core will override completely core-lite in dev-mode

[samuelcolvin]

Because I don’t want to install core, and core-lite.

Three reasons this would be a problem:

1. Ideally they would both have the same package name, with this you couldn’t be sure which package would be imported.
2. Let’s say core is 1.5mb and core-lite is 700kb - it would add significant completely unnecessary overhead to install core-lite to never use it. Make those numbers 15mb and 7mb if it helps the argument.
3. This solution doesn’t help at all in the foobar[prod] or foobar[lite] case where you specifically want to remove packages with a dependency group.

[fungi]

I think you’re missing my point. Any set can be constructed through
purely additive rules with no need to implement a subtractive
mechanism. Yes it does imply having some packages or extras which
exist only to install other extras and dependencies, but a purely
additive solution doesn’t require you to end up with two different
versions of the same library installed simultaneously.

This is much easier to reason about if you ignore the existence of
extras for a bit, and just look at doing it purely with separate
packages: install one package if you want the light build, a
different package if you want the full-featured heavy build. The
idea is to be able to replicate that functionality, but with
something like extras under one package name instead of using
separate packages, and having one of those options be the “default”
when no extra is specified. Doing that would require that whatever
extras-like mechanism handles this allows installing fully disjoint
sets of packages, in particular the ability to not necessarily
install things that would be included in the default set.

You can almost do this today with an “empty” package that has no
dependencies by default and puts them all in extras, you just don’t
get the convenience of the user having an importable module by
default, forcing them to specify one of the extras at install time.

[samuelcolvin]

@fungi it might be worth clarifying who you think is missing your point.

In the web interface at least, it’s not very clear. Sorry for the noise.

[pradyunsg (Pradyun Gedam)]

This has been discussed before, to an extent.

Adding a default extra_require environment

Hi everyone, first message to this forum. I hope this will be compliant with the expected format. There has been a number of discussion opened for a long time on this topic: https://github.com/pypa/setuptools/issues/1139 (opened for 3 years). Possible default extras/dependency categories? Somehow related, I opened yesterday an issue on Github with a potential workaround which effectively is not working: https://github.com/pypa/pip/issues/8686 I would like to take this topic as a first is…

So, the idea is to have default extras, where the syntax would be something like:

rich[-markdown]

Assuming the package has markdown as a default extra, the default experience would then the same as what rich[markdown] is today and to get the equivalent of rich today, you’d have to do rich[-markdown].

There’s some details to figure out around this, like what to do if an extra deselects another extra and stuff.

[fungi]

Yes, sorry I was replying to the most recent post prior to mine, on
the assumption that it was in reply to my suggestion (unfortunately
it also lacked any quoting, so perhaps that was not the case).

I’ve interfaced with Discord via its “mailing list mode” ever since
the distutils-sig ML moved here, but it doesn’t correctly handle
normal inline context replies like a real mailing list, expecting
bespoke markdown quote tags. It also lacks proper threading of
replies based on SMTP reference headers, making such discussions
very hard to follow.

[swills1 (Steven)]

@fungi

One of the things I like about how tox handles dep declaration is
that every testenv section can supply its own complete list of
dependencies, but when supplying the list you can also expand
another testenv’s dependencies:

[testenv:foo]
deps =
    xyzzy
    plugh

[testenv:bar]
deps =
    {[testenv:foo]|deps}
    quux

[testenv:baz]
deps =
    {[testenv:foo]|deps}
    gralpy

Something along those lines could give you sufficient flexibility to
both have extras which don’t share any of the same dependencies, but
also extras which build on top of other extras in order to avoid
having to repeat common sets. I guess the main issue to contend with
there is detecting recursion loops (I haven’t looked to see how tox
deals with that).

Maybe I am misunderstanding what you mean, but can’t you already do this with Poetry?

Managing dependencies | master | Documentation | Poetry - Python dependency...

[pradyunsg (Pradyun Gedam)]

You can do this with pip as well, but it requires embedding the package name. For example, a package named package with extras extra1 and extra2 can have:

[project]
name = “package”

[project.optional-dependencies]
extra1 = [“httpx”]
extra2 = [“package[extra1]”, “six”]

[fungi]

Yes, the fundamental difference in my analogy being that tox can be
configured not to install the package you’re testing in some of the
testenvs, while pip has no way I’m aware of for a package to say not
to install itself/its dependencies and instead only install the
things in the extra. I’ll freely admit I don’t know enough about
poetry to have any clue whether it has such a feature.

[pradyunsg (Pradyun Gedam)]

Ah, see

Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml

That they’re exposed to end users, as dependency metadata. You can pip install package[tests] which… is dumb; and occuldes the fact that pip install package[feature] is something conceptually very different. They aren’t a part of metadata provided by build backends; they’re only for the project’s own development workflows. What’s proposed here is equivalent to Poetry’s dependency groups model. FWIW, one outcome from this could very much be that we all agree that it’s fine to use optional-…

Where this was discussed somewhat recently.

[Kludex (Marcelo Trylesinski)]

@samuelcolvin

(I haven’t spoken to the maintainers of uvicorn about this, I’ll reach out to them)

I’m a maintainer of Uvicorn.

I didn’t read all the thread, I’m just going to share my opinion, as asked.

As said in the description, uvicorn has a single extra called standard. The standard installs most of the uvicorn optional dependencies.

@samuelcolvin

It would be much easier to get started with if pip install uvicorn installed everything you needed for development, then pip install uvicorn[prod] installed the things you most likely need for production.

I don’t feel like uvicorn problem lies exactly there. We can already create the prod extra, but our big problem is that we don’t have a way to ensure the version of optional packages without using the extras. To solve this, for us, it would be nice to have a way to, when resolving dependencies, take in consideration the versions set on the extras.

But something happened yesterday on starlette that may be good for this discussion:

starlette has a module called testing, and in there we have a TestClient. The TestClient had a dependency on requests, but on the last release it changed to httpx, but only httpx >= 0.22.0 was supported, versions below that didn’t work. The version of httpx is set on our extras, but it doesn’t matter, most of the people will install only what they need (me included).

Because of that, there were some issues created with descriptions like “module not found ‘httpx’”, and well, not everybody has the same knowledge, and when people install a library, they expect it to work without any effort.

I’m not sure if all this is helpful, maybe I’m just being noisy here, but that’s the pain I have.

[leofang]

I’d like to have this capability too. Say my_package depends on dep1 and dep2. I expect to have a mechanism allowing me to express the following need:

pip install my_package

would install my_package, dep1, dep2, and their dependencies, as usual, while

pip install my_package[no-dep2]  # syntax TBD

would only install my_package, dep1, and dep1’s dependencies; dep2 is removed from install_requires (or whatever build backend’s dependency format).

The motivation behind this is we want my_package to offer full UX, but for certain groups of users, they don’t care the capability offered by either dep1 or dep2, and wish for a slimmer version of my_package being installed.