Optional dependency groups omitting package requirements

[samuelcolvin]

@fungi it might be worth clarifying who you think is missing your point.

In the web interface at least, it’s not very clear. Sorry for the noise.

[pradyunsg (Pradyun Gedam)]

This has been discussed before, to an extent.

Adding a default extra_require environment

Hi everyone, first message to this forum. I hope this will be compliant with the expected format. There has been a number of discussion opened for a long time on this topic: https://github.com/pypa/setuptools/issues/1139 (opened for 3 years). Possible default extras/dependency categories? Somehow related, I opened yesterday an issue on Github with a potential workaround which effectively is not working: https://github.com/pypa/pip/issues/8686 I would like to take this topic as a first is…

So, the idea is to have default extras, where the syntax would be something like:

rich[-markdown]

Assuming the package has markdown as a default extra, the default experience would then the same as what rich[markdown] is today and to get the equivalent of rich today, you’d have to do rich[-markdown].

There’s some details to figure out around this, like what to do if an extra deselects another extra and stuff.

[fungi]

Yes, sorry I was replying to the most recent post prior to mine, on
the assumption that it was in reply to my suggestion (unfortunately
it also lacked any quoting, so perhaps that was not the case).

I’ve interfaced with Discord via its “mailing list mode” ever since
the distutils-sig ML moved here, but it doesn’t correctly handle
normal inline context replies like a real mailing list, expecting
bespoke markdown quote tags. It also lacks proper threading of
replies based on SMTP reference headers, making such discussions
very hard to follow.

[swills1 (Steven)]

@fungi

One of the things I like about how tox handles dep declaration is
that every testenv section can supply its own complete list of
dependencies, but when supplying the list you can also expand
another testenv’s dependencies:

[testenv:foo]
deps =
    xyzzy
    plugh

[testenv:bar]
deps =
    {[testenv:foo]|deps}
    quux

[testenv:baz]
deps =
    {[testenv:foo]|deps}
    gralpy

Something along those lines could give you sufficient flexibility to
both have extras which don’t share any of the same dependencies, but
also extras which build on top of other extras in order to avoid
having to repeat common sets. I guess the main issue to contend with
there is detecting recursion loops (I haven’t looked to see how tox
deals with that).

Maybe I am misunderstanding what you mean, but can’t you already do this with Poetry?

Managing dependencies | master | Documentation | Poetry - Python dependency...

[pradyunsg (Pradyun Gedam)]

You can do this with pip as well, but it requires embedding the package name. For example, a package named package with extras extra1 and extra2 can have:

[project]
name = “package”

[project.optional-dependencies]
extra1 = [“httpx”]
extra2 = [“package[extra1]”, “six”]

[fungi]

Yes, the fundamental difference in my analogy being that tox can be
configured not to install the package you’re testing in some of the
testenvs, while pip has no way I’m aware of for a package to say not
to install itself/its dependencies and instead only install the
things in the extra. I’ll freely admit I don’t know enough about
poetry to have any clue whether it has such a feature.

[pradyunsg (Pradyun Gedam)]

Ah, see

Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml

That they’re exposed to end users, as dependency metadata. You can pip install package[tests] which… is dumb; and occuldes the fact that pip install package[feature] is something conceptually very different. They aren’t a part of metadata provided by build backends; they’re only for the project’s own development workflows. What’s proposed here is equivalent to Poetry’s dependency groups model. FWIW, one outcome from this could very much be that we all agree that it’s fine to use optional-…

Where this was discussed somewhat recently.

[Kludex (Marcelo Trylesinski)]

@samuelcolvin

(I haven’t spoken to the maintainers of uvicorn about this, I’ll reach out to them)

I’m a maintainer of Uvicorn.

I didn’t read all the thread, I’m just going to share my opinion, as asked.

As said in the description, uvicorn has a single extra called standard. The standard installs most of the uvicorn optional dependencies.

@samuelcolvin

It would be much easier to get started with if pip install uvicorn installed everything you needed for development, then pip install uvicorn[prod] installed the things you most likely need for production.

I don’t feel like uvicorn problem lies exactly there. We can already create the prod extra, but our big problem is that we don’t have a way to ensure the version of optional packages without using the extras. To solve this, for us, it would be nice to have a way to, when resolving dependencies, take in consideration the versions set on the extras.

But something happened yesterday on starlette that may be good for this discussion:

starlette has a module called testing, and in there we have a TestClient. The TestClient had a dependency on requests, but on the last release it changed to httpx, but only httpx >= 0.22.0 was supported, versions below that didn’t work. The version of httpx is set on our extras, but it doesn’t matter, most of the people will install only what they need (me included).

Because of that, there were some issues created with descriptions like “module not found ‘httpx’”, and well, not everybody has the same knowledge, and when people install a library, they expect it to work without any effort.

I’m not sure if all this is helpful, maybe I’m just being noisy here, but that’s the pain I have.

[leofang]

I’d like to have this capability too. Say my_package depends on dep1 and dep2. I expect to have a mechanism allowing me to express the following need:

pip install my_package

would install my_package, dep1, dep2, and their dependencies, as usual, while

pip install my_package[no-dep2]  # syntax TBD

would only install my_package, dep1, and dep1’s dependencies; dep2 is removed from install_requires (or whatever build backend’s dependency format).

The motivation behind this is we want my_package to offer full UX, but for certain groups of users, they don’t care the capability offered by either dep1 or dep2, and wish for a slimmer version of my_package being installed.

[NickCrews]

For another example use case, this just came up as a feature request in Splink. I appreciate everyone’s effort.

I’m not sure this has been mentioned yet, but I think a solution would need to be able to hide transitive internal dependencies. By that I mean: say you wanted to expose two extras named csv and parquet. Each of these share some dependencies, so it would be nice to make an iobase extra that includes anyio, gdrive, s3, fsspec, etc. But this IO base extra is internal and we don’t want an end user to be able to install my package[iobase].

The rust/cargo docs may be inspiring to people. They advocate for making composition always additive.

[EpicWink (Laurie O)]

@NickCrews

But this IO base extra is internal and we don’t want an end user to be able to install my package[iobase].

My instinct would be to rename the extra as _iobase to indicate to Python Devs that the is internal by convention. In any case, it’s an effort to discover a project’s extras, so I think simply not documenting the internal extras should be sufficient

[ntessore (Nicolas Tessore)]

I often wish for this too: providing a fully-featured package by default and having a “bare” extra that removes all of the extraneous stuff for installation on a HPC cluster.

Couldn’t the existing extra environment marker serve this purpose very well if its use was allowed as such:

dependencies = [
    "heavy-package; extra != bare"
]

[project.optional-dependencies]
bare = []  # leave empty or put alternatives here