# Where is nested/recursive optional dependencies documented?

**URL:** <https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648>\
**Category:** Packaging\
**Created:** [October 7, 2023, 9:41pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648 "2023-10-07T21:41:03Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![ketozhang](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@ketozhang](https://discuss.python.org/u/ketozhang)\
**Post date:** [October 7, 2023, 9:41pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/1 "2023-10-07T21:41:03Z")

</div>

From [Hyneck’s blog post](https://hynek.me/articles/python-recursive-optional-dependencies/), you can compose an extra/optional dependency from another extra key:

```toml
[project]
name = "my-pkg"

[project.optional-dependencies]
tests = ["pytest"]
docs = ["sphinx"]
dev = [
    "my-pkg[tests,docs]",
    "pdbpp",
]

```

I took a quick look at PEP 685 and PEP 508 and couldn’t find anything that I understood to document this behavior. The post claims it was released with pip 21.2 and I’ve found nothing in their change log nor the current documentation.

---

<div class="post-metadata">

**Author:** ![dlshriver](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dlshriver/32/8912_2.png) [@dlshriver](https://discuss.python.org/u/dlshriver)\
**Post date:** [October 7, 2023, 9:47pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/2 "2023-10-07T21:47:27Z")

</div>

PEP 621 talks about project metadata in pyproject.toml. That PEP also provides a link to the Python packaging docs which are kept up to date with any changes to the specification. The description of optional dependencies is here: [Declaring project metadata — Python Packaging User Guide](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#dependencies-optional-dependencies)

---

<div class="post-metadata">

**Author:** ![ketozhang](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@ketozhang](https://discuss.python.org/u/ketozhang)\
**Post date:** [October 7, 2023, 10:31pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/3 "2023-10-07T22:31:01Z")

</div>

The feature I’m asking for is a lot more specific than that.

This feature allows you to add your own package’s extra as a dependency of another extra, hence **recursive** optional dependency.

```auto
[project]
name = "my-pkg"

[project.optional-dependencies]
...
dev = [
    "my-pkg[tests,docs]"
]

```

---

<div class="post-metadata">

**Author:** ![jeanas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jeanas/32/8115_2.png) [@jeanas](https://discuss.python.org/u/jeanas)\
**Post date:** [October 8, 2023, 1:45pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/4 "2023-10-08T13:45:32Z")

</div>

I don’t really see what’s so special here that needs documentation. It’s not a specific feature, it’s just a consequence of the fact that you can specify any package as a requirement (including your own package), and you can specify package extras in requirements.

---

<div class="post-metadata">

**Author:** ![ketozhang](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@ketozhang](https://discuss.python.org/u/ketozhang)\
**Post date:** [October 8, 2023, 9:49pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/5 "2023-10-08T21:49:38Z")

</div>

Well it’s special because having a dependency that is your own package is ambiguous. A package installer or the generated metadata would have to resolve a few special cases:

- `my-pkg[extra]` : This should disambiguate to the current package if matches `project.name` and don’t look for an existing package on PyPI.
- `my-pkg[extra]==x.y.z`: What about this? Should it now look for a package on PyPI? What if `x.y.z` matches `project.version`?

---

<div class="post-metadata">

**Author:** ![jeanas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jeanas/32/8115_2.png) [@jeanas](https://discuss.python.org/u/jeanas)\
**Post date:** [October 9, 2023, 12:38am UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/6 "2023-10-09T00:38:33Z")

</div>

I don’t think there is any “ambiguity” here since it is impossible in Python to have two versions of the same package installed in the same environment (without conflicts, anyway). Dependency resolving is “just” solving a set of constraints having the form “package A version X depends on package B version Y”. Mathematically, a self-dependency is no different. So, for example, a dependency on `my-package[foo]==1.0` in `my-package` 1.1 is simply an unsolvable constraint and will prevent `my-package` 1.1 from being used, just like `my-package` 1.1 is unusable if it depends on `libA` 1.0 and `libB` 1.0 where `libB` 1.0 depends on `libA` 2.0.

---

<div class="post-metadata">

**Author:** ![ketozhang](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@ketozhang](https://discuss.python.org/u/ketozhang)\
**Post date:** [October 9, 2023, 1:33am UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/7 "2023-10-09T01:33:42Z")

</div>

Fundamentally there’s nothing wrong with it, but if I had to implement a installer or a resolver, then this is a special case I have to be aware of when parsing the package metadata. Clearly some code was necessary to add this behavior in pip (version 21.2 as Hyneck claims). Anyways, we can agree to disagree else we go into this rabbit hole too much.

Okay, so I’m convinced there is no documentation of this specifically. I think it’s needed. I will take this request elsewhere.

---

<div class="post-metadata">

**Author:** ![jeanas](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jeanas/32/8115_2.png) [@jeanas](https://discuss.python.org/u/jeanas)\
**Post date:** [October 9, 2023, 7:10am UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/8 "2023-10-09T07:10:50Z")

</div>

Moving this to #Packaging then.

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [October 9, 2023, 3:55pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/9 "2023-10-09T15:55:02Z")

</div>

> <https://github.com/pypa/pip/issues/11296>
>
> \### What's the problem this feature will solve?
> 
> This (coupled with maybe a new …explicit test) will provide a guarantee to users that the following use case is supported:
> 
> \`\`\`toml
> \[project\]
> name = "pkg"
> 
> \[project.optional-dependencies\]
> all = \["pkg\[a\]", "pkg\[b\]"\]
> a = \["..."\]
> b = \["..."\]
> \`\`\`
> 
> \### Describe the solution you'd like
> 
> Add a note in docs + a new test that explicitly checks this
> 
> \### Alternative Solutions
> 
> N/A
> 
> \### Additional context
> 
> @ichard26 found that this has been supported since 21.2 (tested by @henryiii) https://github.com/pypa/pip/issues/10393#issuecomment-941885429
> 
> \### Code of Conduct
> 
> \- \[X\] I agree to follow the \[PSF Code of Conduct\](https://www.python.org/psf/conduct/).

---

<div class="post-metadata">

**Author:** ![ketozhang](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@ketozhang](https://discuss.python.org/u/ketozhang)\
**Post date:** [October 9, 2023, 7:46pm UTC](https://discuss.python.org/t/where-is-nested-recursive-optional-dependencies-documented/35648/10 "2023-10-09T19:46:54Z")

</div>

Wonderful! I was hoping that you amongst others were far aware of this already.

Should this be documented only in the installer’s docs (e.g., pip and hatch) or should this be documented more explicitly in the core metadata (i.e., how to interpret `Require-Dist` if it includes `Name` as dependency)?
