Woah that’s a great way to articulate it. I fully agree.
So is “concentrated” to you having a single tool to do this, or coming up with a single way to specify development dependencies that all tools use?
Definitely not that link as that’s the spec, not a tutorial or guide. I think you’re looking for Packaging Python Projects - Python Packaging User Guide .
1 Like
Yes, That looks great! I didn’t realize this page changed so much. It used to be a land of outdated information.
Neither, I’m on the camp that the pyproject.toml status quo for dev dependencies is fine — mixing them with project.optional-dependencies is fine. “Concentrated”, to me, is for Python to add one example of the development workflow of a python package project.
Preferably, the example is tool-agnostic. Here’s one for pyproject.toml file,
[project]
name = "myproject"
version = "0.0.1"
requires-python = ">=3.7"
[project.optional-dependencies]
tests = ["pytest", "rich"]
, and then maybe a few example of installing for development (e.g., pip install -e .[tests], pdm install -G tests).
No reason to make this specific to packaging projects either. Applications need development dependencies too (I know lockfile specification is missing, but let’s not go there 
).
We don’t have a tool-agnostic way to do what I’m trying to achieve – that’s basically why I created this thread.
Anyway, this is digressing strongly into “what are the recommendations for Python development workflows” and “we need better docs” – so I’m gonna step away from this to avoid going into that rabbit hole.
Fair enough! 
3 Likes
I’m wondering if there’s another way. Since the original intention is to have a dependency group that does not show up as extras, maybe we should instead provide a way to declare an optional dependency group as (air-quotes) non-public?
Say
[project.optional-dependencies]
auth = ["cryptography"]
tests = ["pytest"]
[project]
extras = ["auth"]
If the extras key is missing, the default is to populate the value with all keys in optional-dependencies. If it’s explicitly given, it lists the extras that’d go into metadata.
11 Likes
I personally like that solution a lot. It makes it opt-in while also making it explicit once you have opted in. Plus it’s a good mapping to Provides-Extra.
1 Like
That’s a fantastic, pragmatic proposal. What’s the next step, a PEP to update PEP 621?
I like it too. A PEP to update the project table to add this additional key, yes.
At the risk of being a cookie licker on this… I’ll pick this up.
update: I am not writing this PEP – Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml - #58 by pradyunsg
1 Like
At first sight I like it. But if these optional dependencies don’t end up in metadata, how should an installer deal with them?
I’d expect those optional-dependencies entries not listed in extras to be completely ignored by the wheel builder. The wheel installer would never see those and thus doesn’t need any additional logic to deal with them.
So how would something like pip install foo[not-an-extra] work? Would pip need to unpack the sdist and parse the data out of pyproject.toml? What if there was a wheel for foo? Would not-an-extra be rejected or would we be expected to check the sdist just in case?
The idea seems nice in principle, but there are a lot of questions with no obvious answer…
1 Like
I don’t think it makes sense to co-opt dependency specifiers for installer-only extras. You’d probably want to have a new option for these in pip or overload -r.
Since not-an-extra is not an extra, the command would not “work”. The behaviour currently implemented by pip is (IIRC) to install foo without any extras.
I thought the point of the dev-dependencies idea was that pip would be able to install them? If that’s not the case, and we simply say that pip will ignore anything in project.optional-dependencies that’s not carried forward into the wheel metadata, then I’m OK with that. Although I suspect we’ll get requests to add support, and we’ll go through the same debate then - only with “but it’s a standard” as an argument that we should support it.
Let me go on record then that I’m fine with the proposal to use project.extras for this, as long as it’s intended only for build tools and there’s no expectation or implication that pip will support optional dependencies that aren’t listed in extras (either explicitly or implicitly by the omission of the extras key).
If there’s an expectation that pip will support this, my questions stand. And in particular, I don’t think we can behave differently depending on “whether a wheel exists”.
1 Like
The idea is for pip to install dependencies directly from project.optional-dependencies as an alternative to requirements.txts. pip shouldn’t have to care about the hypothetical project.extras - that’s for the backends’ benefit only. It tells them which extras, i.e. which dependency groups, to omit from the distribution metadata. Distribution-less dependency groups are not extras and you would not expect to be able to install them using install x[y]. You’d need some other way to invoke install, e.g. with pip install -d dev, where dev might be:
[project.optional-dependencies]
dev = ["black"]
I’m personally not a fan of re-using optional-dependencies for this - it’s the sort of thing that’s alluring for its apparent elegance but semantically ambiguous.
1 Like
Hmm, yes. If that’s the intention, then I agree, it’s ambiguous and confusing (it certainly confused me!)
1 Like
I ended up down the same line of reasoning as @pf_moore above, and I don’t see a way to make a project.extras key work with the tooling model that we have today.
Dropping into this topic from elsewhere, I wonder if dependency groups and environments really need to be tightly connected. Like, if I can just specify dependency groups easily, then if I were to say use pdm+tox, I could just install the right group into the right environment with pdm install -G mygroup and if I was using hatch, I would just say [tools.hatch.envs.<blah>]features = ['mygroup'].
I don’t have the heart nor time to go back into this topic but the only proposal that I’ve heard that I am in favor of is the one from @uranusjr Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml - #47 by ofek
1 Like