I haven’t actually run the builds to figure out the nuances/gaps (in flit, in this case), but something like this: Comparing pallets:main...zooba:demo · pallets/click · GitHub
No doubt there are specifications we’ve written that prevent this simple structure (I bet we disallowed “…/” in filenames in pyproject.toml…), but that’s the kind of tooling issue that we ought to just fix, rather than letting it force us into designing massively complicated user-facing features to work around it.
There is also flot which is a fork of flit designed to package different files into different distributions. I realise that is not what you are doing here but it would be more useful if there was just generally tooling for splitting up packages int the VCS → PyPI step and tooling that could understand a VCS checkout that was intended to work that way.
That’s fair enough–I don’t know when either was introduced, as I think they both predate a lot of documentation being written[1].
But another point of evidence is that both of them are currently in use, even in combination within the same project. So I don’t think they are complete replacements for one another.
I think they’re both mentioned in the earliest entry for cargo’s changelog ↩︎
I might have missed it in the various discussions, but isn’t there also this circular-dependency approach to default extras?[1]
# main package
[project]
name = "foo"
version = "1.1"
dependencies = [
"foo-default-extras",
]
[project.optional-dependencies]
no-default-extras = ["foo-default-extras==0"]
extra-a = [
]
extra-b = [
]
extra-c = [
]
# default extras on
[project]
name = "foo-default-extras"
version = "1"
dependencies = [
"foo[extra-a, extra-b, extra-c]",
]
# default extras off
[project]
name = "foo-default-extras"
version = "0"
dependencies = []
Although it is actually better suited to switching individual extras on and off. ↩︎
I’m not sure I understand the specification of how this is supposed to work…
My package foo depends on foo-default-extras, without version.
By default, the highest version of the foo-default-extras package gets picked, which in turn depends on the extras in foo that should be installed by default.
The extra foo[no-default-extras] depends on a pinned version 0 of foo-default-extras, which doesn’t depend on anything, so no default extras are picked.
I tried to resurrect a toy index server I used to check this actually works.
With default extras:
$ pip install --dry-run --index-url=https://ntessore.github.io/py-default-extras/simple 'foo'
Would install dep-a-1.0 dep-b-1.0 dep-c-1.0 foo-1.1 foo-default-extras-1.1
Without default extras:
$ pip install --dry-run --index-url=https://ntessore.github.io/py-default-extras/simple 'foo[no-default-extras]'
Would install foo-1.1 foo-default-extras-0
One caveat is that the foo-default-extras dependency is already installed on the next install of foo. This can make versioning trickier when defaults are evolving, even if it can be solved with careful version ranges. So, instead of a full set of default extras, I am actually using this to switch off a single extra (cli, for command-line use of my library) with foo-extra-cli/foo-extra-cli==0.
Aha, I see! That’s a clever way to trick the resolver into doing this, but it does seem like it runs into problems as soon as you make things more complicated and add multiple versions, etc.
The case of one extra is the easiest to transform into a separate package, e.g. foo-cli could just depend on foo.
I am so happy to see this! This has been a pain point for Polars because of CPU feature sets, meaning that good practice (to avoid installing a wheel that people on older CPUs could not run) was to have optional extras ([cpu] with polars and [lts-cpu] with polars-lts-cpu).
Since that then results in packages that don’t by default install the polars dependency, it is common for developers (of packages using Polars, or Polars plugins) to just make the polars dependency a regular (not optional) dependency, thus the LTS CPU users need to uninstall, or might end up accidentally installing both if unaware of precisely what the behaviour is.
default-optional-dependency was exactly what was floated in the discussion around this mid-2024 (link). This would solve the problem upstream, as we hoped would soon happen 
In Rust, features (our extras) and workspaces are entirely orthogonal concepts with quite necessarily different configuration.
I want to second James’s point on the Rust/Cargo reality on the ground. I don’t think this is off-topic, the same thing came to my mind as directly analogous. The proliferation of default-features = false throughout dependencies is indeed common, as packages try to minimise unnecessary feature bloat (as Armin’s blog post linked above mentions).
However, in practice, I also strongly support the original proposal of this PEP (default extras supplied by default from pip installing the package by name) from the perspective of software “just working”, Python having a “batteries included” philosophy. In the case with Polars I mentioned above and plugins/packages depending on it, users will expect installation to pull in sensible default dependencies without additional complexity. It is essentially no better than the current scenario (having to tell everyone to pick an extra) if the default optional extras are not supplied by default.
The average end-user running pip install my-polars-pkg would expect to get Polars, and the “installation story” is significantly less intuitive otherwise. Essentially the majority of happy path installations have to do the trickier thing that would ideally only be for the edge cases (the LTS CPU users using an extra). Hence I’d opt for the authors’ proposed syntax.
i.e. we are here:
pip install my-polars-pkg[cpu] ⇒ installs polarspip install my-polars-pkg[lts-cpu] ⇒ installs polars-lts-cpuand would continue to be here without the “package[] ⇒ no extras” syntax.
Want to throw in my support from the PyTorch end here.
We’ve been exploring using “extras” to denote which hardware accelerator we’d like to give users in the form of:
pip install torch[cpu]
pip install torch[cuda]
pip install torch[rocm]
pip install torch[my_fav_accel]
# with default extras this should ideally default to torch[cpu]
pip install torch
We feel like for python packages built for accelerators (like cuda, rocm, etc.) default extras should give users the flexibility to identify which hardware accelerated version of the package they’d like to consume while also allowing for the base level user to not have to care about it at all.
I agree – I think that should be encouraged
And I’d love to see [minimal] be a default feature, rather than having to be defined by each package author –
It drives me crazy when it’s hard to get just the package itself If I have to hand-install another package in some cases, that’s just fine. e.g. for the “database back-end” example, is it so hard to tell your users:
install this_nifty_package
and then install the database back-end of your choice.
(and of course, in an application’s requirements, you’d simply put the DB in the requirements)
This may be what (sorry I can’t remember who posted it) was said a bit back in the thread – the difference between a library and an application – and aside from using the package system for applications, I think some libraries are both libraries and applications. But package authors what the application to be easy to use, so the “extras” are generally included.
NOTE: this is really bad in the conda-forge world – I often get a LOT of extra kruft when I install a lib – folks include Jupyter, matplotlib, etc – they are needed for the examples, and they expect that users will probably want those things anyway – NO, I DO NOT, thank you – not for my web service!
Is that really a decision that should be made less visible to the user? With the current model, you can detect a naive pip install torch at runtime based on the absence of any accelerator and remind them of the choice. With the proposed model, a pip install torch user with a GPU will have no idea why or possibly even if their fancy GPU is doing nothing more than cluttering their desk.
Actually currently pip install torch installs a CUDA 12.4 version of PyTorch and to install the CPU version of PyTorch you would have to use pip install --index-url https://download.pytorch.org/whl/cpu torch
There’s actually no way to automatically detect hardware to give people the correct thing they might need.
Btw my recommendation that cpu is the default is just an implementation detail and could change. More than likely we would make cuda the default since that’s what users are expecting.
To provide more context to the above
Check the latest release of PyTorch: https://files.pythonhosted.org/packages/24/85/ead1349fc30fe5a32cadd947c91bda4a62fbfd7f8c34ee61f6398d38fb48/torch-2.6.0-cp313-cp313-manylinux1_x86_64.whl.metadata
[...]
Requires-Dist: nvidia-cuda-nvrtc-cu12 (==12.4.127) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-runtime-cu12 (==12.4.127) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-cupti-cu12 (==12.4.127) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cudnn-cu12 (==9.1.0.70) ; platform_system == "Linux" and platform_machine == "x86_64"
[...]
And you have no way to deactivate this behavior today. And to be fair that makes a lot sense to have CUDA the default backend (in this case) given that’s what most users actually want.
A similar situation could be stated with BLAS (OpenBLAS default ?) or MPI (OpenMPI default?) implementation in different packages.
We have pretty much determined with @trobitaille that pip install package[] is valid syntax and we will make it work. We will edit the PEP in this direction. That makes a lot of sense to me.
To support what Eli mentions here. We are working at a complete proposal Wheel Variant Support - WheelNext to “auto detect hardware dependencies” at install time. This specific PEP does not “pretend to fix that” - though as Eli highlights it can be used as a rather nice stop-gap solution to wait for the complete fix being formalized inside the Wheel Variant PEP (please reach out if you’re interested !)
This is really the core of this PEP - how do you provide a “good default to 85%” of users while allowing those who want to “opt-out”. I think you summarize perfectly the frustration leading to this work.
To add further support,
I develop an SDK library for interfacing with the Encord (my employer) platform at: GitHub - encord-team/encord-agents: Build and deploy AI agents for automated labeling, quality assurance, and workflow automation in the Encord ecosystem.. The premise is that to provide supporting code for users to write Agents / webhooks that interface with the platform and a common use case is manipulating / running models on Visual data. As such, we want to have OpenCV as a dependency to allow for batteries included methods for users to use their models easily with our platform.
We’ve had a request to support opencv-headless or no opencv build which makes perfect sense. However as opencv and opencv-headless are mutually incompatible, this falls into the ‘multiple backends, we want to provide a default and also allow no backend option’. Ideally we would have the full opencv as a default dependency allowing users to run their models and follow the demos easily whilst supporting headless and no Visual use-cases.
Instead opting for the empty default means that we are making a breaking change to remove this dependency and require our users to make a change.
This PEP or similar behaviour would exactly match onto our situation.
The default-optional-dependency-keys is such a mouthful. Why not the same name as default-extras or default-optionals/recommends/recommended-extras/…
I’m having a very similar question: In uv, we want to support selecting which extras to install by default in project mode (that means uv sync, not uv pip install, it only affects the current project or workspace): Support `default-extras` config in the project interface. by blueraft · Pull Request #12965 · astral-sh/uv · GitHub. This supports will initially live in [tool.uv], but when trying to align with PEP to make a transition later easier, we found that default-optional-dependency-keys and especially --default-optional-dependency-keys is rather unwieldy, while default-extras would fit nicely with the rest of the configuration.
I think the problem with default-extras is that it’ll add inconsistency in the project table’s mechanisms for referencing dependencies.
I would have preferred default-extras ideally but unfortunately @pradyunsg is correct that e.g. in pyproject.toml, there is no reference to the terminology of ‘extras’ currently, hence the current sub-optimal suggestion of default-optional-dependency-keys – however I can easily be convinced of a better alternative!