Thanks to everyone who has been chiming in! I’ve been trying to digest this all for the last few days, and I think there are a lot of interesting points being raised. Thanks also to @sirosen for the useful summary of the current main concerns with the PEP!
I completely agree with @willingc and @pf_moore that we should try and focus on how we can change the PEP to reach a consensus, and I am not personally tied to the present technical solution. I am open minded about switching to a different approach if we can find one. Having said that, we are not rushing this - this is a discussion that started almost five years ago and so there has been a lot of opportunity for another solution to come up, and so far the one presented in the PEP is the one that has had the broadest consensus. Whether it should be accepted or not is another matter, but I just want to make it clear this is not just a rushed idea and that many alternatives haven’t been considered.
There’s a lot of things I’d like to follow up on, but to keep things manageable, I’d like to focus on some of the general concerns that are not specific to extras, because I think we need to make sure we reach consensus on some bigger picture aspects before delving into the detail.
Let’s consider the approach based on the package splitting approach (whereby package-core is a base package with minimal required dependencies, and package is a meta-package that brings in optional additional dependencies). As we’ve discussed here, and is discussed in the PEP, this is already possible although it is not currently a trivial amount of work, which is why a number of people have said here that they have not or will not do it.
Now imagine that we were to now write a PEP to propose a mechanism to make it possible for people to easily declare additional meta-packages in pyproject.toml that should be built and published at the same time as the main project (and would automatically handle version pinning, etc). The details of how this would be achieved in pyproject.toml are not important here, what matters is what if we made it really easy for packages to build metapackages to solve this problem?
This approach would likely meet some of the same criticism as the current PEP, including that packages switching to this would introduce bloat into the ecosystem because the default package name package would now mean the package plus optional dependencies, and if a single package that a user installs has not updated its dependencies to be package-core instead of package, the user will get the full installation (in fact the situation is a bit worse than for PEP 771 here in the sense that since package-core would not exist prior to a specific version, it would be difficult to allow depending on old versions of package or more recent package-core versions). Package developers for existing packages would be changing the semantic meaning of package in the process, similar to the extras approach.
But, of course, the split package approach is actually already possible, just not trivial, and so some of the concerns above already apply to the current ecosystem. The difference between the status quo and the hypothetical PEP mentioned above or PEP 771 is that we want to make it easier - as @sirosen already pointed out above:
So forget about extras - there is a fundamental question here about whether we should make it easier for package to potentially change meaning and for getting a minimal installation to require a little more effort. If this is something we can’t reach a consensus on, then the details of how we achieve this don’t really matter.
As a side note, the status quo is not without cost. Imagine that we simply can’t reach consensus on the above general point. This would mean not providing any easy way for developers to make it so that package means a package that will be appropriate for most users and to have some opt-in mechanism to get a minimal installation. At this point, some projects might decide to make the required dependencies include more than the strict minimum of dependencies (or keep it that way if it is already the case), which would definitely not be optimal since it would increase bloat without the possibility of opting out. Other projects might switch to the split package approach, which as we saw above still shares some of the same concerns as the extras approach. While one might argue that projects that want to do these will already have done it, I think failing to reach consensus here could be a trigger for some packages/projects to use other approaches which may not be as optimal.
I also think that while we are thinking a lot about misuse of this feature and assuming many developers will use it without understanding the consequences, I do think that more widely used packages are in general developed more responsibly/carefully, and I don’t think many of these packages will blindly switch to using default extras (or whatever approach this PEP settles on) without understanding the consequences.