In general, opt-in approaches are definitely a possibility and I agree that they would be safer/less confusing, although I think that they could end up being less discoverable by users. If we can’t reach consensus on an opt-out approach such as the current PEP 771, then having some standardization around recommended extras would still be an improvement.
Some thoughts related to this:
Not to bikeshed at this point, but I think [recommended] would be a good name for such a standardized extra that is suitable for the average user, and it’s already used by some projects to be precisely the extra which is recommended by default. I doubt many packages use it to not mean things that are recommended (never say never though I guess!)
If we did standardize on this, perhaps the pip install command on PyPI could show [recommended] for packages that define it
Some package installers could potentially choose to always include the recommended extra by default, if they were tools that were primarily geared towards novice users. So in a sense we would be back to it being opt-out on those tools. I think that would be ok though depending on the target audience of the tool.
Some package installers could choose to include a flag/config option to always install recommended extras
Other package installers could potentially show informational messages to help discoverability if they wanted, for example:
$ pip install astropy
Collecting astropy
Using cached astropy-7.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.1 MB)
NOTE: to install astropy with recommended optional dependencies, you can use astropy[recommended]
...
(to be clear this would be up to the pip maintainers, I’m just brainstorming potential ideas/implications here).
Overall I think this approach is definitely feasible. It would require I think more effort to make sure that the users we are trying to help here actually see a benefit. It is a reasonable option if we don’t reach consensus on PEP 771 in its current form though.
I’m not sure an opt-in approach really solves the user side of the problem. You’ll still have people typing pip install foo without an extra and end up getting an incomplete environment in those cases where a default is required.
But if we’re bikeshedding this approach anyway if you do like the standardized opt-in approach, we have to carefully consider the namespace collision problem. Any convention for an opt-in extra name could be used for other purposes already, so how would you handle that? Or we’d need to pick something that can’t possibly collide with existing extra names.
Off the top of my head so haven’t fully considered what the implications would be (and maybe this has already been proposed and I just don’t remember seeing it), a potential way to support a standardized opt-in to default extras could be to add a new term in the dependency specifier grammar that would mean “also include all of the default extras”
Somewhat arbitrarily choosing an optional “+” between the name and extras, that would essentially mean that if a project “foo” has extras “bar”, “baz”, and “qux”, it can define that “bar” and “baz” are the default extras so a dependency specifier foo+ is identical to foo[bar,baz]
That could also mean that it’s safe to add to a dependency specifier for any project, even if it doesn’t define extras, since “all of the default extras” in that case is []
For the same reasons as we couldn’t use a special extra name to mean ‘default extras’ in PEP 771, we wouldn’t be able to use a special name for the standardized recommended extra that isn’t already a valid extra name. See this section of PEP 771 for more information.
If you introduce new syntax that is not valid as per PEP 508 (e.g. package+), you break compatibility with any pre-existing version of package installers, which is definitely not an option (especially if say package A decides to depend on B+, then A becomes un-installable on a whole set of package installer versions)
This was the same reason we couldn’t use a special extra name in PEP 771 as-is to indicate default extras, and why it had to be done otherwise via the package metadata. However in the specific case of opt-in recommended extras, recommended is probably one of the safest user-friendly names.
In any case, in its simplest form, @pf_moore’s proposal anyway pretty much leaves it all up to package installers and other interfaces to decide what to do if a [recommended] extra is present. The only potential things a PEP with this approach would do is:
Suggest using e.g. recommended or another name as meaning things that the average user will need
Potentially requiring package installers to not complain if a user requests [recommended] and it is not present
and the rest would be left up to different package installers and platforms. It almost doesn’t really matter if some packages already use [recommended] - it seems very unlikely that any package would use [recommended] to indicate extras that are going to be problematic if installed.
At the end of the day, I personally think the opt-in approach would not be much of an improvement over the status quo - it would probably be missed by many users, and a lot of users don’t even know about the existence of extras.
Rereading through the previous discussion, it’s not quite the same proposal, but this does run into the same problem as the Syntax for deselecting extras Rejected Idea - namely this half of its rejection reasons
More critically, this would introduce new syntax into dependency specification, which means that if any package defined a dependency using the new syntax, it and any other package depending on it would no longer be installable by existing packaging tools, so this would be a major backward compatibility break.
Personally, I wouldn’t mind that backward compatibility break, because I myself am pretty diligent about updating to the latest versions of tools (mostly thanks to pip’s warning message whenever there’s a new version and the availability of pipx upgrade-all and uv tool upgrade --all), but it’s reasonable to say that modifying the dependency specifier grammar is too big a change to tolerate
edit: I’m also just now realizing that I just repeated the answer you gave me Thomas, sorry about that! (for whatever reason I read your reply as an answer to someone else)
Please be very careful about this line of reasoning. To say it’s not an option is overly strong, to the point of being inaccurate. It’s an option with downsides – maybe major ones which you reject as the PEP author, as is your prerogative – but it is a possible solution.
I make a point of this because if it is the right solution, it’s worth waiting a couple of years for support to arrive in the various installers.
For what it’s worth, I am not a huge fan of pip install "astropy+" as a future state of the world. I think it adds more syntax in the wrong place (a new user’s first interaction with Python) and opens up questions like “what should PyPI show as an install command?”
But if the best solution turns out to involve new dependency specifier syntax, let’s make sure not to write it off.
Is picking a default for all users really the right option for cases like this though? This would still break if someone did [] for their extras, expecting extras to be well… extras, not required.
Wouldn’t cases like selecting some backend be better served by a mechanism for “requires one of”, and then in the case of no extra being selected and one of those backends not already being in the environment, prompting the user to select one of them?
Yes sorry I shouldn’t have been so drastic in my choice of language - what I meant was that the original version of PEP 771 also introduced a proposed syntax change/addition, and that this got strong pushback because of the high cost, so I suspect that a proposal that would be opt-in and therefore would be of less benefit to the average user but with a similar high cost would be difficult to get consensus for. But of course, we should consider all options.
Personally I think that new syntax for a new feature is more reasonable than changing the meaning of existing syntax and adding new syntax for the previous meaning.
If nothing else, it makes having a way to express the old meaning in a backward compatible manner much harder.
Just to put it on the table, if we are open to new syntax, and don’t want to change the meaning of package, but want an opt-out approach, then an alternative would be to define the ability to define e.g. minimal_dependencies in the project metadata and specify a new syntax, e.g. package-, which would indicate that minimal dependencies should be used. As an example:
installing package- would install package with the minimal dependencies. This has the benefit of not having anything to do with extras, and might force maintainers to think a bit more about it. One could even require minimal_dependencies to be a strict subset of dependencies to avoid any issues if both package- and package are in a dependency tree.
This seems like it’s relying on a technicality to claim that it’s not changing the meanining of package. Because you’re changing the meaning of dependencies from “the set of dependencies needed to run the package at all” to “the recommended set of dependencies”.
This whole proposal, in its opt-out form, is based on the idea that “we” want pip install package to install the recommended set of dependencies for a standard install of package, instead of the current meaning of “install the package, along with any dependencies necessary to make the package work correctly”. But I don’t actually think there’s any consensus that this new meaning actually is what people want.
The evidence given is that there’s a number of packages for which the existing meaning isn’t ideal. That’s entirely fair. But it ignores all of those packages where the current meaning is correct - dismissing them as being able to work just as well regardless of the definition. But if that were actually the case, we wouldn’t need any sort of “opt out” for the new behaviour, we could just change the behaviour and everyone would be happy - which clearly isn’t the case.
I think (and I’ve tried to make this point before, but maybe not clearly enough) we’d be better served by taking a step back and looking at what problems the packages that aren’t served well by the current semantics are facing. I can see two, at the moment:
Users see lots of documentation and advice saying that they should run pip install package. While that’s true for the majority of packages, where a package needs additional optional dependencies to provide a good user experience, there’s no way to express that in a way that’s discoverable to new users. The current workaround of using a named extra isn’t sufficiently discoverable, as users don’t read (or follow) documentation.
Packages where some of their dependencies take the form of “one of A, B or C is required, but it doesn’t matter which one”[1]. There’s currently no way of expressing this, except for either making one option mandatory (requiring the user to manually fix the installation after the fact) or making each of A, B and C an extra, and insisting that the user pick one (resulting in a bare pip install package giving the user an installation that doesn’t work).
I think default extras “solve” (1), at the cost of changing the semantics of a default install in a way that I don’t believe we have consensus for, while doing nothing for (2), except possibly in the edge case where there’s only one such “alternative” dependency, and one obvious default choice.
Conversely, I think that default extras leave us in a situation where it’s harder to find a long-term solution for (2), because we’re adding complexity (either opt-in or opt-out syntax) which we’ll then have to deal with in any future changes - as well as encouraging projects in situation (2) to try to use default extras to partially solve their issues just because it’s available, resulting in an even more complex transition when we do solve (2) properly.
This can be further complicated when the user could develop their own replacement for A, B, and C, and manually install that to satisfy the dependency even though the metadata isn’t aware of the user’s code. ↩︎
I considered starting a poll on this, but when I did, I found it hard to formulate in a way that would allow me to express my own preferences. So I’ll just state my preferences explicitly here, and I encourage others to do the same.
First of all, I have very different feelings about a package name explicitly requested by the user in an install command like pip install pkg, compared to a package name in dependency metadata. In dependency metadata, I have a very strong preference for the package name meaning just the package and its required dependencies. I feel that anything else both contradicts existing practice, and risks unexpected breakages.
When the user gives a package name as part of an install command, my feelings are a lot less clear cut. I can see the value in having this give a “ready to use” environment, and I sympathise with projects that find that using an extra that the user has to specify gives a bad UX. But I’ve never had to work with such a project myself, so I don’t know from personal experience what it’s like. I tend to think of pip install pkg as meaning “just install pkg and whatever is needed to make it work”, though.
What I do find extremely uncomfortable is having different semantics for the two cases. Maybe it was a mistake linking “what people request on the command line” and “how we specify dependencies in metadata” but I don’t think that’s something we can change without a significant amount of effort (think something like the wheel-next project). So I think we need to reconcile the different pressures with a single decision - and for me, the need to keep metadata meaning “the project plus its essential dependencies” far outweighs my mild sympathy for the desire to have pip install pkg install a recommended set of extras.
I’m in almost full agreement with Paul on this. I’m uncomfortable with changing what pip install foopkg means.
That said, there is a purely perceptual question about whether this changes the meaning of the package name or adds a new idea, currently spelled foopkg[], which is the “more minimal install of the package”. The phrasing of the proposal, as “default extras”, discourages but does not preclude that reading.
Keeping foopkg consistent in meaning between the CLI (pip) and dependency metadata is a table-stakes, non negotiable item for me.
In my thinking about this proposal, I’m starting to separate the idea of a recommended installation from currently available kinds of metadata.
A “default extra” is just one way to encode the idea of a “recommended installation”. And we want good rules that can lead to people getting the recommended installation when they want it, but not when they don’t.
I don’t think there is any reliable heuristic which can determine whether a request to install a package is or is not expected to include recommended packages. Set aside questions of syntax. If I tell you “you need to install astropy”, without specifying, can you know whether or not I meant that you need it with the extra?
Therefore, the problem the spec seeks to address is a discrepancy between “what the user said” (pip install foopkg) and “what the user meant” (pip install foopkg[all]).
Introducing a new mechanism may actually make things more ambiguous, not less so. I’m uncertain as to the correct course.
Perhaps there’s some interesting ideas down the path of “what if we wanted pipx to install something different from pip?” or “what if we wanted to distribute .pyz as well as .whl files?” I’m not saying either of those specifically is the solution, but they feel in line with the underlying idea, and so maybe skipping ahead to one of those will trigger an idea that also works in the generic case.
(I still prefer splitting up the package into foo and foo-core and putting the CLI/whatever bits in the first one, but trying to prompt helpfully, since the discussion appears stuck.)
As an “end developer” who works with a lot of people who are new to python (I write scripts and small applications for $WORK but don’t maintain any public libraries), this dichotomy aligns with my intuition - interactive installs by novice users should make it very easy to include recommended or default options. But it also is equally unappealing to have dependency specifications[1] install things that are not explicitly needed. And, as made clear up thread, it’s entirely untenable to have different semantics for specifying packages for installation in CLI vs in a dependency specification list (such as pyproject.toml).
I thinking about this, I noticed that the discussion has largely ignored the opportunity for tools to interactively query whether recommended defaults are desired. That gives the opportunity to clarify user intent in precisely the situation where novices are most likely to need help.
There is some precedent for this: poetry init gives the opportunity to interactively define dependencies, with the user being prompted to confirm which package was intended and to specify a version string.[2]
Obviously, the UX for any given installer tool is not within the domain of a PEP. However, a PEP that standardizes a means for package authors to declare which optional dependencies are recommended could allow installers to prompt for additional clarification from the user in scenarios where that is appropriate.
As a preliminary example, a key could be added to pyproject.toml that explicitly declares which extras are recommended. From there, pip, for example, could add an extra interactive check to pip install <package_name> with a message like: Package <package_name> includes recommended extra dependencies. Do you wish to install them? (yes/no) [yes]
It would then be left to the installers to decide exactly when such interactive requests are appropriate (pip likely having the most challenging balance as it’s used both by utter novices and as part of all manner of automated installs).
I would also pose related questions to the various installer tool authors:
Has there been any consideration of interactively prompting users for extras to install?
If so, why has it not been implemented?
Would having extra information from package authors about which extras are recommended change those decisions?
My thoughts on an interactive installer is that, when you want it, it would have only slightly more visibility than the informational messaging hinted at above…
… but would be much too visible when you really don’t want it. In particular, an unattended install has to choose between deadlocking, erroring out or silently taking a guess.
So in absence of some extremely reliable heuristics as to whether this is an unattended install or not, I’d always take messaging over prompting if either were seen as enough for everyone else.
Perhaps the message could be limited to explicitly requested packages if it’s deemed too noisy otherwise. i.e. pip install astropy would message but not any of:
pip install . # where astropy is a dependency
pip install depends-on-astropy
pip install -r requirements.txt
This was exactly what I was trying to get at when I earlier suggested “explicit” vs. “implicit” package installation. IIRC, you and many others were unfavorable about that distinction.
I would push back gently on the idea that an interactive prompt which requires the user to, at minimum, press a key is “only slightly” more visibility than a warning at the end of console output. If nothing else, the YOLO outcome from pressing enter without reading anything would be to get you the recommended extras you probably wanted.
100% agreed that having an interactive prompt lock up a silent install is completely unacceptable. I don’t pretend to know enough about pip to begin to guess how that could be handled, but we have a few people in this thread that can. If nothing else, you could teach "run pip install --with-recommended-extras <package_name> as the new incantation for novice users (or maybe pip install -X <package_name>).
I guess I was thinking exactly this, but if you substitute “message” for “prompt”. IMO, none of these examples should install or even notify about recommended extras (edit: in astropy).[1]
The last might add unacceptable complexity to how pip handles requirements files, but that’s up to pip’s maintainers to weigh in on. ↩︎
Exactly. As I said in my post, different semantics are unacceptable to me. The point I was making was that people keep focusing on the pip install pkg case, when that isn’t the hard case to justify. Unless someone can persuade me that pkgin a dependency specifier should mean “pkg, its dependencies, and its recommended extra(s)” then it’s irrelevant whether you persuade me that pip install pkg should install recommended dependencies.
This is a good idea, although I suspect people wanting default installs won’t like the fact that it doesn’t give any UI guarantees. But limiting the standard to providing a way for projects to declare their recommended dependencies, while leaving the UI to tools, seems like it fits well with our other standards.
Looking at prior art, in the form of apt’s “recommends” and “suggests” metadata would be useful, as well.
If we did go with the approach of limiting the PEP to a standard to declare recommended dependencies while leaving it up to packaging tools to decide what to do with it, then it might make sense to do this via one or more new metadata field(s) such as
Recommended-Extra: recommended
rather than standardizing on an extra name, as this will first avoid any conflict with existing extra names in use, but also it opens the door to having ways of specifying that at least one extra from a set is recommended, which could be useful for frontend or backend scenario, for instance:
The names aren’t important for now, but more the concept that this could be done via metadata. This then also provides a way to deal with the multiple front-end and multiple backend scenario as these fields could appear multiple times:
This would mean that at least one from each list is recommended.
Some tools could even choose to be smart about these and check if any of the dependencies needed for any of these extras are already installed to know if the recommendation is fulfilled.
if you do like the standardized opt-in approach, we have to carefully consider the namespace collision problem. Any convention for an opt-in extra name could be used for other purposes already, so how would you handle that? Or we’d need to pick something that can’t possibly collide with existing extra names.
.