For packages I am contributing to, we have an extra called recommended which are optional but recommended dependencies. I would like that to be the default extra, but then give users the ability to do minimal installs, which is the “no extras” case (the ability to do minimal installs is something that is important for some of our users, as some of the recommended dependencies are quite hefty and not needed for a reasonable fraction of the package). To be clear, this is different from installing with --no-deps since the minimal set of strictly required dependencies is not empty.
I’d be happy with either package[] or package[-*] for this - my point is just that I don’t want users to have to figure out what the name of the default extras are to select them. I personally find package[] to be more pleasing and intuitive, as no one is going to bother writing [] if it has no effect anyway (and [-*] looks like a strange smiley). But both are better options than package[-recommended] in my view.
Given that users have to find out that in order to not get the recommended extras, they have to type something, how much harder is it to explain that they type [-recommended] instead of explaining that they write []?
I don’t think we want to end up with users routinely using package[] “just in case” they need it in order to get a minimal install. And unfortunately that sort of blind copying of inappropriate advice is all too common…
I quite agree with @pf_moore. If we have “default extra”, you could define an extra "minimal: [] and do package[minimal]. Which eventually would deactivate any sort of “default extra”.
The advantage of seeing the usecase that way, we don’t have to complexify a feature request that had already been stalling for 2 years … Doing progress with the initial feature scope would be a good thing already
Ok that sounds good with regards to not having [] mean “no extras” - I can live with [-recommended] and can see how it is probably better to be explicit.
From reading the the thread, I wasn’t sure if there was consensus on whether specifying extras explicitly disables the default extras or simply adds to them - the [minimal] example would only work in the former case, but I thought there were objections to this?
I think that’s the point of “default” anything. You can get the default if you don’t ask for anything or the “not default” if you ask for something.
See it as “how do you want your meal ?” - default=mild_spicy but meal[no_spicy] and meal[super_hot] are available. Quite obviously asking super_hot removes the default
This expectation really depends on how you’re using extras. Your example supposes they are alternatives, but I often see them used as feature sets. For example, if I see a package with default extra common, I would expect package[extra] to be equivalent to package[common,extra].
Right I was totally expecting this reply pizza[tomato_sauce,mozzarella]. Short coming of my example …
But in general a “default” is what you get if you don’t specify anything. Think about a default argument value:
def foo(bar=None):
if bar is None:
bar = [dep_a, dep_b]
print(bar)
You can:
not define bar
define it and override the default value
I gotta say I’m disappointed someone dismantled my awesome & yummy example so fast
We’re repeating things that have been discussed in this thread already.
I suggest we hold off on further discussion until someone writes down a design document proposing a specific design and discussing why they didn’t take the alternatives AKA PEP.
I believe @RonnyPfannschmidt had also spent some time looking at authoring a PEP about this but hasn’t had the time to finish that (please correct me I’m wrong).
If someone else wants to pick this up and you haven’t written a PEP before, please feel welcome to reach out to me about sponsoring the PEP.
@RonnyPfannschmidt - just out of curiosity, did you get a chance to take another stab at the PEP?
I am interested in trying to help with this, so let me know if there is anything I can do. If you don’t have time to work on this I’d be happy to try and take the lead on the PEP - in that case feel free to share what you had already, or let me know if I should just start with a blank slate. Thanks!
@pradyunsg I’d like to try and pick this up and push it forward, and am working on a PEP draft. I’ll reach out to you privately once I have a draft to see if you would be happy to sponsor it (I haven’t written a PEP before).
With the idea that it’s better to have something to discuss and iterate on, I’ve had a chance to try and write up an initial draft, which I’ve put in my fork here, and have opened up a PR to my own fork to make it easy to review/discuss directly:
It’s very much an incomplete draft at this point but I thought it would be a good starting point. I’ve included a link to preview the PEP as-is in that PR. If anyone else would like to join as an author, please just leave a comment in the PR and I can add you!
Following more discussion over at my pull request, the conclusion for now is that introducing the new -extra syntax is going to be a proposal that is very hard to sell as there is no way to introduce new syntax into PEP 508 specifiers without essentially breaking support for older versions of packaging tools such as pip.
Therefore, for now we are instead focusing on the approach of having default extras only being installed by default if no extras are explicitly specified - this was not everyone’s preferred approach (including myself) but it might end up being the best compromise and actually allows for more flexibility and customization than I though might be possible - several scenarios are now mentioned in the PEP to highlight this.
If you are a package maintainer and interested in potentially making use of this feature, could you take a look at the PEP draft at the above link and confirm if you think this will work for you, or if it still does not address your use case? (even just a would be useful)
I think that I would like to use this for some packages. My use case is pure Python packages that have optional non-Python accelerator modules. One example is that SymPy can now make use of python-flint as an optional dependency to speed up some operations. Eventually I would like users who do pip install sympy to get python-flint by default but there will always be a need to opt out because python-flint includes libraries with a different license and also is much more difficult to build. We might also want to have other optional dependencies like numpy, scipy and matplotlib be part of the default extras.
Ideally I would want to have the default requirement conditional on the presence of a wheel for the accelerator module i.e. “install python-flint if there is a wheel but otherwise don’t try to build it”. I guess I could make that effectively happen with environment markers though at least for someone installing from PyPI.
Currently the way to achieve this without default-extras is to split a package up so then you have an empty sympy package and a sympy-minimal package that contains the actual code. The sympy package’s requirements would include the default extra requirements but the sympy-minimal package would not and then downstream packages can depend on sympy-minimal if they want. This is awkward to do mainly just because it is awkward to make multiple packages from a single git repo and coordinate releasing them. An alternative that could achieve what I want then would be if there was better tooling for generating multiple packages from a single git repo although it would still be nicer to use default-extras rather than making another package. (There are other reasons to want multiple packages though like to split out the test suite into a sympy-test package.) The separate packages approach is possibly easier to adapt for downstream packagers like distros etc.
The PEP seems to describe things in terms of setup.cfg but should these things not be in pyproject.toml?
This looks entirely reasonable to me, and I would definitely use this functionality and propose adopting it to upstream dependencies that I would like to include minimal variants of.
Thank you for putting the time into this proposal. I had stopped following the discussion after concluding that consensus was unlikely to be reached, so I hope this (or something like it) is broadly satisfactory.
How do you expect downstream (non-pip) package managers that have no concept of recommended dependencies to handle the ambiguous dependencies? Should apk add py3-some-package unconfigurably install the recommended packages or use the minimal base set?
Also, am I really the only person who doesn’t look forward to the idea of, every time I want to install a package throughout the rest of my Python career, having to find the advanced installation documentation section or checking the source code’s pyproject.toml to figure out if/what synonym of minimal I need to put into the square brackets in order to not pollute my environment with dead weight? Even if it gets standardized, I’d still need to remember to append [minimal] to every package name in every pip install.
That’s a good question - and interestingly I’ve seen that come up a lot in the context of conda packages, which don’t have the ability to use extras/optional dependency sets. In fact, the situation there is very inconsistent, some conda recipes include all optional dependencies, while some use more minimal sets. As a result, some packages now have multiple recipes, e.g. matplotlib-base and matplotlib. So this is to say that there is already a problem anyway even with extras as they are. Packaging systems such as apt do actually include the concept of recommended but not required dependencies so in that case the concept here would map well onto it.
What you might want in your case is for pip to support a command-line flag such as --no-default-extras that automatically disables/ignores default extras - the PEP draft above mentions this and doesn’t specifically recommend it, leaving it up to the individual tools to decide whether to support it. I personally would find such a flag useful.
The flipside of this is needing to read the installation instructions to figure out which version of package[featureA,featureB] actually provides the features that you expected the package was supposed to provide or not realising that package[accelerator] makes your code run 1000x faster.
I do agree with the concern though. I am generally amazed by how much space a venv takes up these days and how many dependencies get pulled in for what seem like simple packages. That happens already now when there are no default extras and we don’t have any way of opting out. Maybe with default extras it would be more common that packages would provide a better minimal install since it can be something distinct from the default install.
Well I do, and I like the idea of something that’s pip.conf-able but… I think it’ll make it even worse. If package foo thinks it depends on package bar but doesn’t realise that it needs bar[thedefaultgroup] then pip install foo would be broken with the flag. Similarly, if bar had no recommended dependencies then decided to move a mandatory to a recommended one, thinking this would not be an incompatible change, then anyone who uses the flag again receives a potentially broken package. To mitigate this, I think we’d end up having to standardise the recommended name then get into the habbit of always writing foo[recommended] when specifiying dependencies. It’s puts quite a burden on package developers to always test a --no-default-extras built setup.
But if we’re going into pip details then does anyone have a plan for how
pip install package[minimal]
pip freeze > requirements.txt
# In a new environment
pip install -r requirements.txt
is going to avoid pulling in package’s recommended dependencies?
Yeah, you can argue it either way. Either a getting started page can say run pip install package[accelerator] then explain what benefits you’d get with the accelerator bit or it can say run pip install package or pip install package[minimal] and again explain the difference. (I’d prefer the explicitness of the first but I know that’s just a preferences thing.)
A lot of packages that have optional cython/rust extensions tend to treat the opt out users as second class citizens (in both documentation and testing). I suspect that pip install package[minimal] users will end up the same.
would be useful)