Adding a default extra_require environment

[pf_moore (Paul Moore)]

@agronholm

That is why it’s important to not subtract from the extras set unless all dependents say that they don’t want this particular default extra.

… and of course, if wheel were already present (along with its packaging dependency but not pyparsing) and then something else (foo) was installed depending on packaging, the installer would need to recognise that the installed packaging needed pyparsing adding in order to now satisfy the dependency of foo. That’s a fairly complex process that I imagine would take some effort to get right, so I fear that this could become a bug magnet

[agronholm (Alex Grönholm)]

@pf_moore

… and of course, if wheel were already present (along with its packaging dependency but not pyparsing) and then something else (foo) was installed depending on packaging, the installer would need to recognise that the installed packaging needed pyparsing adding in order to now satisfy the dependency of foo.

You’re right of course. I’ve been wondering how pip’s new resolver handles this. Back in the day there was a dead-end discussion around the topic of how to gather the existing requirements when installing new stuff so the installer won’t introduce any conflicts. All the options there were deemed either too slow or too error prone.

[pf_moore (Paul Moore)]

@agronholm

I’ve been wondering how pip’s new resolver handles this.

It’s a bit of an open question. See Warn users about dependency conflicts when updating other packages · Issue #7744 · pypa/pip · GitHub.

Also, we don’t currently record what extras were requested when a package was installed (I think there has been some discussion on this, but I don’t believe anything was ever agreed/standardised) so it’s not even possible at the moment to know that “packaging was installed without the parser extra” (for example).

[sinoroc]

@pf_moore

Also, we don’t currently record what extras were requested when a package was installed […] so it’s not even possible at the moment to know that “packaging was installed without the parser extra” (for example).

Which, I believe, would make it difficult for pip freeze, and any other tool that outputs requirements.txt files. For example, those tools would need to infer packaging[-parsing] from the content of the environment (pyparsing is not installed, or maybe it is but for a different reason) and I don’t know if that’s possible (without a relatively large amount of work). But still it might be a worthy goal.

[uranusjr (Tzu-ping Chung)]

@sinoroc

Which, I believe, would make it difficult for pip freeze , and any other tool that outputs requirements.txt files. For example, those tools would need to infer packaging[-parsing] from the content of the environment ( pyparsing is not installed, or maybe it is but for a different reason)

I believe the intention of pip freeze is to provide a way to decribe an environment for the user to replicate as closely as possible elsewhere, not to reverse engineer what the user originally did to populate the environment. So if we ever go with he subtraction design, pip freeze can simply emit the package with all default extras stripped (or as package[]) and record the packages pulled in by the default extras independently.

[sinoroc]

@uranusjr

So if we ever go with he subtraction design, pip freeze can simply emit the package with all default extras stripped (or as package[] ) and record the packages pulled in by the default extras independently.

If that’s the case, yes, looks like it would work. I thought there was still some disagreement as to what package[] should mean. Anyway, that would be a backwards-incompatible change, right?

[uranusjr (Tzu-ping Chung)]

@sinoroc

I thought there was still some disagreement as to what package[] should mean. Anyway, that would be a backwards-incompatible change, right?

Probably. I didn’t think too deep into that (and honestly quite ambivalent to the whole “default extra” idea), and would rather leave y’all to figure out the details

[agronholm (Alex Grönholm)]

@sinoroc

I thought there was still some disagreement as to what package[] should mean

Isn’t it logical that if you install A[extra], then A[] would just mean A without the extra? Assuming of course that specifying any extras doesn’t clear the set.

[sinoroc]

@agronholm

Isn’t it logical that if you install A[extra] , then A[] would just mean A without the extra? Assuming of course that specifying any extras doesn’t clear the set.

I see the logic. I wasn’t sure if it is the current consensus.

(I have a piece of code that produces requirements.txt files, so I keep an eye out for backwards-incompatible changes.)

[pf_moore (Paul Moore)]

@agronholm

Isn’t it logical that if you install A[extra] , then A[] would just mean A without the extra? Assuming of course that specifying any extras doesn’t clear the set.

It seems logical, but that doesn’t mean that it shouldn’t be explicitly documented somewhere. And the implications need to be thought through (for example, I wouldn’t want to see people starting to cargo-cult the idea that requirements should be specified as A[] “just in case” the package adds a default extra that you don’t want…

But like @uranusjr I’m mostly-ambivalent about the whole thing, so I’m happy for the enthusiasts to sort out the details

[pradyunsg (Pradyun Gedam)]

@brettcannon

This part I have never understood the point of (the “PEP + packaging.python.org” part, not why the specs are kept where they are part), so I can’t help answering the question.

Relevant topic: How to propose new specs - #7 by pradyunsg

[njs (Nathaniel J. Smith)]

I wonder if “default extras” are even the right solution. This thread seems to have started by assuming that’s the mechanism we want and trying to figure out how to do it, rather than starting from use cases and figuring out what mechanisms would help them.

Some other possibilities:

• For the “seriously don’t install pandas, I know what I’m doing” use case, maybe what you want is some kind of “override requirement” you can specify: put !override -pandas in your requirements.txt or something, and that has the semantics of forcing pip to delete all dependency arcs pointing to pandas before performing resolution. (Sort of reminiscent of !important in CSS.)

• If the idea is that a package has some dependencies that aren’t strictly necessary but will be wanted in most configurations, then something like Debian’s “Recommends:” field would make sense (i.e., a new variant of install_requires rather than a new variant of extras_require). The semantics would be that normally all install_recommends packages will be installed, unless you run pip --no-recommended or something like that.

[agronholm (Alex Grönholm)]

@njs

For the “seriously don’t install pandas, I know what I’m doing” use case, maybe what you want is some kind of “override requirement” you can specify: put !override -pandas in your requirements.txt or something, and that has the semantics of forcing pip to delete all dependency arcs pointing to pandas before performing resolution. (Sort of reminiscent of !important in CSS.)

Forcing the installer to skip a dependency is not really within the scope of this discussion.

@njs

If the idea is that a package has some dependencies that aren’t strictly necessary but will be wanted in most configurations, then something like Debian’s “Recommends:” field would make sense (i.e., a new variant of install_requires rather than a new variant of extras_require ). The semantics would be that normally all install_recommends packages will be installed, unless you run pip --no-recommended or something like that.

I don’t really see how this would be an improvement over the current proposals – it just adds one more mechanism (one which is less powerful than the proposed ones).

[njs (Nathaniel J. Smith)]

@agronholm

Forcing the installer to skip a dependency is not really within the scope of this discussion.

@agronholm

one more mechanism (one which is less powerful than the proposed ones).

My point is that this discussion seems to have started with a predefined scope and mechanism instead of starting with problems and looking for appropriate solutions, which is putting the cart before the horse.

[agronholm (Alex Grönholm)]

@njs

My point is that this discussion seems to have started with a predefined scope and mechanism instead of starting with problems and looking for appropriate solutions, which is putting the cart before the horse.

Well, nobody has presented any better solutions so far That said, adding a new mechanism to the metadata/installer system is not out of the question but would have to provide a significant improvement over the status quo and the proposed system to outweigh the extra burden on packaging developers.

[pganssle (Paul Ganssle)]

@njs

My point is that this discussion seems to have started with a predefined scope and mechanism instead of starting with problems and looking for appropriate solutions, which is putting the cart before the horse.

People do frequently ask for several related things:

1. “Default extras”: an extra that is overridden by the presence of other extras
2. “Dependency groups” where your requirements can be satisfied by multiple sets of requirements (e.g. “Needs a database, if it’s postgres I need X, Y and Z, if it’s redis I need A and B, etc”)
3. Build-time feature flags (e.g. “give me X but built against MKL”)
4. Enhancing dependencies (i.e. “Recommends:”) — for example, when you have a fast and a slow version of your library where the slow version is harder to distribute / build.
5. “Negative extras” - either an extra that is included by default but can be explicitly removed, or an extra that removes a dependency (this is another flavor of “enhancing dependencies” — your library comes with all the features it supports, and your users can declare a partial dependency on your library if they don’t use certain features).

I think these are all closely related and the use cases for each one gets jumbled up with all the others, because each mechanism lets you satisfy a few of the use cases from the others. For example, “default extras” lets you implement a single “dependency group”. “Feature flags” can often be implemented in terms of various extra-related logic by refactoring your application into a “core” with all the flag defaults, and then putting each additional feature into a separate package.

I agree with @njs’s suggestion that we look closely at the use cases here, because we may find that if we implement say #2 and #4, it would most closely satisfy the needs of everyone asking for #1, #3 and #5.

[agronholm (Alex Grönholm)]

I agree that collecting use cases (preferably from the real world) would really help clarify what needs to be done here.

[ofek (Ofek Lev)]

@agronholm

collecting use cases

I’ll re-post mine: Adding a default extra_require environment - #30 by ofek

@ofek

My use case would be for projects that have both an API/library component and an app/CLI/GUI component. I would like the average end user to simply do pip install project and the dependency definition would be pip install project[lib] . Currently we’re forced to do the opposite: Setup - Agent Integrations

[EpicWink (Laurie O)]

This was suggested above, but in your case @ofek you could split out your library into a separate distribution project-lib. You wouldn’t even have to create a separate package if you use namespace packages.

Is there anything that default dependencies can solve that can’t also be solved (reasonably) by splitting up the project?

[ofek (Ofek Lev)]

I’d say the same as this response: Adding a default extra_require environment