Negative extras (extra !=) are currently technically allowed in the grammar, but they behave strangely: extra != means that package could transitively remove a package from selection through activating an extra. This turns incremental resolution into a global problem: We can’t exit a resolution with a conflict when we’ve exhausted all versions in range including backtracking, we also have to exhaustively search for any package version that may activate the extra which deactivates the extra != requirement which is part of the conflict. This would be fatal to resolver complexity and performance, preventing most of the short-circuiting that makes resolving tractable.
Much weirder, with negative extras, adding a new dependency can make a previously impossible resolution solvable:
[project]
name = "foo"
version = "0.1.0"
dependencies = [
"numpy >2",
"numpy <2; extra != 'old-science'",
]
Only by adding a package with Requires-Dist: foo[old-science], this resolves.
Using PEP 621, users only get positive extras: project.optional-dependencies.foo gets translated into a conjunction of extra == "foo" with each requirement’s marker.
With the power of hindsight, it does feel like we should have only allowed “in” as an operation on an extras variable. Until we have a separate design process to figure out how to move to a better “feature/logic” selection process, restricting this syntactical misfeature is worth removing.
It would be a useful data point to know how much of PyPI uses this sort of dependency declarations. Additionally, having a rough idea for what possibly eventually refusing to parse these forms with appropriate errors across the board (installer, indexes, build-backends) would need to be ordered as – to have a bit of a coordinated/thought through rollout for these changes.
It may worth expanding to “only extra == "x" is defined”.
Currently packaging.markers.Marker.evaluate returns the following for each operator on extra:
marker
extra="gpu"
extra=""
extra="cpu"
extra == "gpu"
True
False
False
extra != "gpu"
False
True
True
extra <= "gpu"
True
False
False
extra >= "gpu"
True
False
False
extra < "gpu"
False
False
False
extra > "gpu"
False
False
False
extra in "gpu"
True
True
False
extra not in "gpu"
False
False
True
extra === "gpu"
raises UndefinedComparison
extra ~= "gpu"
raises UndefinedComparison
<= and >= behave as ==, < and > never match, in/not in do substring matching on the extra string (extra in "gpu" is true for the empty extra), and ~=/=== raise. Only == has a usable meaning.
!= evaluates cleanly as set non-membership, the problem is in the resolution model: whether a != dependency is present depends on the full set of extras requested for the package, which is not settled until every requirer is resolved, and activating an extra can remove a dependency.
Two consequences:
Some graphs have no valid install:
project: Requires-Dist: cpu-only ; extra != "gpu"
Provides-Extra: gpu
Requires-Dist: gpu-only ; extra == "gpu"
cpu-only: Requires-Dist: project[gpu]
cpu-only is required unless gpu is active, and cpu-only activates gpu. If cpu-only is installed, gpu is active, so project does not require it; if cpu-only is absent, gpu is inactive, so project requires it. No install satisfies both, and a resolver either errors or silently installs a wrong set (for example gpu-only with cpu-only dropped, though nothing requested the gpu extra).
Other graphs have a valid install that incremental resolution misses:
project: Requires-Dist: plugin ; extra != "gpu"
Requires-Dist: ancient>=2 ; extra != "docs"
Provides-Extra: gpu, docs
Requires-Dist: gpu-only ; extra == "gpu"
Requires-Dist: docs-theme ; extra == "docs"
plugin: Requires-Dist: project[docs]
ancient: only 1.0 is published, so ancient>=2 is unsatisfiable
{project, plugin, docs-theme} is valid: plugin is required (gpu was not requested), plugin activates docs, and docs being active drops ancient>=2 ; extra != "docs". A resolver that decides the negative-extra deps before docs is activated commits to ancient>=2 and fails.
$ uv pip install --no-build "feast[flink]"
Resolved 68 packages in 94ms
Prepared 33 packages in 799ms
Installed 68 packages in 30ms
[...]
+ feast==0.63.0
[...]
warning: The package `feast==0.63.0` does not have an extra named `flink`
$ uv pip install --no-build "feast[flink]==0.64.0"
× No solution found when resolving dependencies:
╰─▶ Because feast[flink]==0.64.0 depends on pyarrow>=16.1.0,<21.0.0 and feast==0.64.0 depends on pyarrow{python_version < '0'}>=21.0.0, we can conclude that
feast==0.64.0 and feast[flink]==0.64.0 are incompatible.
And because you require feast[flink]==0.64.0, we can conclude that your requirements are unsatisfiable.
$ pip install --only-binary :all: "feast[flink]==0.64.0"
[...]
ERROR: Cannot install feast and feast[flink]==0.64.0 because these package versions have conflicting dependencies.
The conflict is caused by:
feast 0.64.0 depends on pyarrow>=21.0.0; extra != "flink"
feast[flink] 0.64.0 depends on pyarrow<21.0.0 and >=16.1.0; extra == "flink"
[...]
The use case I have in mind (which I opened those two issues in pip and uv for) does not require this.
Specifically, I want to build a package X that by default depends on onnxruntime, but X[gpu] depends on onnxruntime-gpu. The two packages onnxruntime and onnxruntime-gpu overlap / provide the same libraries, so if you try to install both, whichever one is installed second wins. (This is arguably a problem in its own right, but it’s a harder one to solve well.) What I would like is some way to specify that, if you pip install X[gpu] (or uvx X[gpu] etc.), please skip the onnxruntime dependency and only install onnxruntime-gpu instead.
I’m already resigned to the fact that if something in my transitive dependencies pulls in onnxruntime or onnxruntime-gpu directly, then there’s nothing I can do to suppress those dependencies. Or, worse, if I have two things in my transitive dependencies that pull in each of those packages, then I have a file conflict, and the only thing I can do about that is file bugs against my dependencies or stop using those dependencies. That’s already a risk, and I’m not trying to solve that.
So, I don’t see it as a problem that I might have some dependency chain X → Y → X[gpu] that introduces the gpu extra at some later point during resolution. If it does that, then the correct resolution contains both onnxruntime and onnxruntime-gpu, that’s fine. It’s no different from a dependency chain X → Y → onnxruntime-gpu. All I want is that, if the only dependencies on X are for X[gpu], such as with a top-level uvx X[gpu], then don’t install plain onnxruntime. If you start with a top-level uvx X, then onnxruntime gets included in the dependencies and remains there, regardless of what happens later during resolution.
In other words, I’m not requesting a conflict marker. When I say onnxruntime; extra != 'gpu', I don’t mean “if the gpu extra is activated, conflict withonnxruntime”. I simply mean, “if the gpu extra is activated, don’t bring in onnxruntimeon my account”. If something else brought it in, such as a previous node in the dependency graph of X with no extras, then it’s already in the dependency graph and there’s nothing I can do about it. Dependencies still remain additive.
In the semantics I’m proposing, the initial dependency on foo would mean that the extra != 'old-science' marker evaluates as true, and so the numpy <2 dependency stays around. Even if there is a later dependency on foo[old-sicence], it would not suppress that existing dependency. The only way you would be able to get your example to resolve is if you request foo[old-science] at the top level and never request foo.
(Maybe my mental model of this is wrong, and the initial foo and the later foo[old-science] aren’t treated as separate nodes? Or maybe that’s not how the uv resolver treats them or something?)
FWIW I agree that this would not be a breaking change for this reason (even though I would be sad about that outcome).
I had this report generated from 87% of PyPI bigquery metadata (after which the downloader died), but I think that gives a good enough view of the usage: pypi-metadata-extra-not-equal.md · GitHub. I can generate a better, more complete report if there are concerns remaining - these packages are at least broken for that extra as of today.
What you’re proposing sounds like default extras, with foo[old-science] being extra-deselection. This would be a great addition, but should probably be part of the default extras proposal, as it’s different from the extras model we today, where foo[old-science] implies as superset over foo; uv internally splits foo[old-science] into a dependency on foo and on foo[old-science].
Right, I would like to change this (assuming it’s amenable to uv, pip, etc. resolvers, which I don’t know the details of) to saying that both extra == and extra != are evaluated solely on the extras for this particular instance of the package in the dependency graph. If there is another instance somewhere else in the dependency graph with a different set of extras, then those markers are evaluated again on that set of extras, and the resulting requirements are joined additively, just as if they were two different packages.
But this also made me realize another discrepancy between the spec and the pip/uv implementations. Suppose you have
Requires-Dist: something; extra == "this" and extra == "that"
Intuitively, it seems like this should cause foo[this,that] to install something.
By the rules as you’ve described it, this would also cause foo[this] foo[that] (or some equivalent setup, like a dependency chain foo[this] → bar → foo[that]), to install something, because you’d be evaluating set membership in the list of all the accumulated extras for every time we encounter foo in the dependency graph.
However, as best as I can tell, both pip and uv evaluate this condition to false i.e. never install something, which greatly surprises me!
For pip this is because the relevant code (src/pip/_internal/req/req_install.py) evaluates markers one at a time:
if self.markers is not None:
return any(
self.markers.evaluate({"extra": extra}) for extra in extras_requested
)
(See this DPO thread from 2020 about how packaging doesn’t provide an API for the actual thing you want here, which is set membership.) I don’t yet understand why uv has this behavior; a cursory look at the code (which I’m not very familiar with) implies it does try to do actual set membership instead of looping through the set and evaluating the markers one at a time.
Konsti or Mike, if it’s easy for you to do, I’d be curious if there are any /extra ==.*and extra ==/ requirements in the wild, because those are also broken.
Yeah, I’m okay if the answer is “this is clearly a workaround for not yet having default extras.” I was vaguely under the impression that there was more complexity there, which is why I reached for the option that seems to already be allowed by the spec, but let me actually read through that proposal and discussion.
Another way to solve this, of course, would be to split my package X into three dists, X-impl which has the code but no dependency on either onnxruntime or onnxruntime-gpu, X which depends on X-impl and onnxruntime, and X-gpu which depends on X-impl and onnxruntime-gpu. Then uvx X and uvx X-gpu would both work out of the box.
I would absolutely love it if we could get real conflict declarations in Python packaging, and we’re probably at the point where enough of the tools use real resolution libraries that can handle conflicts without difficulty.
To make this useful for my use case, I’d also want a way to express a dependency on eitheronnxruntime or onnxruntime-gpu. I think there’s roughly two ways to do it. The more obvious one is to just add disjunctions, so I can do something like (syntax TBD):
Requires-Dist: onnxruntime | onnxruntime-gpu
Requires-Dist: onnxruntime-gpu; extra == "gpu"
which means my package always requires one or the other, but if the gpu extra is requested or if something else in the dependency tree pulls in onnxruntime-gpu, then that satisfies the disjunction. Then if onnxruntime and onnxruntime-gpu declare conflicts with each other in their own metadata, I’ll be sure to get exactly one or the other.
The other option is to revive something like Provides-Dist, where onnxruntime-gpu can say that it can be used to satisfy a dependency on onnxruntime. Then I can do simply
Requires-Dist: onnxruntime
Requires-Dist: onnxruntime-gpu; extra == "gpu"
and get either the real onnxruntime or onnxruntime-gpu. (In Debian-style packaging, which is what I’m most familiar with, this sort of thing can be done by having onnxruntime-gpuspecify all three of Provides:, Conflicts:, and Replaces:onnxruntime. The Provides line indicates it satisfies dependencies for the requested name, Conflicts indicates that they cannot be installed simultaneously, and Replaces instructs the resolver that it should consider solutions that involve uninstalling the other package. I think there’s a close equivalent in RPM etc.)
Both of these would also help with the case where an indirect dependency pulls in onnxruntime-gpu, even if my own package is installed without the [gpu] extra. Right now there’s nothing I can do about that short of opening pull requests to introduce extras in upstream packages. And I think both of these would also be straightforward for anything backed by a real resolution library.
For some context on extra <= and related nonsense, see Attempt to clarify environment marker evaluation by ncoghlan · Pull Request #1988 · pypa/packaging.python.org · GitHub - the current intention is that for all comparisons of strings that are not version strings, such as sys.platform, < and > are always false, and <= and >= are (therefore) equivalent to ==. extra is particularly special in that == and != test set membership/non-membership; the other deprecated comparison operators are consistent with that. The current spec says you can use extra == and extra != and nothing else (and Konsti’s proposal is to remove the text about !=).
(I don’tlike it and would personally support a declaration that this was a bad idea, presumably those with the commit bit at that project think otherwise)
The “this almost certainly won’t work, and if it does appear to work, it probably won’t do what you want” situation is always a good foundation for a SHOULD NOT recommendation.
I’ll let the thread run for at least a week before merging anything, but wearing my PyPUG editor hat, I’m inclined to accept the PR.
However, I’m also wondering if there are implications for the “Set of Strings” fields allowed in lock files (extras and dependency_groups): should those also disallow not in? My recollection is that those were defined on the ability to make sense of the standalone expressions rather than considering if they actually formed a useful set algebra when combined (for example, given a marker like "A" in extras and "B" not in extras, the fact that the transitive resolution of a dependency declaration on foo[A,B] is necessarily not a strict superset of the transitive resolution of foo[A] is genuinely weird. While that can sometimes happen anyway due to the combined declaration pulling in older versions of indirect dependencies, that feels more reasonable to me than the environment marker evaluations getting confused)
I do not think it was a bad idea. I think it is what was possible with the means available at the time it was implemented to solve a real problem. (That was before not in extras was invented.)
Probably, Poetry could deprecate using negative extras in the project section and only allow it (or alternatively not in extras in the tool.poetry section of the pyproject.toml) so that it is not used to create package metadata but only for locking.
I think, in lock files the negation is useful to describe the resolution for conflicting extras. Resolving conflicting extras is also the reason why Poetry allows negative extras at all. (This was implemented before PEP 751 introduced not in extras so that for Poetry, extra != "foo" and extra != "bar" is the same as "foo" not in extras and "bar" not in extras. Thus, I think not in must not be disallowed for extras and dependency_groups, which are markers for lock files.
I think it is what was possible with the means available at the time it was implemented to solve a real problem. (That was before