Hello everyone,
IMO PEP 440 is not clear about it, so I hope to get some clarity here: Are wildcards alllowed for versions in a comparison clause like >=3.7.*
There are packages in the wild that define their requirements like this, but is this correct according PEP 440? It’s neither explicit excluded nor allowed in the PEP.
Based on the result of the discussion here, can we update the PEP to clarify this?
fin swimmer
I say let’s declare them invalid; they are not useful in practice (>= et al. already performs segment-stripping on their own without the wildcard) and only make things more complex than needed.
IMO PEP 440 is sufficiently clear - that is allowed (>=3.7.* means "strip everything after the first 2 components of the version, then test if the result is >= 3.7, using the normal rules). It’s not much use in practice, but it’s valid.
Is there an actual problem being caused by this? How does the packaging library implement it?
if we update the PEP to clarify that wildcards are allowed, I’d consider that OK (not necessary, but being clearer is never a problem). If we decide it’s not valid, I think that’s a spec change (and clearly it’s incompatible with the cases you’ve found) so it’d need a new PEP to change the specification.
It might be good to migrate PEP 440 to Version specifiers - Python Packaging User Guide, so we can deal with these things there (which is the correct approach) rather than by modifying an approved PEP (which is a non-standard workflow peculiar to packaging PEPs). But I don’t know if anyone is likely to have the time/inclination to do that.
Am I misreading the PEP? The .* suffix is defined under “version matching” (==) and is only inherited by “version exclusion” (!=).
Apologies, you are correct. It was me who misread. So I will therefore reverse my position - .* is “clearly” (
) only valid for == and !=. I guess that confirms that a clarification would be useful, too, if only to stop me from giving dumb answers in future 
I’d still ask how the packaging library handles this case. If we’ve been accepting wildcards in other comparisons “by accident”, we might need to continue doing so, simply because doing otherwise would (apparently) break real-life packages, and there’s no way of changing a release’s dependencies “after the fact” (even yanking won’t work, as projects pinning a yanked version will still get that version (see here).
Thanks for your answers!
Problems surfaced because at poetry we reworked the code that interprets the version requirements and handling the wildcards in this case now as invalid: Ignore period for version parse · Issue #4176 · python-poetry/poetry · GitHub
AFAIK pip and poetry<1.2 accidently allows the wildcard.
What if we add a paragraph, that says that something like >=3.7.* is discouraged (because it’s useless) and programs that parse those constraints must ignore the trailing wildcard?
packaging does not support using .* with operators other than == and != and tests that this usage produces an error, see packaging/tests/test_specifiers.py at e2bc8f6ef2d66f6eaaf522fcf70a0f203b734064 · pypa/packaging · GitHub.
A Specifier raises an error, but a SpecifierSet (which is what pip uses) works. Pip vendors packaging 21.0, which is the latest version on PyPI and
>>> from pip._vendor.packaging.version import Version
>>> from pip._vendor.packaging.specifiers import SpecifierSet
>>> s = SpecifierSet(">=1.2.*")
>>> s
<SpecifierSet('>=1.2.*')>
>>> s.contains(Version("1.4"))
True
>>> from pip._vendor.packaging.specifiers import Specifier
>>> Specifier(">=1.2.*")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Gustav\.virtualenvs\917e17bf5657136\lib\site-packages\pip\_vendor\packaging\specifiers.py", line 105, in __init__
raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pip._vendor.packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=1.2.*'
So I’m afraid it looks like packaging and pip have been accepting that usage, which means we need to be careful to understand the impact if we now implement the spec as written 
Oh, that is a very unfortunate oversight.
Fortunately (?) SpecifierSet parses this as a LegacySpecifier
>>> from pip._vendor.packaging.specifiers import SpecifierSet
>>> s = SpecifierSet('>=1.2.*')
>>> s._specs
frozenset({<LegacySpecifier('>=1.2.*')>})
So we could deprecate this usage as a part of the move to mandate PEP 440 compliance and drop support for it when we drop LegacyVersion. This is still not very easy to do (we’ve been delaying that for quite a while…) but at least we are technically allowed to pull it at any time.
I’ve no idea. From a standards point of view, the PEP is OK, but maybe a clarification would be worthwhile.
For the practical matter of how we start enforcing the PEP, that’s down to individual projects. Pip defers to packaging, so we’re OK (in a “not our problem” sense 
). But I don’t have a particular view on what (for example) packaging or poetry should do.
Regarding Ignore period for version parse · Issue #4176 · python-poetry/poetry · GitHub, I would say:
Personally, I think we’re going to have to bite the bullet and break compatibility at some point, but we should do a significant communication and outreach exercise to let the community know, and to help users deal with the inevitable fallout. But PyPA isn’t organised in such a way that we can handle such activities, so I don’t know how that can happen without a governance change (turning PyPA into much more of an “authority” than it currently is).
Don’t take any of the above as anything other than my personal opinion. My only “official” role here is around how we manage any change to the various standards (which as I say is just a matter of possibly clarifying the existing intent).
Has anyone made sure there’s an issue open on packaging at GitHub - pypa/packaging: Core utilities for Python packages to discuss it there?
Not that I know of. But @uranusjr noted that this is simply packaging falling back to LegacySpecifier when the expression doesn’t conform to PEP 440, so I’m not actually sure there’s an issue to raise. I’ll leave it to @finswimmer to raise something if they can work out what they want to ask…
Does packaging emit a DeprecationWarning when a LegacySpecifier is created? We should add it if not.
Yes, although it says “LegacyVersion”.
Is this really a problem by packaging?
I don’t know how tight pip and packaging are coupled together. I would say, that it is ok if packaging falls back to LegacyVersion if it cannot parse the version range and it would be pip's job to say: “we don’t support this kind of version”.
Is this even a problem in pip? Why is it so important that pip drop support for legacy specifiers before packaging does?
I don’t think that it is “important”. It’s just the question about who is responsible and take the lead - someone has to.
For me it seems, that this change has the potential to remind users to the python2-python3 transition, because lots of packages (how many??) cannot be installed anymore. So I still think that it might be much more painless to adopt the PEP to the reality for this special case …
In the meantime we decided at poetry to include a hack/workaround to allow wildcards in comparison clause for now.