Should we permit `.*` with more comparision operators in version specifiers?

As some of you have likely noticed, packaging removed non-PEP backed parsing of versions and version specifiers in 22.0.

This has flagged that the following version specifiers with .* are invalid under the specification, as of today, despite having “obvious” equivalent semantics (i.e. ignore the .*):

  • > 1.0.*
  • >= 1.0.*
  • < 1.0.*
  • <= 1.0.*

A potentially actionable thing here is permitting these specifiers and using the “obvious” equivalent semantics for them.

Does anyone think we should permit such version specifiers?

x-ref Wildcard in setup.py install_requires list causing parsing error · Issue #645 · pypa/packaging · GitHub

Personally, I don’t have strong feelings on this, especially with a clearer error message like:

packaging.requirements.InvalidRequirement: .* suffix can only be used with `==` or `!=` operators
    black (>=20.*) ; extra == 'format'
           ~~~~~^

I also don’t want to unilaterally decide on this aspect and it’s worth having a broader discussion – hence moving this out of packaging’s issue tracker to here. :slight_smile:

The only reason I can come up with, to be more relaxed and allow this, is that existing packages won’t be broken by this change. I don’t feel particularly strongly about allowing that TBH.

Why is the * needed at all?
Why isn’t >= 1.0 usable now?

I’d prefer we don’t relax this restriction. No really strong reason, just that it’s straightforward to fix the issue by removing the .*, and changing this would be a PEP change, which is a pain (and I don’t particularly want to manage the necessary discussion and approval of such a change).

Let’s reframe the question, though. Is this important enough that we want to require every other implementation of PEP 440 to make this change? IMO, it’s not.

2 Likes

Well, yes but what about existing releases which have >= 1.0.*? It’s non-trivial to change that.

I never said it isn’t. I’m saying that people have been using the .* form already, and for… historical reasons… we’ve had this be the logic for parsing requirements and versions:

>>> from packaging.version import parse as parse_version
>>> from packaging.requirements import Requirement
>>> Requirement("name >= 48752389.ty89324ut589438vm29ct3c9j4m")
<Requirement('name>=48752389.ty89324ut589438vm29ct3c9j4m')>
>>> parse_version("89573498t298gh23gc54892mx34th9849 u2523r24hgm198t t8914hr9t1 4cf8019h 0th rf9084h02 n0r58r03rh`n98g")
<LegacyVersion('89573498t298gh23gc54892mx34th9849 u2523r24hgm198t t8914hr9t1 4cf8019h 0th rf9084h02 n0r58r03rh`n98g')>

Understood, but arguably that’s no different to any other bug in a dependency specification. It only works accidentally at the moment, and only for consumers using packaging (which agreed, is likely the vast majority of them).

But as I said, I’m not that bothered. If you want to raise a request for the PEP to be changed, go for it. You’ll need to follow the process here and in particular I’d consider it as something that “has the potential to affect software interoperability” - anything that implements its own version comparison rather than using packaging will be impacted[1]. Having said that, I don’t expect it to be controversial - just tedious :slightly_smiling_face:


  1. And the whole point of standardising is so that we don’t assume that everyone uses the canonical library. ↩︎

I’m not too concerned about the process or its tediousness TBH – I first want to see if others think this is meaninfully useful to allow. :slight_smile:

Personally, I’m ambivalent but, if enough people say that “yes, I think this is worth allowing”, I’d be happy to drive this.

Do we have a way to check how many recent releases on PyPI use the now invalid syntax? Is there a way to check this other than downloading all releases from PyPI and inspecting them?

I did a quick query of the JSON data on PyPI (I have an offline copy). It’s not going to be entirely accurate, and my check was extremely superficial, but I found 464 projects on PyPI with metadata that might be impacted. That’s not a huge number - and 120 of those appear to be part of an extra, so are not used by default.

List of problem values
Project name                             Requirement

aiocogeo                                 stac-pydantic>=1.3.*
aioflowdock                              aiohttp>=3.*
aioflowdock                              pyee>=6.*
aiohubot                                 pyee<7,>=6.*
aiohubot                                 aiohttp<4,>=3.*
airflow-ditto                            apache-airflow<=2.*,>=1.10.10
airflow-hdinsight                        apache-airflow<=2.*,>=1.10.10
airium                                   requests<3,>=2.12.*; extra == "parse"
airium                                   beautifulsoup4<5.0,>=4.10.*; extra == "parse"
amnes                                    flake8-bugbear<21,>=20.*; extra == "dev"
amnes                                    mkdocs-material<5.1,>=5.0.*; extra == "dev"
animask                                  opencv-python>=4.1.0.*
animask                                  pygifsicle>=1.0.*
apdaemon                                 psutil>=5.4.*
apispec-starlette                        apispec<=4.*,>=3.3.*
astrool                                  astropy<4.*,>=3.0
astrool                                  plotly<4.*,>=3.0
autogenes                                matplotlib>=3.0.*
autots                                   pandas>=0.25.*
autots                                   statsmodels>=0.10.*
autots                                   scikit-learn>=0.20.*
autots                                   fbprophet>=0.4.*; extra == "additional"
bdacore                                  sphinx>=1.5.*; extra == "test"
bdacore                                  sphinx-bootstrap-theme>=0.4.*; extra == "test"
bdacore                                  nbsphinx>=0.2.*; extra == "test"
bdacore                                  sphinx-gallery>=0.1.*; extra == "test"
binderhub                                kubernetes>=4.*
blog                                     PyYAML>=5.1.*
blog                                     Jinja2>=2.10.*
blog                                     Markdown>=3.1.*
blog                                     Pygments>=2.4.*
blog                                     pymdown-extensions>=6.*
blog                                     pyembed-markdown>=1.1.*
blog                                     docopt>=0.6.*
boml                                     tensorflow<=1.15.*,>=1.13.*
chooser                                  wxPython>=4.0.*
codecomb                                 numpy>=1.16.*
coreir                                   hwtypes>=1.0.*
dagster-gcp                              google-cloud-bigquery<3,>=1.19.*
dantro                                   sphinx-book-theme>=0.3.*; extra == "dev"
dantro                                   sphinx-book-theme>=0.3.*; extra == "doc"
deriver                                  pandas>=1.0.*
dgenies                                  matplotlib>=2.1.*
dgenies                                  Markdown>=2.6.*
dgenies                                  python-crontab==2.*,>=2.2.*
directory-healthcheck                    django<3.3.*,>=1.11.22
django-calaccess-processed-data          django>=4.0.*
django-calaccess-raw-data                django>=4.0.*
django-calaccess-raw-data                django-postgres-copy>=2.6.*
django-calaccess-scraped-data            django>=3.2.*
django-gcloud-connectors                 django<=3.3.*,>=2.0
django-on-chain                          django<=4.0.*,>=2.2.*
django-saml-idp                          Django>=1.9.*
dropbox                                  stone>=2.*
eapt                                     pygeoip>=0.*
eapt                                     pycurl>=7.*.*.*
eapt                                     haversine>=2.*.*
evolutionary-keras                       tensorflow>2.1*
excel2xx                                 mako>=1.1.*
fan-tools                                rest-framework-dyn-serializer>=1.3.*; extra == "doc_utils"
fastapi-plugins                          fastapi>=0.74.*
fastapi-plugins                          tenacity>=8.0.*
fastapi-plugins                          python-json-logger>=2.0.*
fastapi-plugins                          redis>=4.3.*
fastapi-plugins                          aiojobs>=1.0.*
fastapi-plugins                          fastapi>=0.74.*; extra == "all"
fastapi-plugins                          tenacity>=8.0.*; extra == "all"
fastapi-plugins                          python-json-logger>=2.0.*; extra == "all"
fastapi-plugins                          redis>=4.3.*; extra == "all"
fastapi-plugins                          aiojobs>=1.0.*; extra == "all"
fastapi-plugins                          aiomcache>=0.7.*; extra == "all"
fastapi-plugins                          fastapi>=0.74.*; extra == "fakeredis"
fastapi-plugins                          tenacity>=8.0.*; extra == "fakeredis"
fastapi-plugins                          python-json-logger>=2.0.*; extra == "fakeredis"
fastapi-plugins                          redis>=4.3.*; extra == "fakeredis"
fastapi-plugins                          aiojobs>=1.0.*; extra == "fakeredis"
fastapi-plugins                          fakeredis>=1.8.*; extra == "fakeredis"
fastapi-plugins                          fastapi>=0.74.*; extra == "memcached"
fastapi-plugins                          tenacity>=8.0.*; extra == "memcached"
fastapi-plugins                          python-json-logger>=2.0.*; extra == "memcached"
fastapi-plugins                          redis>=4.3.*; extra == "memcached"
fastapi-plugins                          aiojobs>=1.0.*; extra == "memcached"
fastapi-plugins                          aiomcache>=0.7.*; extra == "memcached"
fastapi-plugins                          fastapi>=0.74.*; extra == "test"
fastapi-plugins                          tenacity>=8.0.*; extra == "test"
fastapi-plugins                          python-json-logger>=2.0.*; extra == "test"
fastapi-plugins                          redis>=4.3.*; extra == "test"
fastapi-plugins                          aiojobs>=1.0.*; extra == "test"
fastapi-plugins                          aiomcache>=0.7.*; extra == "test"
fastapi-plugins                          fakeredis>=1.8.*; extra == "test"
feast                                    googleapis-common-protos<2,>=1.52.*
feast                                    PyYAML<7,>=5.4.*
feast                                    dask<2022.02.0,>=2021.*
feast                                    google-cloud-datastore<3,>=2.1.*; extra == "ci"
feast                                    google-cloud-storage<3,>=1.34.*; extra == "ci"
feast                                    google-cloud-datastore<3,>=2.1.*; extra == "dev"
feast                                    google-cloud-storage<3,>=1.34.*; extra == "dev"
feast                                    google-cloud-datastore<3,>=2.1.*; extra == "docs"
feast                                    google-cloud-storage<3,>=1.34.*; extra == "docs"
feast                                    google-cloud-datastore<3,>=2.1.*; extra == "gcp"
feast                                    google-cloud-storage<3,>=1.34.*; extra == "gcp"
firelab                                  tb-nightly>=1.14.*
flange                                   jsonschema>=3.*
fsleyes                                  jinja2>=2.*
fsleyes-props                            fsleyes-widgets>=0.6.*
fslgui                                   nibabel>=2.3.*
gitir-download                           pytest<=4.*; extra == "dev"
gluonts                                  rpy2<3.*,>=2.9.*; extra == "r"
gluonts                                  pyarrow>=6.*; python_version == "3.6.*" and extra == "arrow"
gluonts                                  pyarrow>=6.*; python_version == "3.6.*" and extra == "dev"
gluonts                                  pyarrow>=6.*; python_version == "3.6.*" and extra == "pro"
good-first-issues                        pytest<=4.*; extra == "dev"
hepdata-lib                              PyYAML>4.*
hicexplorer                              numpy>=1.15.*
hicexplorer                              scipy>=1.1.*
hicexplorer                              matplotlib>2.2.*
hicexplorer                              tables>=3.4.*
hicexplorer                              pandas>=0.23.*
hicexplorer                              pyBigWig>=0.3.*
hicexplorer                              six>=1.11.*
hicexplorer                              future>=0.17.*
hicexplorer                              jinja2>=2.10.*
hicexplorer                              unidecode>=1.0.*
hicmatrix                                numpy>=1.16.*
hicmatrix                                scipy>=1.2.*
hicmatrix                                tables>=3.5.*
hicmatrix                                pandas>=0.25.*
hicmatrix                                intervaltree>=3.0.*
httpx-ntlm                               httpx>=0.21.*
i2t2                                     fastscript>=0.1.*
i2t2                                     ipykernel>=5.2.*
i2t2                                     ipython>=7.13.*
i2t2                                     jupyter>=1.0.*
i2t2                                     jupyterlab>=2.0.*
i2t2                                     matplotlib>=3.2.*
i2t2                                     nbdev>=0.2.*
i2t2                                     numpy>=1.18.*
i2t2                                     pandas>=1.0.*
i2t2                                     pillow>=7.1.*
i2t2                                     pydicom>=1.4.*
i2t2                                     twine>=3.1.*
i2t2                                     scipy>=1.4.*
i2t2                                     h5py>=2.10.*
jhub-authenticators                      jupyterhub>=3.*
jhub-swarmspawner                        docker>=5.0.*
jhub-swarmspawner                        jupyterhub>=3.*
jhub-swarmspawner                        flatten-dict>=0.4.*
jhub-swarmspawner                        jhub-authenticators>=0.3.*
jhub-swarmspawner                        traitlets>=5.3.*
json2python-models                       python-dateutil>=2.7.*
json2python-models                       inflection>=0.3.*
json2python-models                       unidecode>=1.0.*
json2python-models                       Jinja2>=2.10.*
json2python-models                       typing-extensions>=3.1.*
ldap-hooks                               ldap3>=2.5.*
ldap-hooks                               traitlets>=5.3.*
ldap-hooks                               gen>=0.*
ldap-hooks                               tornado>=6.0.*
lemay-ai-sidecar                         pandas>=0.24.*
lemay-ai-sidecar                         tensorflow<1.14.*,>=1.11.*
lemay-ai-sidecar                         numpy>=1.16.*
lusidtools                               lusid-sdk-preview>=0.11.*
maccabistats                             setuptools>=28.*
matmov                                   pandas<2.*,>=1.1.2
matmov                                   numpy<2.*,>=1.19.2
matmov                                   matplotlib<4.*,>=3.3.2
mieaa                                    requests>=2.19.*
mip                                      gurobipy>=8.*; extra == "gurobi"
mlx-jira-juggler                         python-dateutil<3.*,>=2.8.0
mlx-jira-juggler                         natsort<8.*,>=7.1.0
mlx-traceability                         Sphinx<6.*,>=2.1
mlx-traceability                         matplotlib<4.*
mlx-warnings                             junitparser<2.*,>=1.0.0
nkocr                                    opencv-contrib-python>=4.*
node-conformity                          numpy>=1.15.*
node-conformity                          tqdm>=4.20.*
nomeroff-net                             numpy>=1.16.*
nomeroff-net                             imgaug>=0.4.*
ocelot-materials                         numpy>=1.*
ocelot-materials                         pandas>=0.24.*
ocelot-materials                         scipy>=1.*
ocelot-molecular                         numpy>=1.*
ocelot-molecular                         pandas>=0.24.*
ocelot-molecular                         scipy>=1.*
ocelot-quantum                           numpy>=1.*
ocelot-quantum                           pandas>=0.24.*
ocelot-quantum                           scipy>=1.*
openergy                                 python-slugify<2.*,>=1.2.1
orange3-imageanalytics                   numpy>=1.16.*
orange3-imageanalytics-dda               numpy>=1.16.*
pjlsa                                    cmmnbuild-dep-manager<3.*,>=2.5.0
pjlsa                                    jpype1<2.*,>=1.0.2
pjlsa                                    cmmnbuild-dep-manager<3.*,>=2.5.0; extra == "all"
pjlsa                                    jpype1<2.*,>=1.0.2; extra == "all"
pjlsa                                    cmmnbuild-dep-manager<3.*,>=2.5.0; extra == "core"
pjlsa                                    jpype1<2.*,>=1.0.2; extra == "core"
pretty-format-json                       data-process>=0.3.*
prospecting                              uritemplate>=3.*
pulsar-client                            protobuf<=3.20.*,>=3.6.1; extra == "all"
pulsar-client                            protobuf<=3.20.*,>=3.6.1; extra == "functions"
pyawsutils                               pytrustplatform>=0.13.*
pyddapi                                  six>=1.10.*
pyddapi                                  psycopg2-binary>=2.6.*
pyddapi                                  nbconvert<=5.4.*; extra == "test"
pyddapi                                  sphinx>=1.5.*; extra == "test"
pyddapi                                  sphinx-bootstrap-theme>=0.4.*; extra == "test"
pyddapi                                  nbsphinx>=0.2.*; extra == "test"
pyddapi                                  sphinx-gallery>=0.1.*; extra == "test"
pygeostat                                matplotlib>=2.2.*
pylabrad                                 pyOpenSSL>=16.2.*
pylabrad                                 pyparsing>=2.1.*
pylabrad                                 twisted>=16.0.*
pylabrad                                 futures>=3.0.*; python_version < "3"
pylabrad                                 numpy>=1.12.*; python_version < "3.7"
pylabrad                                 numpy>=1.15.*; python_version == "3.7"
pylabrad                                 numpy>=1.17.*; python_version >= "3.8"
pyquant-ms                               scipy>=0.18.*
pytest-httpx                             pytest<8.*,>=6.*
pythainlp                                nltk>=3.3.*; extra == "full"
pythainlp                                nltk>=3.3.*; extra == "wordnet"
pytorch-lightning                        torch>=1.9.*
pytorch-lightning                        torchtext>=0.10.*; extra == "all"
pytorch-lightning                        torchvision>=0.10.*; extra == "all"
pytorch-lightning                        torchtext>=0.10.*; extra == "dev"
pytorch-lightning                        torchvision>=0.10.*; extra == "examples"
pytorch-lightning                        torchtext>=0.10.*; extra == "extra"
pytrustplatform                          pykitcommander>=2.3.2*
receptivefield                           pillow>=6.2.*
receptivefield                           matplotlib>=3.1.*
receptivefield                           numpy>=1.17.*
redexpect                                redssh>=2*
redexpect                                redssh>=2*; extra == "docs"
redexpect                                redssh>=2*; extra == "tests"
robotframework-jenkins                   python-jenkins>=1.*
robotframework-jenkins                   robotframework>=3.*
robotframework-jenkins                   requests>=2.*
s3like                                   marshmallow>=3.*
stonky                                   teletype>=1.*
stwfsapy                                 scikit-learn>=0.24.*
superai                                  protobuf<4.*,>=3.20.1; extra == "ai"
superai                                  protobuf<4.*,>=3.20.1; extra == "complete"
superai                                  protobuf<4.*,>=3.20.1; extra == "test"
telluric                                 fiona<2.*,>=1.8.4
telluric                                 shapely<2.*,>=1.6.3
teroshdl                                 yowasp-yosys>=0.8.*
textract                                 extract-msg<=0.29.*
thonny-microbit                          thonny<3.2.*,>=3.0.0
thonny-pi                                thonny<3.2.*,>=3.0.0
uplink-httpx                             httpx>=0.12.*
uplink-httpx                             uplink>=0.9.*
wfuzz                                    pyparsing>=2.4*; python_version >= "3.5"
xkcd-wrapper                             requests<2.25.*,>=2.23.*; extra == "sync"
mistex                                   mistune>=2.0.*
taktile-profiling                        numpy<2,>=1.21.*
fedot                                    ete3>=3.1.*
fedot                                    numpy>=1.19.*
fedot                                    pytest>=6.2.*
fedot                                    anytree>=2.8.*
fedot                                    typing>=3.7.*
fedot                                    Pillow>=8.2.*
fedot                                    seaborn>=0.9.*
fedot                                    joblib>=0.17.*
fedot                                    SALib>=1.3.*
fedot                                    deap>=1.3.*
fedot                                    lightgbm>=2.3.*
fedot                                    catboost>=0.25.*
fedot                                    testfixtures>=6.18.*
fedot                                    requests>=2.*
flask-requestpreprocessor                Flask>=1.1.*
blizz                                    nbsphinx>=0.7.*; extra == "docs"
blizz                                    pytest>=6.2.*; extra == "testing"
blizz                                    pytest-cov>=2.10.*; extra == "testing"
blizz                                    flake8>=3.8.*; extra == "testing"
blizz                                    flake8-comprehensions>=3.3.*; extra == "testing"
blizz                                    isort>=5.7.*; extra == "testing"
lazuli                                   protobuf<4.0.*,>=3.2.*
geotiff                                  zarr>=2.10.*
insolver                                 h2o>=3.36.0.*
insolver                                 django>=3.2.*; python_version >= "3.8"
libpyka                                  beautifulsoup4>=4.9.*
libpyka                                  chardet>=4.*
libpyka                                  cssselect>=1.*
libpyka                                  requests>=2.*
libpyka                                  types-chardet>=0.*
libpyka                                  types-requests>=0.1.*
lmdirect                                 authlib<1.*,>=0.15.5
reqwrapper                               requests>=2.*
endeless                                 numpy>=1.17.*
fastface                                 pytorch-lightning>=1.1.*
fastface                                 scipy>=1.*.*
mlflow-openshift                         mlflow>=1.13.*
mlflow-openshift                         numpy>=1.19.*
mlflow-openshift                         openshift-client>=1.0.*
pychu                                    pydantic<2.*,>=1.7.3
xgbse                                    pandas>=1.0.*
xgbse                                    pandas>=1.0.*; extra == "all"
kelvin-app                               pyarrow>=3.0.*; extra == "tests"
bluewhale3-imageanalytics                numpy>=1.16.*
iotprovision                             pyedbglib>=2.17.*
iotprovision                             pymcuprog>=3.7.*
iotprovision                             pydebuggerupgrade>=3.4.*
iotprovision                             pykitcommander>=2.6.4.*
iotprovision                             pytrustplatform>=0.15.*
iotprovision                             pyawsutils>=0.11.10.*
iotprovision                             pyazureutils>=0.8.*
iotprovision                             pydebuggerconfig>=3.8.3.*
matomo                                   requests>=2.*
aiotodoist                               aiohttp<4.*,>=3.*
aiotodoist                               todoist-python>=8.*
demcompare                               xarray>=0.13.*
metdig                                   meteva>1.3.*
polars                                   pyarrow>=4.0.*; extra == "pyarrow"
polars                                   pyarrow>=4.0.*; extra == "pandas"
pulsar-client-sn                         protobuf<=3.20.*,>=3.6.1; extra == "all"
pulsar-client-sn                         protobuf<=3.20.*,>=3.6.1; extra == "functions"
amazon-textract-helper                   Pillow>=9.2.*
amazon-textract-helper                   PyPDF2>=2.5.*
amazon-textract-overlayer                Pillow>=9.2.*
amazon-textract-overlayer                PyPDF2>=2.5.*
flask-openapi3                           pydantic<2.*,>=1.2
pyaml-env                                PyYAML<=6.*,>=5.*
flask-gatekeeper                         Flask>=1.1.*
bodo                                     pandas<1.5,>=1.3.*
amazon-textract-pipeline-pagedimensions  Pillow>=9.2.*
amazon-textract-pipeline-pagedimensions  PyPDF2>=2.5.*
ccdc-opencivicdata                       Django>=4.0.*
img2text                                 colorama>=0.4.*
nbmanips                                 cloudpickle>=1.6.*
nbmanips                                 click>=7.1.*
nbmanips                                 Pygments>=2.10.*
nbmanips                                 colorama>=0.4.*
nbmanips                                 beautifulsoup4>=4.10.*
ptlflow                                  einops<=0.4.*,>=0.3.0
ptlflow                                  numpy<=1.22.*,>=1.17.0
ptlflow                                  opencv-python<=4.6.*,>=4.0.0.21
ptlflow                                  packaging<=21.*,>=20.0
ptlflow                                  pandas<=1.4.*,>=1.1.0
ptlflow                                  pillow<=9.2.*,>=5.0
ptlflow                                  plotly<=5.9.*,>=5.0.0
ptlflow                                  pytorch-lightning!=1.3.*,!=1.4.*,<=1.6.*,>=1.1.0
ptlflow                                  requests<=2.28.*,>=2.0.0
ptlflow                                  scipy<=1.9.*,>=1.0.0
ptlflow                                  torch<=1.12.*,>=1.8.1
ptlflow                                  torchmetrics<=0.9.*,>=0.2
ptlflow                                  torchvision<=0.13.*,>=0.9.2
ptlflow                                  tqdm<=4.64.*,>=4.41.0
sqlfileexecutor                          psycopg2>=2.*
sqlfileexecutor                          libpyka>=0.*
sqlfileexecutor                          requests>=2.*
sqlfileexecutor                          click>=8.*
fsqlexec                                 psycopg2>=2.*
fsqlexec                                 libpyka>=0.*
fsqlexec                                 requests>=2.*
fsqlexec                                 click>=8.*
django-sca-helper                        django>=2.1.*
medical-shape                            torch>=1.7.*
torchsde                                 numpy>=1.19.*; python_version >= "3.7"
as-ws-wrapper                            zeep>=4.0.*
fortes-webservice-wrapper                zeep>=4.0.*
jntajis-python                           flake8>=3.9.*; extra == "dev"
jntajis-python                           jinja2>=3.*; extra == "dev"
jntajis-python                           pytest>=6.2.*; extra == "dev"
jntajis-python                           sphinx>=4.*; extra == "dev"
jntajis-python                           sphinx-rtd-theme>=0.5.*; extra == "dev"
knarrow                                  importlib-metadata>=3.9.*; python_version != "3.10"
knarrow                                  importlib-metadata<3.9.*; python_version == "3.10"
tables-io                                numpy<1.22.*; python_version < "3.8"
tables-io                                astropy<5.*; python_version < "3.8"
tables-io                                pandas<1.4.*; python_version < "3.8"
amaranth                                 amaranth-yosys>=0.10.*; extra == "builtin-yosys"
cuquantum-python                         cuquantum>=22.7.*
dj-cloud-task                            google-cloud-tasks>=2.5.*
django3-livesync                         watchdog>=2.*
django3-livesync                         tornado>=6.*
dpm360-lightsaber                        scikit-learn>=1.*
dpm360-lightsaber                        pytorch-lightning>=1.*
dpm360-lightsaber                        ray>=1.13.*
fancy-sa-filemodel                       SQLAlchemy>=1.4.*
lightning-nets                           torch>=1.7.*
mlvisualizationtools                     dash-bootstrap-components>=1.0.0*; extra == "dash"
stubgenj                                 JPype1<2.*,>=1.2.1
stubgenj                                 JPype1<2.*,>=1.2.1; extra == "all"
stubgenj                                 JPype1<2.*,>=1.2.1; extra == "core"
actorch                                  ray<2.*,>=1.13.0
cegalprizm-investigator                  cegalprizm-hub<2,>=1.*
cegalprizm-pythontoolpro                 cegalprizm-hub<2,>=1.*
marshmallow-sa-core                      SQLAlchemy>=1.4.0.*
micropipelines                           numpy<=1.22.*
pipkin                                   filelock>=3.0.*
timetracker-cli2                         click>=8.0.*
timetracker-cli2                         dateparser>=1.1.*
timetracker-cli2                         requests>=2.27.*
timetracker-cli2                         rich>=11.1.*
mk-feature-store                         Click>=7.*
mk-feature-store                         PyYAML>=5.4.*
mk-feature-store                         tenacity>=7.*
mk-feature-store                         dask<2022.02.0,>=2021.*
mk-feature-store                         google-cloud-datastore>=2.1.*; extra == "ci"
mk-feature-store                         google-cloud-storage<1.41,>=1.34.*; extra == "ci"
mk-feature-store                         google-cloud-datastore>=2.1.*; extra == "dev"
mk-feature-store                         google-cloud-storage<1.41,>=1.34.*; extra == "dev"
mk-feature-store                         google-cloud-datastore>=2.1.*; extra == "gcp"
mk-feature-store                         google-cloud-storage<1.41,>=1.34.*; extra == "gcp"
cegalprizm-scripting-server              cegalprizm-hub<2,>=1.*
ednaml                                   torch>=1.10.*
ednaml                                   torchvision>=0.11.*
ednaml                                   Pillow>=9.0.*
ednaml                                   tqdm>=4.63.*
elemeno-ai-feast                         PyYAML>=5.4.*
elemeno-ai-feast                         tenacity>=7.*
elemeno-ai-feast                         dask<2022.02.0,>=2021.*
elemeno-ai-feast                         google-cloud-datastore>=2.1.*; extra == "ci"
elemeno-ai-feast                         google-cloud-storage<1.41,>=1.34.*; extra == "ci"
elemeno-ai-feast                         google-cloud-datastore>=2.1.*; extra == "dev"
elemeno-ai-feast                         google-cloud-storage<1.41,>=1.34.*; extra == "dev"
elemeno-ai-feast                         google-cloud-datastore>=2.1.*; extra == "gcp"
elemeno-ai-feast                         google-cloud-storage<1.41,>=1.34.*; extra == "gcp"
finetuning-scheduler                     torch>=1.9.*
gocd-tools                               requests>=2.27.*
gocd-tools                               PyYAML>=5.*
hikari-toolkit                           uncertainties>=3.*
moai-hydra-searchpath-plugin             moai-mdk>=0.1.*
polars-u64-idx                           pyarrow>=4.0.*; extra == "pandas"
polars-u64-idx                           pyarrow>=4.0.*; extra == "pyarrow"
pysequansutils                           pyedbglib>=2.19.3.*
pysequansutils                           pykitcommander>=2.6.3.*
timeeval                                 scikit-learn>=0.24.*
dataloop-upipe                           numpy<=1.22.*
galahad                                  sortedcontainers>==2.4.*
galahad                                  scikit-learn>=0.24.*; extra == "all"
galahad                                  scikit-learn>=0.24.*; extra == "contrib"
galahad                                  scikit-learn>=0.24.*; extra == "sklearn"
mdade                                    torch>=1.7.*
pymoode                                  numpy>=1.19.*
pymoode                                  scipy>=1.7.*
rp-tagger                                sqlalchemy>=1.4.*
rp-tagger                                flask>=1.1.*
rstar                                    pyarrow>=4.0.*; extra == "pandas"
rstar                                    pyarrow>=4.0.*; extra == "pyarrow"
termfactory                              numpy>=1.18.*
transtab                                 pandas>=1.3.*
betty-ml                                 torch>=1.6.*
betty-ml                                 numpy>=1.9.*
geopolars                                pyarrow>=4.0.*; extra == "pandas"
geopolars                                pyarrow>=4.0.*; extra == "pyarrow"
pruby                                    uncertainties>=3.*
scresonators-fit                         matplotlib>=3.0.*
scrfit                                   matplotlib>=3.0.*
torchde                                  torch>=1.7.*
polars-lts-cpu                           pyarrow>=4.0.*; extra == "pyarrow"
polars-lts-cpu                           pyarrow>=4.0.*; extra == "pandas"
togglreports                             requests>=2.25.*
togglreports                             datetime>=2.7.*
togglreports                             argparse>=1.3.*
spd-eda                                  pandas>=1.4.*
spd-eda                                  xlsxwriter>=3.0.*
spd-eda                                  pyodbc>=4.0.*
spd-eda                                  seaborn>=0.12.*
spd-eda                                  scipy>=1.9.*
spd-eda                                  awswrangler>=2.16.*
spyglass-neuro                           jupyterlab>=3.*
spyglass-neuro                           pydotplus>=2.0.*
spyglass-neuro                           pymysql>=1.0.*
torchfl                                  torch<1.12.0,>=1.11.*

Actually, though, it doesn’t look like the current behaviour of packaging is correct anyway:

>>> Version("1.0") in SpecifierSet("<=1.0")
True
>>> Version("1.0") in SpecifierSet("<=1.0.*")
False
>>> Version("1.0") in SpecifierSet("==1.0.*")
True

I don’t think this behaviour is “obvious”, so surely even if we decide do permit the “obvious” equivalent (as @pradyunsg suggested), those projects would be broken anyway (and it would be a silent breakage, which is worse than an easily actionable error message).

2 Likes

What version of packaging are you using? That looks like 21.3 from within pip, if I were to hazard a guess (we’re on 23.0 now).

But regardless, yea, that’s certainly a bit weird. :slight_smile:

Pretty sure you’re not on the latest version.

>>> import packaging
>>> packaging.__version__
'23.0'
>>> from packaging.specifiers import SpecifierSet
>>> SpecifierSet("<=1.0.*")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pradyunsg/Developer/github/pip/.venv/lib/python3.10/site-packages/packaging/specifiers.py", line 711, in __init__
    parsed.add(Specifier(specifier))
  File "/Users/pradyunsg/Developer/github/pip/.venv/lib/python3.10/site-packages/packaging/specifiers.py", line 245, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '<=1.0.*'

Regardless, thanks for checking that @pf_moore – that basically confirms for me that it’s fine for us to just… not bother with this.

That’s right. I was trying to demonstrate that projects would be getting the wrong answers with the previous version of packaging anyway, so there’s no compatibility benefit from allowing this.

1 Like

In a word, I disagree to permit wildcards in comparison specifiers.

According to the legacy implementation, >=X.* and >X.* are equivalent to >=X.0, and <X.* and <=X.* are equivalent to <X.0, in the transition period, a simple normalization function can fix this.

What about a silent normalization and a warning message to notify the users to migrate.

@frostming I’m confused by your reply.

I take this to mean that you wish to disallow comparison specifiers (it’s ambiguous what you’re disagreeing with).

This would imply allowing it, which is contradictory.


Given the use of disagree, I’m gonna assume that you meant that you disagreed with me… and want to permit such specifiers.

We can do that in packaging and we’ve already been printing warnings on such specifiers via the only mechanism that it has for over 2 years (or, erroring out).

packaging can’t be printing user-facing warnings and, given how late we’re catching this, I assume that we’re gonna have to do that in pip as part of migrating to this.

Here’s what I’m going to say… I don’t think there’s a gradient here: it’s either we permit this, or we don’t permit this. The question of migration models is a separate thing that I’d prefer we figure out elsewhere.

I don’t feel strongly about accepting this, and if someone else does, I’m happy to coauthor a tiny PEP to make the case for this. The process is straightforward (PEP 12 is long, but clear about what you need to do).

FTR we are assessing the impact of upgrading packaging from 21 to 23 in Fedora.
Our RPM dependency generators use packaging to parse requirements from installed distribution metadata. We asserted that packages that require things like foo>=1.2.* fail the build.

We have ~3.6k packages with Python dist-info/egg-info. Together they have around the same number of unique requirements.

Will report back when we are done.

1 Like

I don’t think these should be allowed, they cause users confusion over >= 1.0.* vs >= 1.0 and what > 1.0.* even means and PEP 440 is very clear that they are forbidden. I’d rather ban more things from PEP 440 and get a simpler, easier to understand standard.

To add some numbers, I’ve downloaded the pypi requires_dist for all files on pipy (8,175,706 entries) in the following format (which fwiw is partially not the real requires_dist in METADATA):

{"name": "pandas", "version": "1.4.4", "requires_dist": ["python-dateutil (>=2.8.1)", "pytz (>=2020.1)", "numpy (>=1.18.5) ; platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\"", "numpy (>=1.19.2) ; platform_machine == \"aarch64\" and python_version < \"3.10\"", "numpy (>=1.20.0) ; platform_machine == \"arm64\" and python_version < \"3.10\"", "numpy (>=1.21.0) ; python_version >= \"3.10\"", "hypothesis (>=5.5.3) ; extra == 'test'", "pytest (>=6.0) ; extra == 'test'", "pytest-xdist (>=1.31) ; extra == 'test'"], "filename": "pandas-1.4.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"}

regex suggestions welcome, this is just the first thing i came up with:

# legal
$ rg -c '==\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
27452
# all
$ rg -c '(>|<)=?\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
6721
# individual categories
$ rg -c '<=\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
962
$ rg -c '>=\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
5465
$ rg -c '<\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
393
$ rg -c '>\s*\d+(\.\d+)*\.\*' pipy_requires_dist.ndjson
134

That is it affects 6,721 of 8,175,706 files. I unfortunately don’t have the data to check how old these files are, i.e. how many of them are 10+ year old pre-PEP 440 releases.

I’d also be interested in the fedora data, i’d expect that none of them are pre-PEP440 ancient files and are better maintained and should have less matches.

1 Like

Problems from Fedora:

pcs

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/pcs/

python3dist() provides generator:

packaging.version.InvalidVersion: Invalid version: '0.11.4.15-f7301'

pyodbc

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/pyodbc/

python3dist() provides generator:

packaging.version.InvalidVersion: Invalid version: '4.0.0-unsupported'

See Use PEP 440 version name required for wheels · mkleehammer/pyodbc@996fce1 · GitHub

This is a verified bug in the RPM package of pyodbc (the above commit just makes it not manifest in this way).

python-btchip

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-btchip/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    python-pyscard>=1.6.12-4build1; extra == "smartcard"
                  ~~~~~~~~~~^

python-dipy

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-dipy/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected closing RIGHT_PARENTHESIS
    fury (>=0.8.0scikit-learn) ; extra == 'all'
         ~~~~~~~~^

python-dropbox

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-dropbox/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    stone>=2.*
         ~~~^

python-haversion

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-haversion/

python3dist() provides generator:

packaging.version.InvalidVersion: Invalid version: 'main'

This is a verified bug in Fedora’s RPM packaging of pyhaversion.

python-lacrosse

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-lacrosse/

python3dist() provides generator:

packaging.version.InvalidVersion: Invalid version: 'unknown'

This is a verified bug in Fedora’s RPM packaging of pylacrosse.

python-pdir2

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-pdir2/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    typing-extensions>=4.2.*
                     ~~~~~^

This is caused by downstream patching.

python-pvc

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-pvc/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    pyvmomi>=5.5.0-2014.1.1
           ~~~~~~~~~~~~^

python-pytest-httpx

https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/package/python-pytest-httpx/

python3dist() requires generator:

packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    pytest<8.*,>=6.*
          ~~^
2 Likes

A lot of those look like local versions that aren’t using the local version syntax?

1 Like