Pip3 install from requirement file duplicate entries

Packaging
1

Hello Everyone,
I am running pip 20.1.1

From reading the pip documentation it doesn’t state that i am not allowed to have duplicate entries in my requirements file.

pip3 install -r requirements_py3.txt

I get
ERROR: Double requirement given: requests>=2.18.4 (from -r requirements_py3.txt (line 87)) (already in requests==2.23.0 (from -r requirements_py3.txt (line 84)), name=‘requests’)

When i run pip3 install “requests>=2.18.4”

it installs
pip3 list
Package Version

certifi 2020.4.5.2
chardet 3.0.4
idna 2.9
pip 20.1.1
requests 2.24.0
setuptools 41.2.0
urllib3 1.25.9

Can i get a hint about what is happening here i am confused. requests==2.23.0 is greater than 2.18.4 so the requests>=2.18.4 shouldn’t be blocked.

Personally I don’t like the duplicate entries and i would want to remove it. What is going on here?

2

What does your requirements_py3.txt look like?

3

Hello Py,

Thank you for the quick response.

The requirements file i pulled from the repo looks like this.
I see multiple duplicates and my gut feeling is to delete them and run it till it works but I would like to understand why this breaks.

thanks,

2to3==1.0
alabaster==0.7.12
appdirs==1.4.4
appnope==0.1.0
astroid==2.4.1
attrs==19.3.0
autopep8==1.5.3
Babel==2.8.0
backcall==0.1.0
bcrypt==3.1.7
black==19.10b0
boto==2.49.0
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
click==7.1.2
ConfigArgParse==1.2.3
cryptography==2.9.2
decorator==4.4.2
deepdiff==4.3.2
dnspython==1.16.0
docutils==0.16
EasyProcess==0.3
Faker==4.1.0
Flask==1.1.2
Flask-BasicAuth==0.2.0
future==0.18.2
gevent==1.5a3
geventhttpclient==1.4.2
geventhttpclient-wheels==1.3.1.dev2
gitdb==4.0.5
GitPython==3.1.3
greenlet==0.4.16
idna==2.9
imagesize==1.2.0
importlib-metadata==1.6.1
ipdb==0.13.2
ipython==7.15.0
ipython-genutils==0.2.0
isort==4.3.21
itsdangerous==1.1.0
jedi==0.17.0
jenkins==1.0.2
Jinja2==2.11.2
lazy-object-proxy==1.4.3
locustio==0.14.5
lxml==4.5.1
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.2.0
msgpack==1.0.0
mypy==0.770
mypy-extensions==0.4.3
netaddr==0.7.19
ordered-set==4.0.1
packaging==20.3
paramiko==2.4.0
parso==0.7.0
pathspec==0.8.0
pep8==1.7.1
pexpect==4.8.0
pickleshare==0.7.5
pip-review==1.1.0
pluggy==0.13.1
prompt-toolkit==3.0.5
psutil==5.7.0
ptyprocess==0.6.0
py==1.8.1
pyasn1==0.4.8
pycodestyle==2.6.0
pycparser==2.20
Pygments==2.6.1
pylint==2.4.4
PyNaCl==1.3.0
pyparsing==2.4.7
pyreadline==2.1
pytest==5.4.1
python-dateutil==2.6.1
pytz==2019.3
PyVirtualDisplay==0.2.1
PyYAML==5.3.1
pyzmq==19.0.1
regex==2020.4.4
requests==2.23.0
retrying==1.3.3
selenium==3.6.0
requests>=2.18.4
pytest==5.4.1
waiting<=1.4.1
netaddr<=0.7.19
PyVirtualDisplay<=0.2.1
paramiko==2.4.0
Faker<=0.7.11
boto<=2.48.0
python_dateutil<=2.6.1
dnspython<=1.15.0
GitPython==3.1.1
jenkins<=1.0.2
pexpect<=4.2.1
retrying<=1.3.3
PyYAML<=3.12
urllib3>=1.22
boto3==1.4.8
configparser==3.5.0
jira==1.0.10
ijson==2.3
lxml==4.1.1
tabulate==0.8.1
deepdiff==3.3.0
urlparse2==1.1.1
uritools==2.0.0
xunitmerge>=1.0.4
kubernetes==9.0.0
ruamel.yaml==0.16.5
netifaces==0.10.7
pyjwt==1.6.4
pycrypto==2.6.1
pandas==0.23.3
psycopg2==2.8.3
splunk-sdk==1.6.6
beautifulsoup4<=4.7.1
pem==19.2.0
pyarrow==0.16.0
future==0.18.2
hypothesis==5.10.4
cython==0.29.17
pytest-html==2.1.1
six==1.15.0
smmap==3.0.4
snowballstemmer==2.0.0
Sphinx==3.0.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
tabulate==0.8.7
text-unidecode==1.3
toml==0.10.1
traitlets==4.3.3
typed-ast==1.4.1
typing-extensions==3.7.4.2
urllib3==1.25.9
waiting==1.4.1
wcwidth==0.1.9
Werkzeug==1.0.1
wrapt==1.12.1
zipp==3.1.0
zope.event==4.4
zope.interface==5.1.0
4

I’m not entirely sure. My guess is that pinned a dependency == is a constraint to use a specific version. Any other duplicate entry is either redundant or violates that constraint. There are experts here that can advise more. It’s likely safe to remove one of the duplicates.

Keep in mind, there is a new dependency resolver which will have new behavior in future versions of pip. It may be worth deleting duplicates and moving on.

5

I suspect pip is being explicit/conservative in saying that while the two constraints agree in total, individually they don’t agree (i.e. one would allow requests==2.18.4 and the other one wouldn’t). And since having two rules is probably typically a user error, it’s easier/clearer to just say, “nope, you need to be very clear which constraint you want applied”.

2 Likes
6

Hi Brett,
Thank you for responding. This helps out alot!