ImportError: cannot import name 'DEPRECATED_SERVICE_NAMES' from 'botocore.docs'

Python Help
1

Hello everyone, I have a python script that pip install the following dependencies:

dependencies = [
#“pyspark”,
“python-dateutil~=2.8.1”,
“pydantic==1.6.1”,
“pandas”,
“pyarrow==1.0.1”,
“pymongo”,
“sqlparse”,
“scipy”,
“scikit-learn”,
“requests”,
“urllib3<=1.26.15”,
“pyod”,
“xgboost”,
“shap”,
“ray”,
“boto3==1.33.13”,
“botocore==1.33.13”,
“s3fs”
]

since I need to call the invoke_model function provided by bedrock, I need to import botocore.
However, I have noticed that keeping the same code, the first time I execute it I always get:

ImportError: cannot import name ‘DEPRECATED_SERVICE_NAMES’ from ‘botocore.docs’ (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages/botocore/docs/init.py)

Whereas everything works fine the second time i run it, and this pattern repeats on and on.
I have tried to install boto3 and botocore simultaneously as suggested from other users on the Internet, but it didn’t solve. What could be the problem?
Actually i am using python 3.7.
Thanks in advance!

2

up, i haven’t solved it yet

3

Welcome to the forums.

Is the pip command run from the same script that then tries to do the import? Does it maybe check for the existence of some module before it goes for pip?

What this reminds me of is the behaviour of Jupyter notebooks where, if you pip install something, you usually have to restart in order for it to be picked up as an available module. As this is inconvenient, I assume it’s something they can’t avoid. :man_shrugging:

I think it is generally true that if you import a name and it fails, Python remembers the module is not present (even if it later is).

1 Like
4

Explicitly install compatible versions of boto3 and botocore

pip install boto3==1.33.13 botocore==1.33.13

Sometimes other packages in your dependencies list might force conflicting versions of boto3 or botocore. Use pipdeptree to check for dependency conflicts.

pip install pipdeptree
pipdeptree

Python 3.7 is reaching end-of-life, and some libraries may behave inconsistently with it. If feasible, upgrade to Python 3.8 or 3.9 for better compatibility.

1 Like
5

3.7 has been EOL for more than a year and 3.8 has been for a month. I’d suggest 3.12 if possible.

1 Like
6

A lot of the libraries aren’t compatible with 3.12

1 Like
7

That’s why I said “if possible” :slight_smile: in general I’d go for the most recent version for which all dependencies are available.

2 Likes
8

As far as I can tell, pyspark and xgboost are supported up to Python 3.11 and the others are supported up to more recent versions, so that looks like a good one to try.

3 Likes
9

Isn’t that what the OP is doing already? I’m assuming the list exhibited is effectively a pip requirements file.

Yes to installing a more modern Python, however.

2 Likes
10

Thank you all for the answers!
Actually, I don’t have the possibility to move to a newer Python version, I need somehow to make it work on Python 3.7. The dependencies exhibited are not in a requirements file, but instead are part of the python script, which pip install them by doing:

for package in dependencies:
    if hasattr(pip, 'main'):
        pip.main(['install', package])
    else:
        pip._internal.main(['install', package])

Now, I have noticed that I get the error when I go to import boto3:

ImportError: cannot import name ‘DEPRECATED_SERVICE_NAMES’ from ‘botocore.docs’

when in the dependencies installation logs I have:

Requirement already satisfied: six>=1.5 in ./myenv/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.27.60,>=1.27.59->aiobotocore~=2.4.2->s3fs) (1.16.0)
Requirement already satisfied: zipp>=0.5 in ./myenv/lib/python3.7/site-packages (from importlib-metadata->attrs>=17.3.0->aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (3.15.0)
WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
Installing collected packages: botocore
  Attempting uninstall: botocore
    WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
    Found existing installation: botocore 1.33.13
    Uninstalling botocore-1.33.13:
      Successfully uninstalled botocore-1.33.13
WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
s3transfer 0.8.2 requires botocore<2.0a.0,>=1.33.2, but you have botocore 1.27.59 which is incompatible.
boto3 1.33.13 requires botocore<1.34.0,>=1.33.13, but you have botocore 1.27.59 which is incompatible.
Successfully installed botocore-1.27.59

Instead, everything seems to work fine when in the logs i have:

Collecting botocore==1.33.13
  Using cached botocore-1.33.13-py3-none-any.whl (11.8 MB)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./myenv/lib/python3.7/site-packages (from botocore==1.33.13) (1.0.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in ./myenv/lib/python3.7/site-packages (from botocore==1.33.13) (1.26.15)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./myenv/lib/python3.7/site-packages (from botocore==1.33.13) (2.8.2)
Requirement already satisfied: six>=1.5 in ./myenv/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.33.13) (1.16.0)
WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
Installing collected packages: botocore
  Attempting uninstall: botocore
    WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
    WARNING: No metadata found in ./myenv/lib/python3.7/site-packages
    Found existing installation: botocore 1.27.59
    Can't uninstall 'botocore'. No files were found to uninstall.
WARNING: Ignoring invalid distribution -otocore (/mnt/c/Users/myuser/Desktop/dqse/myenv/lib/python3.7/site-packages)
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
aiobotocore 2.4.2 requires botocore<1.27.60,>=1.27.59, but you have botocore 1.33.13 which is incompatible.
Successfully installed botocore-1.33.13

So if I manage to avoid installing botocore-1.27.59 and I keep botocore-1.33.13, I guess everything would work fine. But what else can I do more than specifying “boto3==1.33.13”, “botocore==1.33.13”,in the dependencies?

1 Like
11

That’s a helpful discovery.

It looks like something may be wrong with the use of pip here, such that it is losing the "b" from "botocore", or in the lead-up to that. I would check exactly what is being passed in the part of the script you show. (Print out ['install', package].) Maybe it’s garbling other things too. In the second log file it looks like this invalid name has got into your site-packages cache.

A virtual env is good for this kind of thing and keeps your global installation clean. (Clean that out first.)

At bottom though, it appears pip is unable to resolve dependency versions that meet all your constraints. This bit tells you so:

I’m not sure why it starts that way, though, as you specify. python-dateutil~=2.8.1 which I thought meant 2.8.*. Irrespective of that puzzle, I think you have to pin fewer things.

2 Likes
12

Thank you Jeff, actually it is working if I execute the script locally.
I managed to solve by removing all the folders related to “otocore” from dqse/myenv/lib/python3.7/site-packages and by changing the dependencies list to:

dependencies = [
    #"pyspark",
    "python-dateutil>=2.8.2",
    "pydantic==1.6.1",
    "pandas",
    "pyarrow==1.0.1",
    "pymongo",
    "sqlparse",
    "scipy",
    "scikit-learn",
    "requests",
    "urllib3<=1.26.15",
    "pyod",
    "xgboost",
    "shap",
    "ray",
    "s3fs",
    "boto3==1.33.13",
    "botocore==1.33.13"
]

However, running the script with this new list of dependencies on my emr cluster (always Python 3.7), I am getting always the error:

ImportError: cannot import name ‘DEPRECATED_SERVICE_NAMES’ from ‘botocore.docs’

In this case I have noticed that I don’t have any warning related to “otocore”, but in the logs I have:

Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib64/python3.7/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.9.4)
Collecting botocore<1.27.60,>=1.27.59
  Using cached botocore-1.27.59-py3-none-any.whl (9.1 MB)
Requirement already satisfied: aioitertools>=0.5.1 in /usr/local/lib/python3.7/site-packages (from aiobotocore~=2.4.2->s3fs) (0.11.0)
Requirement already satisfied: wrapt>=1.10.10 in /usr/local/lib64/python3.7/site-packages (from aiobotocore~=2.4.2->s3fs) (1.16.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/site-packages (from attrs>=17.3.0->aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (6.7.0)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (3.10)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from botocore<1.27.60,>=1.27.59->aiobotocore~=2.4.2->s3fs) (1.0.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.7/site-packages (from botocore<1.27.60,>=1.27.59->aiobotocore~=2.4.2->s3fs) (1.26.15)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/site-packages (from botocore<1.27.60,>=1.27.59->aiobotocore~=2.4.2->s3fs) (2.9.0.post0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->attrs>=17.3.0->aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (3.15.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.27.60,>=1.27.59->aiobotocore~=2.4.2->s3fs) (1.13.0)
Installing collected packages: botocore
  Attempting uninstall: botocore
    Found existing installation: botocore 1.33.13
    Uninstalling botocore-1.33.13:
      Successfully uninstalled botocore-1.33.13
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

s3transfer 0.8.2 requires botocore<2.0a.0,>=1.33.2, but you'll have botocore 1.27.59 which is incompatible.
boto3 1.33.13 requires botocore<1.34.0,>=1.33.13, but you'll have botocore 1.27.59 which is incompatible.
Successfully installed botocore-1.27.59
Running pip with arguments: ['install', 'boto3==1.33.13']
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: Running pip install with root privileges is generally not a good idea. Try `dqse_ad_job.py install --user` instead.
Requirement already satisfied: boto3==1.33.13 in /usr/local/lib/python3.7/site-packages (1.33.13)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3==1.33.13) (1.0.0)
Requirement already satisfied: botocore<1.34.0,>=1.33.13 in /usr/local/lib/python3.7/site-packages (from boto3==1.33.13) (1.33.13)
WARNING: No metadata found in /usr/local/lib/python3.7/site-packages
Requirement already satisfied: s3transfer<0.9.0,>=0.8.2 in /usr/local/lib/python3.7/site-packages (from boto3==1.33.13) (0.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4; python_version < "3.10" in /usr/local/lib/python3.7/site-packages (from botocore<1.34.0,>=1.33.13->boto3==1.33.13) (1.26.15)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/site-packages (from botocore<1.34.0,>=1.33.13->boto3==1.33.13) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.13->boto3==1.33.13) (1.13.0)
Running pip with arguments: ['install', 'botocore==1.33.13']
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: Running pip install with root privileges is generally not a good idea. Try `dqse_ad_job.py install --user` instead.
Requirement already satisfied: botocore==1.33.13 in /usr/local/lib/python3.7/site-packages (1.33.13)
WARNING: No metadata found in /usr/local/lib/python3.7/site-packages
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from botocore==1.33.13) (1.0.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4; python_version < "3.10" in /usr/local/lib/python3.7/site-packages (from botocore==1.33.13) (1.26.15)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/site-packages (from botocore==1.33.13) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.33.13) (1.13.0)
Traceback (most recent call last):
  File "/mnt/tmp/spark-dxxxxxxxx-b436-4f47-xxxx-fb8axxxxxe92/dqse_ad_job.py", line 55, in <module>
    import boto3
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 17, in <module>
    from boto3.session import Session
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 25, in <module>
    from .resources.factory import ResourceFactory
  File "/usr/local/lib/python3.7/site-packages/boto3/resources/factory.py", line 17, in <module>
    from ..docs import docstring
  File "/usr/local/lib/python3.7/site-packages/boto3/docs/__init__.py", line 15, in <module>
    from botocore.docs import DEPRECATED_SERVICE_NAMES
ImportError: cannot import name 'DEPRECATED_SERVICE_NAMES' from 'botocore.docs' (/usr/local/lib/python3.7/site-packages/botocore/docs/__init__.py)

So, it seems that everything works fine when botocore 1.33.13 is installed, but not when it decides to install botocore 1.27.59. Furthermore, also in this case I have a situation in which the first time that I run the script, it gives the error, whereas the second time I run it, it works fine.

What else can i try?

1 Like
13

I haven’t seen pip used as a library before. Have you read this: User Guide - pip documentation v24.3.1 ?

pip is giving you lots of warnings that it is struggling to do what you asked. Also, you’re calling pip repeatedly to install one dependency at a time, which means it never gets to think about the problem as a whole. Is it possible to supply all this in one command line?

Although what you have isn’t used as a requirements file, it is very much like one. It might be easier to experiment with the requirements in the form of an actual requirements file.

2 Likes
14

Thank you!
I solved by modyfing the .sh file in order to install the dependencies directly during the bootstrapping phase of the emr cluster.

1 Like