I am having issues when I try to import requests and beautifulsoup into my script. I have created a virtual environment in this path: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv and have installed the two libraries here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Lib\site-packages while my script is here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts. The IDE I am using is VSCode. As far as I have understood (I’m new to Python), the libraries should import, since they are in the same virtual environment. The Python interpreter is in the sys.path: C:\Users\eldet\AppData\Local\Programs\Python\Python311, the requests version is 2.31.0, bs4 is 0.0.1 and beautigulsoup4, 4.12.2. When debugging, I get the Module not found error.
Hi, did you install all the libraries?
Hi! I have installed requests and beautifulsoup. Is there something more I should install in order to be able to import them to my script?
Look at the line of code that the traceback points to for the error.
Post the traceback if you are not sure where to look.
You should see an import statement at that line and you will need to install
the package that is referred to.
That line of code may not be in the code you have written.
You should have all the source code installed so it is usually not
a problem to see where the error is.
Yeah, this is my issue. As mentioned, I have requests and beautifulsoup installed in \venv\Lib\site-packages. My script is in \venv\Scripts. As far as I understand, the libraries should import, being in the same virtual environment as the script. Nevertheless, I get a ModuleNotFoundError. Here’s the traceback:
Traceback (most recent call last):
File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\google_maps_scraper.py", line 78, in <module>
main()
File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\google_maps_scraper.py", line 10, in main
import requests
ModuleNotFoundError: No module named 'requests'
It probably should not be. This directory is for scripts that are put there by the installers for the libraries that you are using.
Also, you should not try to make the virtual environment inside the Python installation folder itself. Put it somewhere else convenient, such as on the desktop.
Most importantly, though: make sure that the virtual environment is activated before giving the command to run the script. Just because your code is inside a sub-folder of some particular Python installation does not mean that that Python installation will be used to run the code. In your case, there are already at least two: the user installation of Python 3.11, and the venv that you created based off of that. If you use the command line to run the code, use the corresponding activation script first. If you run the code from VSCode, you will need to understand how to choose the interpreter - I do not use this IDE (or any other), so I cannot tell you how this works.
This already confirms the problem. You installed the libraries for the virtual environment’s Python (as you should - this is the reason to use virtual environments!), but the code is running with the original Python.
To use a venv in VS Code, you can either run the activate.ps1 script from the terminal, or change the active interpreter from the menu. In the bottom-right corner you should see the language (e.g. MagicPython) and the version (3.11.1 64-bit); click on the second one and it will open the interpreter picker. The terminal will start each line with venv while it is the active interpreter.
Thanks, but I am not sure if I have followed your instructions well. Here’s what I did:
-
I have left the python interpreter in C:\Users\eldet\AppData\Local\Programs\Python\Python311 and have deleted venv to get out of the virtual environment?
-
I have moved my script to C:\Users\eldet\Documents\Python Scripts\Web Scraping
-
I have created a new virtual environment here: C:\Users\eldet\Documents\python_virtual_environments and have installed the libraries.
-
I have activated the virtual environment and then I have run the script.
Unfortunately, this is the traceback:
Traceback (most recent call last):
File "c:\Users\eldet\Documents\Python Scripts\Web Scraping\google_maps_scraper.py", line 78, in <module>
main()
File "c:\Users\eldet\Documents\Python Scripts\Web Scraping\google_maps_scraper.py", line 10, in main
import requests
ModuleNotFoundError: No module named 'requests'
I obviously, must be doing something wrong.
Thank you for your reply. I have done all this and I am using the correct interpreter. The problem is that the libraries are not importing and I am obviously doing something wrong. I have followed the instructions that @kknechtel has provided, but I still cannot get the libraries imported.
To use a venv you MUST run the python.exe that is in the venv and not any other.
One way is to run the python.exe in the venv.
I’ll create a venv
: 20:17:34.54 C:\Users\barry> py -3 -m venv example
Install requests into it
: 20:18:17.47 C:\Users\barry> example\Scripts\python.exe -m pip install requests
Collecting requests
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.7/96.7 kB 5.4 MB/s eta 0:00:00
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1
Downloading urllib3-2.0.3-py3-none-any.whl (123 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 7.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB 9.2 MB/s eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.31.0 urllib3-2.0.3
[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: C:\Users\barry\example\Scripts\python.exe -m pip install --upgrade pip
Now run python and import requests:
: 20:18:38.08 C:\Users\barry> example\Scripts\python.exe
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
:>>> import requests
:>>> requests
<module 'requests' from 'C:\\Users\\barry\\example\\Lib\\site-packages\\requests\\__init__.py'>
:>>>
Another way is to activate the venv.
20:20:41.86 C:\Users\barry> example\Scripts\activate.bat
(example) : 20:20:48.55 C:\Users\barry> python
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
:>>> import requests
:>>> requests
<module 'requests' from 'C:\\Users\\barry\\example\\Lib\\site-packages\\requests\\__init__.py'>
:>>>
To be able to comment on this,
… we would need to see exactly how you attempted these parts - step by step, the exact commands used (and/or actions taken in the VSCode GUI).
Unless, of course, the instructions from @barry-scott are enough for you to see what you did differently and what’s wrong with it 
Thank you for your interest.
This is what I am doing:
C:\Users\eldet\maps_scraper>pip install requests
Requirement already satisfied: requests in c:\users\eldet\maps_scraper\lib\site-packages (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (2023.5.7)
C:\Users\eldet\maps_scraper>install bs4
'install' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\eldet\maps_scraper>pip insatll bs4
ERROR: unknown command "insatll" - maybe you meant "install"
C:\Users\eldet\maps_scraper>pip install bs4
Collecting bs4
Using cached bs4-0.0.1-py3-none-any.whl
Requirement already satisfied: beautifulsoup4 in c:\users\eldet\maps_scraper\lib\site-packages (from bs4) (4.12.2)
Requirement already satisfied: soupsieve>1.2 in c:\users\eldet\maps_scraper\lib\site-packages (from beautifulsoup4->bs4) (2.4.1)
Installing collected packages: bs4
Successfully installed bs4-0.0.1
C:\Users\eldet\maps_scraper>python
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
>>>
What does pip --version say?
The pip version is 23.1.2
What hapens if you explicity run:
c:\users\eldet\maps_scraper\bin\python
(instead of just “python”) and try the import then?
This is to check that you’re running the “python” from the venv, since
the requeusts module (etc) is installed in the venv.
Cheers,
Cameron Simpson cs@cskk.id.au
Thank you for your reply. The correct path to where packages and script are located would be C:\Users\eldet\maps_scraper\Lib\site-packages. Since maps_scraper is my venv, I would firstly activate it in order to run python from there. Here’s what occurs and where my issue lies:
(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>python
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> google_maps_scraper.py
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'google_maps_scraper' is not defined
>>> exit()
(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>dir
Volume in drive C is Windows-SSD
Volume Serial Number is D2DD-4186
Directory of C:\Users\eldet\maps_scraper\Lib\site-packages
24/06/2023 23:14 <DIR> .
22/06/2023 08:32 <DIR> ..
22/06/2023 08:36 <DIR> beautifulsoup4-4.12.2.dist-info
22/06/2023 08:36 <DIR> bs4
24/06/2023 12:27 <DIR> bs4-0.0.1.dist-info
22/06/2023 08:34 <DIR> certifi
22/06/2023 08:34 <DIR> certifi-2023.5.7.dist-info
24/06/2023 22:33 <DIR> cffi
24/06/2023 22:33 <DIR> cffi-1.15.1.dist-info
22/06/2023 08:34 <DIR> charset_normalizer
22/06/2023 08:34 <DIR> charset_normalizer-3.1.0.dist-info
22/06/2023 08:32 151 distutils-precedence.pth
24/06/2023 22:33 <DIR> file
24/06/2023 22:33 <DIR> file-0.3.0.dist-info
24/06/2023 21:51 2,596 google_maps_scraper.py
22/06/2023 08:34 <DIR> idna
22/06/2023 08:34 <DIR> idna-3.4.dist-info
22/06/2023 08:32 <DIR> pip
22/06/2023 08:32 <DIR> pip-23.1.2.dist-info
22/06/2023 08:32 <DIR> pkg_resources
24/06/2023 22:33 <DIR> pycparser
24/06/2023 22:33 <DIR> pycparser-2.21.dist-info
24/06/2023 23:14 <DIR> pyflakes
24/06/2023 23:14 <DIR> pyflakes-3.0.1.dist-info
24/06/2023 12:26 <DIR> requests
24/06/2023 12:26 <DIR> requests-2.31.0.dist-info
22/06/2023 08:32 <DIR> setuptools
22/06/2023 08:32 <DIR> setuptools-65.5.0.dist-info
22/06/2023 08:36 <DIR> soupsieve
22/06/2023 08:36 <DIR> soupsieve-2.4.1.dist-info
22/06/2023 08:34 <DIR> urllib3
22/06/2023 08:34 <DIR> urllib3-2.0.3.dist-info
24/06/2023 22:33 181,760 _cffi_backend.cp311-win_amd64.pyd
22/06/2023 08:32 <DIR> _distutils_hack
24/06/2023 22:07 <DIR> __pycache__
3 File(s) 184,507 bytes
32 Dir(s) 844,862,267,392 bytes free
That is not the way to run your script.
cd to where your google_maps_scraper.py then do this after activating the venv.
(maps_scraper) python google_maps_scraper.py
You do not need to install google_maps_scraper.py into site-packages.
I have been doing that but this is what happens:
(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>python google_maps_scraper.py
(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>
Try adding a print to the file so that you can see that it is doing something.
print('At start of google_maps_scraper.py')
Now you should see that message print out.
I suspect that you have not added a main function to your code that is run.
For example here an example:
import requests
def do_something():
print('Doing something')
# add your code here
# run the code
do_something()
Or the fuller version:
import sys
import requests
def do_something(argv):
print('Doing something')
print('Command line args: %r' % (argv[1:],))
# add your code here
# run the code with the command args
if __name__ == '__main__':
sys.exit(do_something(sys.argv)
Ok, so I have added the print and it does print. I have also added a logger but the log file is empty. Here is the script as I have modified it:
import logging
logging.basicConfig(filename="google_maps_scraper.log", level=logging.INFO)
import requests
import bs4
import sys
def get_restaurants(query):
"""Gets a list of restaurants from Google Maps.
Args:
query: The query to search for.
Returns:
A list of restaurant objects. Each restaurant object has the following properties:
name: The name of the restaurant.
address: The address of the restaurant.
phone_number: The phone number of the restaurant.
website: The website of the restaurant.
rating: The rating of the restaurant.
reviews: The number of reviews for the restaurant.
"""
url = "https://www.google.com/maps/search/mexican+restaurants+in+athens,+greece/@37.971871,23.7168781,12z/data=!3m1!4b1?entry=ttu".format(query)
response = requests.get(url)
soup = bs4.BeautifulSoup(response.content, "html.parser")
restaurants = []
for restaurant in soup.find_all("div", class_="section-result"):
try:
name = restaurant.find("div", class_="section-result__title").text.strip()
except AttributeError:
name = None
try:
address = restaurant.find("div", class_="section-result__address").text.strip()
except AttributeError:
address = None
try:
phone_number = restaurant.find("div", class_="section-result__phone").text.strip()
except AttributeError:
phone_number = None
try:
website = restaurant.find("div", class_="section-result__website").text.strip()
except AttributeError:
website = None
try:
rating = restaurant.find("div", class_="section-result__rating").text.strip()
except AttributeError:
rating = None
try:
reviews = restaurant.find("div", class_="section-result__reviews").text.strip()
except AttributeError:
reviews = None
restaurant_info = {
"name": name,
"address": address,
"phone_number": phone_number,
"website": website,
"rating": rating,
"reviews": reviews,
}
logger.info(str(restaurant_info))
restaurants.append(restaurant_info)
return restaurants
def main():
"""This is the main action of the script.
It calls the `get_restaurants()` function to get a list of restaurants from Google Maps.
It then prints the list of restaurants.
"""
query = sys.argv[1]
restaurants = get_restaurants(query)
print("The script is running.")
if __name__ == "__main__":
main()