Trouble importing libraries

I am having issues when I try to import requests and beautifulsoup into my script. I have created a virtual environment in this path: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv and have installed the two libraries here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Lib\site-packages while my script is here: C:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts. The IDE I am using is VSCode. As far as I have understood (I’m new to Python), the libraries should import, since they are in the same virtual environment. The Python interpreter is in the sys.path: C:\Users\eldet\AppData\Local\Programs\Python\Python311, the requests version is 2.31.0, bs4 is 0.0.1 and beautigulsoup4, 4.12.2. When debugging, I get the Module not found error.

Hi, did you install all the libraries?

Hi! I have installed requests and beautifulsoup. Is there something more I should install in order to be able to import them to my script?

Look at the line of code that the traceback points to for the error.
Post the traceback if you are not sure where to look.

You should see an import statement at that line and you will need to install
the package that is referred to.
That line of code may not be in the code you have written.

You should have all the source code installed so it is usually not
a problem to see where the error is.

Yeah, this is my issue. As mentioned, I have requests and beautifulsoup installed in \venv\Lib\site-packages. My script is in \venv\Scripts. As far as I understand, the libraries should import, being in the same virtual environment as the script. Nevertheless, I get a ModuleNotFoundError. Here’s the traceback:

Traceback (most recent call last):
  File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\", line 78, in <module>
  File "c:\Users\eldet\AppData\Local\Programs\Python\Python311\venv\Scripts\", line 10, in main
    import requests
ModuleNotFoundError: No module named 'requests'

It probably should not be. This directory is for scripts that are put there by the installers for the libraries that you are using.

Also, you should not try to make the virtual environment inside the Python installation folder itself. Put it somewhere else convenient, such as on the desktop.

Most importantly, though: make sure that the virtual environment is activated before giving the command to run the script. Just because your code is inside a sub-folder of some particular Python installation does not mean that that Python installation will be used to run the code. In your case, there are already at least two: the user installation of Python 3.11, and the venv that you created based off of that. If you use the command line to run the code, use the corresponding activation script first. If you run the code from VSCode, you will need to understand how to choose the interpreter - I do not use this IDE (or any other), so I cannot tell you how this works.

This already confirms the problem. You installed the libraries for the virtual environment’s Python (as you should - this is the reason to use virtual environments!), but the code is running with the original Python.

To use a venv in VS Code, you can either run the activate.ps1 script from the terminal, or change the active interpreter from the menu. In the bottom-right corner you should see the language (e.g. MagicPython) and the version (3.11.1 64-bit); click on the second one and it will open the interpreter picker. The terminal will start each line with venv while it is the active interpreter.

Thanks, but I am not sure if I have followed your instructions well. Here’s what I did:

  • I have left the python interpreter in C:\Users\eldet\AppData\Local\Programs\Python\Python311 and have deleted venv to get out of the virtual environment?

  • I have moved my script to C:\Users\eldet\Documents\Python Scripts\Web Scraping

  • I have created a new virtual environment here: C:\Users\eldet\Documents\python_virtual_environments and have installed the libraries.

  • I have activated the virtual environment and then I have run the script.

Unfortunately, this is the traceback:

Traceback (most recent call last):
  File "c:\Users\eldet\Documents\Python Scripts\Web Scraping\", line 78, in <module>
  File "c:\Users\eldet\Documents\Python Scripts\Web Scraping\", line 10, in main
    import requests
ModuleNotFoundError: No module named 'requests'

I obviously, must be doing something wrong.

Thank you for your reply. I have done all this and I am using the correct interpreter. The problem is that the libraries are not importing and I am obviously doing something wrong. I have followed the instructions that @kknechtel has provided, but I still cannot get the libraries imported.

To use a venv you MUST run the python.exe that is in the venv and not any other.

One way is to run the python.exe in the venv.

I’ll create a venv

: 20:17:34.54 C:\Users\barry> py -3 -m venv example

Install requests into it

: 20:18:17.47 C:\Users\barry> example\Scripts\python.exe -m pip install requests
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.7/96.7 kB 5.4 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.0.3-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 7.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB 9.2 MB/s eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.31.0 urllib3-2.0.3

[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: C:\Users\barry\example\Scripts\python.exe -m pip install --upgrade pip

Now run python and import requests:

: 20:18:38.08 C:\Users\barry> example\Scripts\python.exe
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
:>>> import requests
:>>> requests
<module 'requests' from 'C:\\Users\\barry\\example\\Lib\\site-packages\\requests\\'>

Another way is to activate the venv.

 20:20:41.86 C:\Users\barry> example\Scripts\activate.bat

(example) : 20:20:48.55 C:\Users\barry> python
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
:>>> import requests
:>>> requests
<module 'requests' from 'C:\\Users\\barry\\example\\Lib\\site-packages\\requests\\'>

To be able to comment on this,

… we would need to see exactly how you attempted these parts - step by step, the exact commands used (and/or actions taken in the VSCode GUI).

Unless, of course, the instructions from @barry-scott are enough for you to see what you did differently and what’s wrong with it :slight_smile:

Thank you for your interest.

This is what I am doing:

C:\Users\eldet\maps_scraper>pip install requests
Requirement already satisfied: requests in c:\users\eldet\maps_scraper\lib\site-packages (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\eldet\maps_scraper\lib\site-packages (from requests) (2023.5.7)

C:\Users\eldet\maps_scraper>install bs4
'install' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\eldet\maps_scraper>pip insatll bs4
ERROR: unknown command "insatll" - maybe you meant "install"

C:\Users\eldet\maps_scraper>pip install bs4
Collecting bs4
  Using cached bs4-0.0.1-py3-none-any.whl
Requirement already satisfied: beautifulsoup4 in c:\users\eldet\maps_scraper\lib\site-packages (from bs4) (4.12.2)
Requirement already satisfied: soupsieve>1.2 in c:\users\eldet\maps_scraper\lib\site-packages (from beautifulsoup4->bs4) (2.4.1)
Installing collected packages: bs4
Successfully installed bs4-0.0.1

Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

What does pip --version say?

The pip version is 23.1.2

What hapens if you explicity run:


(instead of just β€œpython”) and try the import then?

This is to check that you’re running the β€œpython” from the venv, since
the requeusts module (etc) is installed in the venv.

Cameron Simpson

Thank you for your reply. The correct path to where packages and script are located would be C:\Users\eldet\maps_scraper\Lib\site-packages. Since maps_scraper is my venv, I would firstly activate it in order to run python from there. Here’s what occurs and where my issue lies:

(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>python
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'google_maps_scraper' is not defined
>>> exit()

(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>dir
 Volume in drive C is Windows-SSD
 Volume Serial Number is D2DD-4186

 Directory of C:\Users\eldet\maps_scraper\Lib\site-packages

24/06/2023  23:14    <DIR>          .
22/06/2023  08:32    <DIR>          ..
22/06/2023  08:36    <DIR>          beautifulsoup4-4.12.2.dist-info
22/06/2023  08:36    <DIR>          bs4
24/06/2023  12:27    <DIR>          bs4-0.0.1.dist-info
22/06/2023  08:34    <DIR>          certifi
22/06/2023  08:34    <DIR>          certifi-2023.5.7.dist-info
24/06/2023  22:33    <DIR>          cffi
24/06/2023  22:33    <DIR>          cffi-1.15.1.dist-info
22/06/2023  08:34    <DIR>          charset_normalizer
22/06/2023  08:34    <DIR>          charset_normalizer-3.1.0.dist-info
22/06/2023  08:32               151 distutils-precedence.pth
24/06/2023  22:33    <DIR>          file
24/06/2023  22:33    <DIR>          file-0.3.0.dist-info
24/06/2023  21:51             2,596
22/06/2023  08:34    <DIR>          idna
22/06/2023  08:34    <DIR>          idna-3.4.dist-info
22/06/2023  08:32    <DIR>          pip
22/06/2023  08:32    <DIR>          pip-23.1.2.dist-info
22/06/2023  08:32    <DIR>          pkg_resources
24/06/2023  22:33    <DIR>          pycparser
24/06/2023  22:33    <DIR>          pycparser-2.21.dist-info
24/06/2023  23:14    <DIR>          pyflakes
24/06/2023  23:14    <DIR>          pyflakes-3.0.1.dist-info
24/06/2023  12:26    <DIR>          requests
24/06/2023  12:26    <DIR>          requests-2.31.0.dist-info
22/06/2023  08:32    <DIR>          setuptools
22/06/2023  08:32    <DIR>          setuptools-65.5.0.dist-info
22/06/2023  08:36    <DIR>          soupsieve
22/06/2023  08:36    <DIR>          soupsieve-2.4.1.dist-info
22/06/2023  08:34    <DIR>          urllib3
22/06/2023  08:34    <DIR>          urllib3-2.0.3.dist-info
24/06/2023  22:33           181,760 _cffi_backend.cp311-win_amd64.pyd
22/06/2023  08:32    <DIR>          _distutils_hack
24/06/2023  22:07    <DIR>          __pycache__
               3 File(s)        184,507 bytes
              32 Dir(s)  844,862,267,392 bytes free

That is not the way to run your script.

cd to where your then do this after activating the venv.

(maps_scraper) python

You do not need to install into site-packages.

I have been doing that but this is what happens:

(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>python

(maps_scraper) C:\Users\eldet\maps_scraper\Lib\site-packages>

Try adding a print to the file so that you can see that it is doing something.

print('At start of')

Now you should see that message print out.

I suspect that you have not added a main function to your code that is run.

For example here an example:

import requests

def do_something():
    print('Doing something')
    # add your code here

# run the code

Or the fuller version:

import sys
import requests

def do_something(argv):
    print('Doing something')
    print('Command line args: %r' % (argv[1:],))
    # add your code here

# run the code with the command args
if __name__ == '__main__':

Ok, so I have added the print and it does print. I have also added a logger but the log file is empty. Here is the script as I have modified it:

import logging

logging.basicConfig(filename="google_maps_scraper.log", level=logging.INFO)

import requests
import bs4
import sys

def get_restaurants(query):
    """Gets a list of restaurants from Google Maps.

        query: The query to search for.

        A list of restaurant objects. Each restaurant object has the following properties:
            name: The name of the restaurant.
            address: The address of the restaurant.
            phone_number: The phone number of the restaurant.
            website: The website of the restaurant.
            rating: The rating of the restaurant.
            reviews: The number of reviews for the restaurant.

    url = ",+greece/@37.971871,23.7168781,12z/data=!3m1!4b1?entry=ttu".format(query)
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.content, "html.parser")

    restaurants = []
    for restaurant in soup.find_all("div", class_="section-result"):
            name = restaurant.find("div", class_="section-result__title").text.strip()
        except AttributeError:
            name = None
            address = restaurant.find("div", class_="section-result__address").text.strip()
        except AttributeError:
            address = None
            phone_number = restaurant.find("div", class_="section-result__phone").text.strip()
        except AttributeError:
            phone_number = None
            website = restaurant.find("div", class_="section-result__website").text.strip()
        except AttributeError:
            website = None
            rating = restaurant.find("div", class_="section-result__rating").text.strip()
        except AttributeError:
            rating = None
            reviews = restaurant.find("div", class_="section-result__reviews").text.strip()
        except AttributeError:
            reviews = None

        restaurant_info = {
            "name": name,
            "address": address,
            "phone_number": phone_number,
            "website": website,
            "rating": rating,
            "reviews": reviews,


    return restaurants

def main():
    """This is the main action of the script.

    It calls the `get_restaurants()` function to get a list of restaurants from Google Maps.
    It then prints the list of restaurants.

    query = sys.argv[1]
    restaurants = get_restaurants(query)

    print("The script is running.")

if __name__ == "__main__":