Urllib traceback error

Python Help
1

I am learning python and am very new to the whole thing, learning through a MOOC. This week our lesson was about scraping data from web sources. I cannot even get the most basic thing to work, getting a million traceback errors with this code. Can anyone help me understand what the error(s) is/are?

import urllib.request, urllib.parse, urllib.error

x = urllib.request.urlopen('https://py4e-data.dr-chuck.net/comments_42.html').read()

Here are the traceback errors I get when running this from Terminal
Traceback (most recent call last):
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 1037, in _send_output
self.send(msg)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 975, in send
self.connect()
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py”, line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py”, line 513, in wrap_socket
return self.sslsocket_class._create(
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py”, line 1071, in _create
self.do_handshake()
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py”, line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/Users/ser41/Library/CloudStorage/OneDrive-ThePennsylvaniaStateUniversity/Data Science/Python Michigan/Homework Assignments/Exercise 12.py”, line 4, in
with urllib.request.urlopen(‘Welcome to the comments assignment from www.py4e.com’) as response:
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 216, in urlopen
return opener.open(url, data, timeout)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 519, in open
response = self._open(req, data)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 496, in _call_chain
result = func(*args)
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File “/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py”, line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

1 Like
2

It seems like you would have to provide an SSL certificate in order to receive the response otherwise the request might be considered as a threat.

3

Oh wow, this wasn’t covered in our class – do you have a link on how to do that?

Also, if I remove the s (which I assume has something to do with security), there is still a webpage with the info I need – if I do this, do I still need a certificate?

Thanks so much for your help!

4

No worries.

Removing the s won’t be helpful as the URL doesn’t exist if they use HTTPS.

And yeah, this link might be able to help you out.

2 Likes
5

In this case, fortunately, there is a non-secured website with the info I need – but this is super helpful since I am new to everything!

6

Yeah, that sounds great. Have fun!

7

I had the very same problem and am very grateful to all who have posted here. I’m also learning Python using courses on LinkedIn Learning, and came across the same error.

The Stack Overflow paged linked to another which gave two solutions:

  1. Using finder, navigate to /Applications/Python 3.11 and double click on Install Certificates.command (this worked)
  2. Run pip install certifi (this didn’t work)

In case any other newbies like myself run into the same, that’s what worked. Since I’m using all of this as a learning opportunity, is it possible to run this file from the command line? I couldn’t. The file itself looks like a Python file, but python3 "Install Certificates.command" didn’t do anything.

8

The certifi package updates quite frequently (not too surprising). Basically, anytime I executed apt update <anything> on my Linux laptop, a new version would come along for the ride.

Can you explain what didn’t work about the pip install command? I can think of a few possible causes.

  • You might need to be root.
  • The certifi package was probably already installed, so you’d have to tell pip to update it: pip install --upgrade certifi.
  • You might well not have found pip at all. python -m pip is generally better. Even so, the actual python executable in a Mac package won’t be on your PATH, so you’d have to give the full path to the Python binary.
9 
% pip install certifi
Requirement already satisfied: certifi in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (2022.12.7)

Now, of course, it works. ¯\_(ツ)_/¯
Thank you for the hints, though.

10

Of course. The car is rattling, driving the owner nuts. They take it to the mechanic and the rattle mysteriously disappears. A couple weeks later, it’s back. :wink:

Wait a few weeks and try

pip install -U certifi

If there is a new version, it will be installed.