"SSL: CERTIFICATE_VERIFY_FAILED" error on Python 3.9.6 (Windows 10)

Python Help
1

Hi!
I’m new with Python and I have been following some tutorials and based on one of them I have the following piece of code:

from scrapy import Selector
from urllib.request import urlopen

html = urlopen("https://www.pythonparatodos.com.br/formulario.html")
sel = Selector(text = html.read())
lista = sel.xpath('//input[@type="text"]')
print(lista)
for selector in lista:
    print(selector)

The URL is easily accessed through a browser but when I run the code I got the following errors:

C:\Users\username\Envs\webcrawler_part_ii\Scripts\python.exe C:/Users/username/PycharmProjects/webcrawler_part_ii/aula44_scrapy_lxml_exemplos.py
Traceback (most recent call last):
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1257, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1303, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1252, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1012, in _send_output
    self.send(msg)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 952, in send
    self.connect()
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1426, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\username\PycharmProjects\webcrawler_part_ii\aula44_scrapy_lxml_exemplos.py", line 5, in <module>
    html = urlopen("https://www.pythonparatodos.com.br/formulario.html")
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>

Process finished with exit code 1

I’ve found some possible solutions on google to this with certifi or with the two lines of code below but none of them worked. Can someone help me on this?

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

Thanks in advance!

2

Well, this is all strange. I am new to Python and have not used urllib before or done any scraping. But I know a thing or two about HTTP and certificates.

Guesses about the certificate error

I do not get the error that you are getting. I get another one. But let’s start with the one that you report. I’ve checked on the certificate, and as you correctly note it is valid. But it is has a short validity period. The certificate that I am seeing is valid from April 1, 2022 to June 30, 2022. So perhaps you should just try again.

If trying again gives you the same error, you may also wish to check that your Python environment knows the correct date. So include something like

from datetime import date

print(date.today())

to see that the environment in which python is running on your system knows what day it is.

The error I am seeing

I am not getting a certificate error, but I am getting an HTTP 406. It turns out that the web server is picky about what web clients it will talk to. So I told python to say it is Mozilla.

from urllib import request

br_site = "https://www.pythonparatodos.com.br/formulario.html"
req = request.Request(br_site,
                        headers={'User-Agent': 'Mozilla'})
http = request.urlopen(req)

This first sets up the request to provide the HTTP header 'User-Agent:'Mozilla', and then calls urlopen` on that request.

3

Hi @jpgoldberg
Thank you for the reply.
Regarding the date that my python environment sees, there is nothing wrong about that.
Regarding the ‘Mozilla’ header, did that work on your side? I try that but I’m getting the same certificate error message:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>

I also tried with Chrome but to no avail…

4

I am also getting the same error “SSL: CERTIFICATE_VERIY_FAILED” [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129). Did you solve that error ?

5

It’s a Let’s Encrypt certificate. These used to be signed with a certificate chain that expired:

Windows is supposed to use the new chain, but apparently sometimes something can mess up and the old one gets used. This can be one reason for this (confusing in that situation) error.

Maybe try comparing the Windows certificate store with some other computer that doesn’t have the problem.

6

I got this error today and fixed it.

requests.get('https://github.com', verify='/ca/file/path')

More info here : 5 Ways to fix SSL: CERTIFICATE_VERIFY_FAILED in Python - howtouselinux