SSLCertVerificationError with Python3 on Ubuntu 20.04

I’ve been attempting to run an ansible playbook that uses the module /usr/lib/python3/dist-packages/ansible/modules/get_url.py to download this file: https://raw.githubusercontent.com/devture/matrix-synapse-shared-secret-auth/1.0.2/shared_secret_authenticator.py, however I keep running into the following error:

Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

I’ve been trying to troubleshoot this error with the maintainer of the Ansible playbook, but have hit a wall and can’t seem to figure out what is causing this error. Any ideas how I might fix this issue would be much appreciated.

So far some things I’ve tried and additional information:

If I run ls -l $(which python3), I get the result /usr/bin/python3 -> python3.8

anible --version is:

ansible [core 2.11.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xthursdayx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/xthursdayx/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
  jinja version = 2.10.1
  libyaml = True

I’ve reinstalled python3. python3-pip, ansible, openssl, python3-certifi, and ca-certificates.

Running curl https://raw.githubusercontent.com/devture/matrix-synapse-shared-secret-auth/1.0.2/shared_secret_authenticator.py works and results in the contents of the file as expected.

If I run curl --insecure -vvI https://raw.githubusercontent.com/devture/matrix-synapse-shared-secret-auth/1.0.2/shared_secret_authenticator.py 2>&1 | awk 'BEGIN { cert=0 } /^\* Server certificate:/ { cert=1 } /^\*/ { if (cert) print }' and I get the following output:

* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=www.github.com
*  start date: May  6 00:00:00 2020 GMT
*  expire date: Apr 14 12:00:00 2022 GMT
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b0d6d5aec0)
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* Connection #0 to host raw.githubusercontent.com left intact

When I run pip3 install --upgrade certifi, but get this output:

Requirement already up-to-date: certifi in /usr/local/lib/python3.8/dist-packages (2021.10.8)

The result of echo 'import certifi; print(certifi.__version__);' | python3 is: 2021.10.08, so seemingly the most up-to-date version.

Running apt search python3-certif gives the result python3-certifi/focal,focal,now 2019.11.28-1 all [installed,automatic]

If I launch python3 and run the following lines of code:

import urllib.request
url = 'https://raw.githubusercontent.com/devture/matrix-synapse-shared-secret-auth/1.0.2/shared_secret_authenticator.py'
f = urllib.request.urlopen(url)
print(f.read().decode('utf-8'))

I get the output:

>>> import urllib.request
>>> url = 'https://raw.githubusercontent.com/devture/matrix-synapse-shared-secret-auth/1.0.2/shared_secret_authenticator.py'
>>> f = urllib.request.urlopen(url)
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 1421, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

Then:

>>> print(f.read().decode('utf-8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'f' is not defined
>>>

Python’s standard library does not use certifi on Linux. It uses the default trust store that is provided by your operating system. Almost all Linux distributions ship a root CA certificates or ensure that root CA certs are installed with OpenSSL and Python. The Debian and Ubuntu family are the only distros that deviate and therefore break Python.

I filed bugs for Debian and Ubuntu a while ago and eventually got an answer why Debian doesn’t ship CA certificates by default (quoting a Debian maintainer):

And historically Debian has tried to avoid admitting subservince to the CA cartel. So, there are political minefields there.

The situation will improve in the future. There is some effort to ensure that future versions of Debian’s and Ubuntu’s Python will work as designed. Existing releases like Ubuntu 20.04 haven’t seen any fix, though.

In the mean time you have to manually install the ca-certificates package on all your Debian and Ubuntu hosts. Once Debian and Ubuntu fall in line, we will no longer need the certifi package on Linux, too. It’s a problematic hack anyway.

I’m sorry for your inconvenience.

3 Likes

Thanks a lot for the information!

Strangely, I actually already have ca-certificates installed on this VPS.

If I run apt search ca-certificates I get the following output:

ca-certificates/focal-updates,focal-security,focal-updates,focal-security,now 20210119~20.04.2 all [installed]
  Common CA certificates

ca-certificates-java/focal,focal 20190405ubuntu1 all
  Common CA certificates (JKS keystore)

ca-certificates-mono/focal,focal 6.8.0.105+dfsg-2 all
  Common CA certificates (Mono keystore)

liblwp-protocol-https-perl/focal,focal 6.07-2ubuntu2 all
  HTTPS driver for LWP::UserAgent

I tried reinstalling ca-certificates just to see if that might fix things, but I’m still getting the same error.

Any suggestions for what else I might try in order to figure what where the problem is coming from? I already reinstalled Python(3) and OpenSSL, but I can try to fully remove them and then reinstall if that might help…