Can connect to URL with curl but not with requests (i.e. requests ignoring my CA bundle?)

Python Help
1

(Cross-posting from python - can connect to URL with curl but not with requests (i.e. requests ignoring my CA bundle?) - Stack Overflow)

I am able to connect to a certain URL with cURL, after I installed the corresponding SSL certificates:

$ curl -vvvv $MY_URL  # Fails
$ sudo openssl x509 -inform pem -outform pem -in /tmp/custom-cert.pem -out /usr/local/share/ca-certificates/custom-cert.crt
$ sudo update-ca-certificates
$ curl -vvvv $MY_URL  # OK

However, requests (or httpx, or any other library I use) refuses to do so:

In [1]: import os
   ...: import requests
   ...: requests.get(os.environ["MY_URL"])
---------------------------------------------------------------------------
SSLCertVerificationError                  Traceback (most recent call last)
...

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

My understanding is that requests uses certifi and as such these custom certificates are not available here:

In [1]: import certifi

In [2]: certifi.where()
Out[2]: '/tmp/test_ca/.venv/lib/python3.10/site-packages/certifi/cacert.pem'

I have already tried a number of things, like trying to use the system CA bundle:

  • export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt same error
  • requests.get(..., verify="/etc/ssl/certs/ca-certificates.crt") same error
  • switched to httpx + a custom SSL context as explained in the docs, same error
  • attempted truststore as discussed in this httpx issue, same error

How can I make Python (requests, httpx, raw ssl, anything) use the same certificates that cURL is successfully using?

2

Well, of course using verify=False or equivalent works. But I’d like to avoid that if possible.

3

What is your operating system and distribution?

4

Sorry, forgot to add that - latest Linux Mint (Ubuntu-based)

5

curl uses capath on Debian-based system. Did update-ca-certificates add your cert to `` `/etc/ssl/certs/ca-certificates.crt``` ?

6

Yes, a grep confirms that it’s there. That explains why cURL is able to do it, but it doesn’t explain why adding verify=/etc/ssl/certs/ca-certificates.crt to requests.get doesn’t work :question:

7

I added a bounty to the SO question.

Either (1) this is a specific problem of my machine that can’t be reproduced elsewhere, or (2) it can be reproduced. And beyond, (1) this might be a problem with this URL and certificate, or (2) this might be a more general problem of Python’s certifi not reading certain certificates from the specified trust store.

8

Got confirmation from someone else that could make it work, which is the indication I needed to keep digging :slight_smile: