How to download an https:// URL in Windows 10?

Python Help
1

I am trying to use the following code to download a simple https:// URL in Windows 10:

import ssl
from urllib.request import urlopen
urlopen('https://xkcd.com/', context=ssl.create_default_context())

But that code crashes with:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>

This error appears to indicate that Python couldn’t validate the SSL certificate from the xkcd.com domain.

I believe ssl.create_default_context() (used above) tells Python to use the same certificate authorities as elsewhere on Windows 10. In a different application (Edge) I can successfully visit xkcd.com so it appears that the system certificate authorities match the certificate provided by that domain. So I’m not sure why Python can’t validate the domain’s certificate.

How can I alter the above code to use the Windows 10 certificate authorities and not give a certificate validation error?

2

This is very easy with requests (or even urllib3) instead of urllib, you just need to pip install requests at the command line (something already installed it in my global Python so I didn’t use a venv). Both handle SSL verification automatically for you. This worked just now in Windows 11.

import requests
resp = requests.get('https://www.xkcd.com')
resp.text

Your code does work for me as it is too (in Python 3.11. Perhaps my security settings are lax e.g. dev mode). Do you have a specific requirement for urllib, or to use a special ssl context?

You can play with SSLContext.load_default_certs() and other methods too, to try and find where they are on your system.

3

I’m actually using the lower-level HTTPSConnection (from http.client) in my production code, which requires an SSLContext (just like urlopen does).

So the following would be a more accurate snippet that doesn’t work for me on Windows 10†:

from http.client import HTTPSConnection
import ssl
conn = HTTPSConnection('xkcd.com:443', context=ssl.create_default_context())
conn.request('GET', '/')  # fails with: ssl.SSLCertVerificationError

I haven’t tried the above snippet on Windows 11 yet, but I can whip up a VM this evening to check.

† Specifically I’m running the above code on a fresh Windows 10 in a VM with all OS updates applied and the clock set correctly

4

Cheers. That version still works for me out of the box.

If you need to run it in a VM anyway, can you run it in Windows Subsystem for Linux, or is this code for regular Windows users?

5

My Windows 7 system certificates contains the CA cert for xkcd.com but my Windows 10 system certificates do NOT:

xkcd certificate (and related CA certificate)

Screen Shot 2023-08-31 at 7.15.45 PM
Screen Shot 2023-08-31 at 7.15.45 PM1047×967 64.3 KB

Windows 7 and 10 system CA certificates

Screen Shot 2023-08-31 at 9.38.28 PM
Screen Shot 2023-08-31 at 9.38.28 PM922×616 138 KB

Screen Shot 2023-08-31 at 9.24.56 PM
Screen Shot 2023-08-31 at 9.24.56 PM1876×716 99.6 KB

Baffling.

Unsurprisingly, certifi contains the CA cert for xkcd.com:

certifi CA certificates

Screen Shot 2023-08-31 at 9.19.08 PM
Screen Shot 2023-08-31 at 9.19.08 PM1880×1202 337 KB

However I don’t want to rely on certifi since any version of certifi I statically bundle with my app will become out of date after a few years and I want old versions of the app to keep running for a long time.

My app is bundled as a native Windows .exe using py2exe so I don’t think I can use Windows Subsystem for Linux.

6

I wondered how to check those - thanks for that. The GlobalSign cert is there on my Win 11:

import ssl

ctx = ssl.create_default_context()

print(', '.join(c['serialNumber'] 
                for c in ctx.get_ca_certs() 
                if 'Global' in repr(c)
                )
     )

04000000000121585308A2,

Did you miss a Windows update from MS that pushed out extra certs?

What version of libssl-1_1.dll have you got in the DLLs folder in your sys.path? Mine’s 1.1.1.21

Is this a problem with your machine or OS only, at the end of the day? If your users’ machines have recent Windows installations like mine, your code will work for them.

7

I hope so! This is just my Windows 10 testing VM, initially created a year or two ago.

I verified that my Windows 10 VM thinks it is up to date. But still no xkcd.com certificate:

Windows 10 fully updated

Screen Shot 2023-09-01 at 8.37.18 AM
Screen Shot 2023-09-01 at 8.37.18 AM2072×580 110 KB

Retry certificate check. Fails.

Screen Shot 2023-09-01 at 8.55.04 AM
Screen Shot 2023-09-01 at 8.55.04 AM1870×540 40.4 KB

Missing: 04000000000121585308A2

I’ll try rebuilding my Windows 10 VM from scratch from Microsoft’s official evaluation image…

In the meantime, I’ve finally got a fresh Windows 11 VM provisioned, so I’ll check there too…

8

Looks like libssl-1_1.dll is 1.1.14 for me, although it is for Python 3.9 rather than Python 3.11 (which is what I think you’re using):

libssl-1_1.dll Info

Screen Shot 2023-09-01 at 9.04.53 AM
Screen Shot 2023-09-01 at 9.04.53 AM834×992 34.1 KB

By the way, thank you @JamesParrott for helping me debug this far!

1 Like
9

The above works for me on a fresh Windows 11 VM and a fresh Windows 10 VM :tada:

I guess my takeaway is that it’s important to provide a way for the user to extend the system certificates if they get messed up.

I discovered that various programs inspect environment variables to find additional CA certs:

  • $SSL_CERT_FILE - OpenSSL
  • $REQUESTS_CA_BUNDLE - requests
  • $CURL_CA_BUNDLE - curl

So I’ll alter my app to recognize one/more of those as well.

1 Like