How are the openssl binaries built for python's windows x64 installers?

I am running python 3.9.1 for windows 10 (x64). I have noticed the openssl dll files shipped with python (libcrypto-1_1.dll & libssl-1_1.dll) are only version 1.1.1.7. I would therefore like to replace these with the latest versions. I would be really grateful if someone on the core development team could let me know how openssl is built for python’s x64 windows installers so that I can replicate and update python’s openssl as soon as as security update is released.

I have been able to successfully build openssl 1.1.1.9 in windows 10 following the instructions in the release. I am using Microsoft visual studio 2019 + strawberry perl + nasm. I ran the following config command using the x64 native tools command prompt for vs 2019:

perl Configure VC-WIN64A

I then run the nmake command. This produces the two dlls I need: libcrypto-1_1-x64.dll and libssl-1_1-x64.dll

I renamed both files to remove the “-x64” at the end of the name and then used them to replace the existing dlls in the python installation.

When I try to import the ssl library in python, I get the error:

ImportError: DLL load failed while importing _ssl: The specified module could not be found.

I then built openssl for x86, using the x86 native tools command prompt for vs 2019 and the command:

perl Configure VC-WIN32

I then ran nmake which produced: libcrypto-1_1.dll and libssl-1_1.dll

I replaced the dlls in the python installation with those. When I try and import the ssl library in python I get:

ImportError: DLL load failed while importing _ssl: %1 is not a valid Win32 application.

I am reasonably certain I need the x64 version of the dlls because (1) my python installation is x64 (2) when I used the visual studio dumpbin.exe tool on the python shipped dlls to inspect the headers they say “x64” in them (3) the x64 dlls are a similar size to the ones shipped with python and the x86 versions are smaller.

I was once able to do this following the above steps for an older version of python. That version shipped with openssl 1.1.3 (could be 4) and I built the dlls for 1.1.6 (could be 7) and replaced them and python had no issues. Is it possible that the default configuration scripts have changed over these versions such that the default windows x64 configuration no longer builds openssl in a way that works with python and I therefore need to use custom settings?

I think the info at section 1.3 of this page may be relevant, but if it is I am not sure how to implement that in the openssl build process.

1 Like

Have a look at cpython/prepare_ssl.bat at v3.9.1 · python/cpython · GitHub, which should at least get you pointed in the right direction, if not just do what you want :). Note that the version used lives in cpython/get_externals.bat at v3.9.1 · python/cpython · GitHub and cpython/python.props at v3.9.1 · python/cpython · GitHub, and they’ll need to agree. You may or may not also need to rebuild the _ssl and _hashlib modules, see _hashlib.vcxproj and _ssl.vcxproj in the same directory.

2 Likes

You probably won’t have to change _ssl or _hashlib if you’re building a similar version, one of the advantages of using DLLs rather than statically linking.

However, you may need to patch one file in OpenSSL, which will be the bit that looks for its current executable to load the function table (either in uplink.c or applink.c, I forget right now). In CPython, this table is in _ssl.pyd, so we have a little patch that looks there as well. cpython/openssl.vcxproj at 1e5d33e9b9b8631b36f061103a30208b206fd03a · python/cpython · GitHub

2 Likes

Steve makes an excellent point here.

As long as you do not use any fancy config options, you can just swap out libcrypto and libssl with a newer version. OpenSSL patch releases are ABI backwards compatible. Some options affect ABI, e.g. OpenSSL builds without TLS 1.0.

2 Likes

Many thanks for all the info. I will try and get this working and then post my steps for the benefit of other readers.

It would be great if you could automatically update the underlying openssl version using a pip command or something similar. That would decouple the act of updating openssl from updating the python version itself. openssl security updates can then be delivered as soon as they are available without needing to wait until a new python release, and users can update openssl without being forced to update their python version.

1 Like

Following on from the info provided by Zachary, Steve and Christian above, I have now been able to update OpenSSL on my windows x64 python installation.

The scripts used by CPython to build OpenSSL actually runs the command:

Perl Configure VC-WIN64A-masm no-asm

instead of

Perl Configure VC-WIN64A .

After running the correct command on the setup described in my original post i was able to swap out the dlls and python accepted them without issue.

ssl.OPENSSL_VERSION now outputs ‘OpenSSL 1.1.1i 8 Dec 2020’.

1 Like

Perhaps, but it would require us volunteers signing up to handle a much broader test matrix than we really want to.

If you need this, it’s possible to do it yourself, or there are companies that provide support for CPython who can offer it. You can even cut some corners by pulling our builds out of a newer release, which should be fine, but we don’t “support” it.

1 Like

Fair enough. Perhaps the windows installers should come with a warning instead.

1 Like

People who would know what the warning means or could do anything about it wouldn’t need the warning. We already announce updates as prominently as we can.

Ultimately, the installers are a convenience and not the “product” itself. They are totally sufficient for development use, and many production uses, but if you need a “properly” managed runtime for your environment, we don’t provide that (though many of us do provide it for our employers - that kind of support is something people should be happy to pay for).

1 Like

Agreed

I take a different view here. Python is the entry language for many new developers. They will not necessarily be aware that python uses OpenSSL under the hood and will expect their hand to be held at least on issues of security. If the windows binaries come with a compiled extension/artifact that relates to security, then either security updates for it should be taken care of or there should be an express warning that they are not, and possibly a link to a blog article on what can be done about it (i’m happy to help writing one if necessary). As far as i’m aware (and please correct me if I am wrong), a security update to OpenSSL will not trigger an update to python. The next python release will include the latest version in the windows binaries, but that may be quite sometime after an OpenSSL security update is released. In my opinion (and this is constructive suggestion rather than a criticism) windows users should be warned of this. The warning would essentially be:

  1. python uses OpenSSL to make encrypted connections;
  2. the windows installers come with a fixed version of OpenSSL bundled in;
  3. OpenSSL release security updates from time to time;
  4. the windows installation will make no attempt to update the version of OpenSSL it uses;
  5. whilst the latest python installers will include the latest version of OpenSSL at the time that version of python was released, an update to OpenSSL will not trigger an update to python and therefore you may be running the latest security release of python but still be running a vulnerable version of OpenSSL.
  6. You can manually update OpenSSL yourself, but we do not support this. See this blog here…

The installer from the Windows Store is our recommended default (though not, unfortunately, on the python.org website, as some people oppose the idea of linking to a “*.microsoft.com” page). That version will automatically update for any releases we make.

Many people still recommend avoiding this, so if you’d like to write a post pointing out the value in automatic updates, feel free :slight_smile:

On occasion the update will trigger a Python update, but we do it responsibly (i.e. check what was actually fixed in OpenSSL and determine whether it impacts “normal”/commonly used functionality in Python). Many critical security fixes apply to areas that aren’t exposed by the Python APIs, and so we don’t normally consider a faster update.

And of course, once a Python version is in source-only release mode, it’s probably not getting a binary release anyway. We have done them on rare occasion, such as 3.7.9, but again, that was determined based on the impact of the vulnerability and not the mere existence of one.

As for the warning you propose, turn it into a couple of sentences and I’m sure there’s a place for it somewhere in our docs. But I’m not hugely concerned - I don’t think it needs to be a highly prominent warning, given most people cannot/will not act on it, and even the 3. Using Python on Windows — Python 3.9.1 documentation doc page is largely ignored, though it would be the best place for the message. I doubt we’d link to a blog from the docs though, and we don’t want the link mechanism to become a public/supported API, so that part will likely stay (officially) undocumented.

2 Likes