How are the openssl binaries built for python's windows x64 installers?

Core Development
1

I am running python 3.9.1 for windows 10 (x64). I have noticed the openssl dll files shipped with python (libcrypto-1_1.dll & libssl-1_1.dll) are only version 1.1.1.7. I would therefore like to replace these with the latest versions. I would be really grateful if someone on the core development team could let me know how openssl is built for python’s x64 windows installers so that I can replicate and update python’s openssl as soon as as security update is released.

I have been able to successfully build openssl 1.1.1.9 in windows 10 following the instructions in the release. I am using Microsoft visual studio 2019 + strawberry perl + nasm. I ran the following config command using the x64 native tools command prompt for vs 2019:

perl Configure VC-WIN64A

I then run the nmake command. This produces the two dlls I need: libcrypto-1_1-x64.dll and libssl-1_1-x64.dll

I renamed both files to remove the “-x64” at the end of the name and then used them to replace the existing dlls in the python installation.

When I try to import the ssl library in python, I get the error:

ImportError: DLL load failed while importing _ssl: The specified module could not be found.

I then built openssl for x86, using the x86 native tools command prompt for vs 2019 and the command:

perl Configure VC-WIN32

I then ran nmake which produced: libcrypto-1_1.dll and libssl-1_1.dll

I replaced the dlls in the python installation with those. When I try and import the ssl library in python I get:

ImportError: DLL load failed while importing _ssl: %1 is not a valid Win32 application.

I am reasonably certain I need the x64 version of the dlls because (1) my python installation is x64 (2) when I used the visual studio dumpbin.exe tool on the python shipped dlls to inspect the headers they say “x64” in them (3) the x64 dlls are a similar size to the ones shipped with python and the x86 versions are smaller.

I was once able to do this following the above steps for an older version of python. That version shipped with openssl 1.1.3 (could be 4) and I built the dlls for 1.1.6 (could be 7) and replaced them and python had no issues. Is it possible that the default configuration scripts have changed over these versions such that the default windows x64 configuration no longer builds openssl in a way that works with python and I therefore need to use custom settings?

I think the info at section 1.3 of this page may be relevant, but if it is I am not sure how to implement that in the openssl build process.

1 Like
2

Have a look at cpython/prepare_ssl.bat at v3.9.1 · python/cpython · GitHub, which should at least get you pointed in the right direction, if not just do what you want :). Note that the version used lives in cpython/get_externals.bat at v3.9.1 · python/cpython · GitHub and cpython/python.props at v3.9.1 · python/cpython · GitHub, and they’ll need to agree. You may or may not also need to rebuild the _ssl and _hashlib modules, see _hashlib.vcxproj and _ssl.vcxproj in the same directory.

2 Likes
3

You probably won’t have to change _ssl or _hashlib if you’re building a similar version, one of the advantages of using DLLs rather than statically linking.

However, you may need to patch one file in OpenSSL, which will be the bit that looks for its current executable to load the function table (either in uplink.c or applink.c, I forget right now). In CPython, this table is in _ssl.pyd, so we have a little patch that looks there as well. cpython/openssl.vcxproj at 1e5d33e9b9b8631b36f061103a30208b206fd03a · python/cpython · GitHub

2 Likes
4

Steve makes an excellent point here.

As long as you do not use any fancy config options, you can just swap out libcrypto and libssl with a newer version. OpenSSL patch releases are ABI backwards compatible. Some options affect ABI, e.g. OpenSSL builds without TLS 1.0.

2 Likes
5

Many thanks for all the info. I will try and get this working and then post my steps for the benefit of other readers.

It would be great if you could automatically update the underlying openssl version using a pip command or something similar. That would decouple the act of updating openssl from updating the python version itself. openssl security updates can then be delivered as soon as they are available without needing to wait until a new python release, and users can update openssl without being forced to update their python version.

1 Like
6

Following on from the info provided by Zachary, Steve and Christian above, I have now been able to update OpenSSL on my windows x64 python installation.

The scripts used by CPython to build OpenSSL actually runs the command:

Perl Configure VC-WIN64A-masm no-asm

instead of

Perl Configure VC-WIN64A .

After running the correct command on the setup described in my original post i was able to swap out the dlls and python accepted them without issue.

ssl.OPENSSL_VERSION now outputs ‘OpenSSL 1.1.1i 8 Dec 2020’.

1 Like
7

Perhaps, but it would require us volunteers signing up to handle a much broader test matrix than we really want to.

If you need this, it’s possible to do it yourself, or there are companies that provide support for CPython who can offer it. You can even cut some corners by pulling our builds out of a newer release, which should be fine, but we don’t “support” it.

1 Like
8

Fair enough. Perhaps the windows installers should come with a warning instead.

1 Like
9

People who would know what the warning means or could do anything about it wouldn’t need the warning. We already announce updates as prominently as we can.

Ultimately, the installers are a convenience and not the “product” itself. They are totally sufficient for development use, and many production uses, but if you need a “properly” managed runtime for your environment, we don’t provide that (though many of us do provide it for our employers - that kind of support is something people should be happy to pay for).

1 Like
10

Agreed

I take a different view here. Python is the entry language for many new developers. They will not necessarily be aware that python uses OpenSSL under the hood and will expect their hand to be held at least on issues of security. If the windows binaries come with a compiled extension/artifact that relates to security, then either security updates for it should be taken care of or there should be an express warning that they are not, and possibly a link to a blog article on what can be done about it (i’m happy to help writing one if necessary). As far as i’m aware (and please correct me if I am wrong), a security update to OpenSSL will not trigger an update to python. The next python release will include the latest version in the windows binaries, but that may be quite sometime after an OpenSSL security update is released. In my opinion (and this is constructive suggestion rather than a criticism) windows users should be warned of this. The warning would essentially be:

  1. python uses OpenSSL to make encrypted connections;
  2. the windows installers come with a fixed version of OpenSSL bundled in;
  3. OpenSSL release security updates from time to time;
  4. the windows installation will make no attempt to update the version of OpenSSL it uses;
  5. whilst the latest python installers will include the latest version of OpenSSL at the time that version of python was released, an update to OpenSSL will not trigger an update to python and therefore you may be running the latest security release of python but still be running a vulnerable version of OpenSSL.
  6. You can manually update OpenSSL yourself, but we do not support this. See this blog here…
11

The installer from the Windows Store is our recommended default (though not, unfortunately, on the python.org website, as some people oppose the idea of linking to a “*.microsoft.com” page). That version will automatically update for any releases we make.

Many people still recommend avoiding this, so if you’d like to write a post pointing out the value in automatic updates, feel free :slight_smile:

On occasion the update will trigger a Python update, but we do it responsibly (i.e. check what was actually fixed in OpenSSL and determine whether it impacts “normal”/commonly used functionality in Python). Many critical security fixes apply to areas that aren’t exposed by the Python APIs, and so we don’t normally consider a faster update.

And of course, once a Python version is in source-only release mode, it’s probably not getting a binary release anyway. We have done them on rare occasion, such as 3.7.9, but again, that was determined based on the impact of the vulnerability and not the mere existence of one.

As for the warning you propose, turn it into a couple of sentences and I’m sure there’s a place for it somewhere in our docs. But I’m not hugely concerned - I don’t think it needs to be a highly prominent warning, given most people cannot/will not act on it, and even the 4. Using Python on Windows — Python 3.12.1 documentation doc page is largely ignored, though it would be the best place for the message. I doubt we’d link to a blog from the docs though, and we don’t want the link mechanism to become a public/supported API, so that part will likely stay (officially) undocumented.

2 Likes
12

Hi Patrick/Steve/Zachary/Team,

Need Help on python build/link with different openssl

I am right in your situation now.
We have requirement to enable fips in our application

so we have specific 1.0.2 which is fips based, we have built the 1.0.2 openssl code using
perl Configure VC-WIN64a fips --with-fipsdir=path to the above openssl-fips location

Two libs got generated libeay32.dll and ssleay32.dll

Now we need to use Python 3.9.2 which by default comes with 1.1.1i openssl libs.

So our aim is to link python 3.9.2 with openssl 1.0.2 so that we are using fips based libs

what we did till now for python is , got the source code of 3.9.2 and built it on Vs 2017 and generated all pyd, dlls etc. Next step we thought we need to link it to openssl 1.0.2 by replacing the include dir and libs in vcproj of _ssl.vcproj and rebuild again

can you please provide some inputs on this. is there any easy and best way which can be done

Thanks in advance

13

Sounds like you’re on the right track. You’ll probably also have to update the references in _ssl.vcxproj and _hashlib.vcxproj to the right OpenSSL libs, since the names have changed. You shouldn’t have to rebuild the rest of Python though.

I don’t think we drastically changed any of the interfaces between 1.0.2 and 1.1.1, but there may have been. So if the rebuild works, it ought to just work.

You’ll need to apply the patch to OpenSSL that I mentioned above, but I think it moved between those versions.

1 Like
14

Thanks a lot Steve. :slightly_smiling_face:

Sure will follow the steps as mentioned above

15

Steve,

You’ll need to apply the patch to OpenSSL that I mentioned above, but I think it moved between those versions. —

If you dont mind which patch i need to apply to OpenSSL . can you please provide some info on this

16

The patch is here cpython/openssl.vcxproj at 1e5d33e9b9b8631b36f061103a30208b206fd03a · python/cpython · GitHub

You’ll have to read through the steps carefully - it’s only part of one line to add a few GetModuleHandle calls, so there isn’t a big patch file.

1 Like
17

Thanks Steve :+1: