PEP 644: Require OpenSSL 1.1 or newer

PEP: 644
Title: Require OpenSSL 1.1 or newer
Author: Christian Heimes christian@python.org
BDFL-Delegate: n/a
Discussions-To: https://discuss.python.org/
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Oct-2020
Python-Version: 3.10
Post-History: 27-Oct-2020

Abstract

This PEP proposes for CPython’s standard library to support only OpenSSL
1.1.1 LTS or newer. Support for OpenSSL versions past end-of-lifetime,
incompatible forks, and other TLS libraries are dropped.

Motivation

Python makes use of OpenSSL in hashlib, hmac, and ssl modules. OpenSSL
provides fast implementations of cryptographic primitives and a full TLS
stack including handling of X.509 certificates. The ssl module is used by
standard library modules like urllib and 3rd party modules like urllib3
to implement secure variants of internet protocols. pip uses the ssl
module to securely download packages from PyPI. Any bug in the ssl module’s
bindings to OpenSSL can lead to a severe security issue.

Over time OpenSSL’s public API has evolved and changed. Version 1.0.2
introduced new APIs to verify and match hostnames. OpenSSL 1.1.0 made
internal structs opaque and introduced new APIs that replace direct access of
struct members. Version 3.0.0 will deprecate more APIs due to internal
reorganization that moves cryptographic algorithms out of the core and into
providers. Forks like LibreSSL and BoringSSL have diverged in different
directions.

Currently Python versions 3.6 to 3.9 are compatible with OpenSSL 1.0.2,
1.1.0, and 1.1.1. For the most part Python also works with LibreSSL >= 2.7.1
with some missing features and broken tests.

Due to limited resources and time it becomes increasingly hard to support
multiple versions and forks as well as test and verify correctness. Besides
multiple incompatible APIs there are build time flags,
distribution-specific patches, and local crypto-policy settings that add to
plethora of combinations. On the other hand, the Python core team has only
a couple of domain experts who are familiar with TLS and OpenSSL internals
and even fewer who are active maintainers.

Requiring OpenSSL 1.1.1 would allow us to give the vast majority of users a
better experience, reduce our maintenance overhead and thus free resources
to implement new features. Users would be able to rely on the presence of
new features and consistent behavior, ultimately resulting in a more robust
experience.

Impact

OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all
supported platforms and distributions. It’s also the only version that still
receives security support from upstream [9]_.

No macOS and Windows user will be affected by the deprecation. The python.org
installer and alternative distributions like Conda ship with most recent
OpenSSL version.

As of October 2020 and according to DistroWatch [1]_ most current BSD and
Linux distributions ship with OpenSSL 1.1.1 as well. Some older releases of
long term support (LTS) and enterprise distributions have older versions of
OpenSSL or LibreSSL. By the time Python 3.10 will be generally available,
several of these distributions will have reached end of lifetime, end of
general support, or moved from LibreSSL to OpenSSL.

Other software has dropped support for OpenSSL 1.0.2 as well. For example
PyCA cryptography 3.2 (2020-10-25) removed compatibility with OpenSSL 1.0.2.

OpenSSL 1.0.2 LTS

released: 2015-02
end of lifetime: 2019-12

OpenSSL 1.0.2 added hostname verification, ALPN support, and elliptic curves.

  • CentOS 7 (EOL 2024-06)
  • Debian 8 Jessie (EOL 2020-07)
  • Linux Mint 18.3 (EOL 2021-04)
  • RHEL 7 (full support ends 2019-08, maintenance 2 support ends 2024-06)
  • SUSE Enterprise Linux 12-SP5 (general supports ends 2024-10)
  • Ubuntu 16.04 LTS / Xenial (general support ends 2021-04)

OpenSSL 1.1.0

released: 2016-08
end of lifetime: 2019-09

OpenSSL 1.1.0 removed or disabled insecure ciphers by default and added
support for ChaCha20-Poly1305, BLAKE2 (basic features), X25519 and CT. The
majority of structs were made opaque and new APIs were introduced. OpenSSL
1.1.0 is not API compatible with 1.0.2.

  • Debian 9 Stretch (estimated EOL 2022-06)
  • Ubuntu 18.04 LTS / Bionic (general support ends 2023-04)

OpenSSL 1.1.1 LTS

released: 2018-08
end of lifetime: 2023-09 (planned)

OpenSSL 1.1.1 added TLS 1.3, SHA-3, X448 and Ed448.

  • Alpine (switched back to OpenSSL in 2018 [4]_)
  • Arch Linux current
  • CentOS 8.0+
  • Debian 10 Buster
  • Fedora 29+
  • FreeBSD 11.3+
  • Gentoo Linux stable
  • HardenedBSD (switched back to OpenSSL in 2018 [3]_)
  • Linux Mint 19.3+
  • macOS (python.org installer)
  • NetBSD 8.2+
  • openSUSE 15.2+
  • RHEL 8.0+
  • Slackware current
  • SUSE Enterprise Linux 15-SP2
  • Ubuntu 18.10+
  • Ubuntu 20.04 LTS / Focal
  • Windows (python.org installer, Conda)

OpenSSL 3.0.0

released: n/a (planned for early 2021)

OpenSSL 3.0.0 is currently under development. Major changes include
relicensing to Apache License 2.0 and a new API for cryptographic algorithms
providers. Most changes are internal refactorings and don’t affect public
APIs. [8]_

LibreSSL

created: 2014-04 (forked from OpenSSL 1.0.1g)

  • DragonFly BSD
  • Hyperbola GNU/Linux-libre
  • OpenBSD
  • OpenELEC (discontinued)
  • TrueOS (discontinued)
  • VOID Linux (currently moving back to OpenSSL [5]_)

Some distributions like FreeBSD, Gentoo, and OPNsense also feature LibreSSL
instead of OpenSSL as non-standard TLS libraries.

OpenBSD ports has a port security/openssl/1.1 which is documented as
“[…] is present to provide support for applications which cannot be made
compatible with LibReSSL” [7]_. The package could be used by OpenBSD to
provide a working ssl module.

BoringSSL

created: 2014-06

BoringSSL is Google’s fork of OpenSSL. It’s not intended for general use and
therefore not supported by Python. There are no guarantees of API or ABI
stability. Vendored copies of BoringSSL are used in Chrome/Chromium browser,
Android, and on Apple platforms [6]_.

Benefits

TLS 1.3

OpenSSL 1.1.1 introduced support for the new TLS 1.3 version. The latest
version of the TLS protocol has a faster handshake and is more secure than
the previous versions.

Thread and fork safety

Starting with release 1.1.0c, OpenSSL is fully fork and thread safe.
Bindings no longer need any workarounds or additional callbacks to support
multithreading.

SHA-3

Since 1.1.0, OpenSSL ships with SHA-3 and SHAKE implementations.
Python’s builtin SHA-3 support is based on the reference implementation. The
internal _sha3 code is fairly large and the resulting shared library close
to 0.5 MB. Python could drop the builtin implementation and rely on OpenSSL’s
libcrypto instead.

So far LibreSSL upstream development has refused to add SHA-3 support. [2]_

Compatibility

OpenSSL downstream patches and options

OpenSSL features more than 70 configure and build time options in the form
of OPENSSL_NO_* macros. Around 60 options affect the presence of features
like cryptographic algorithms and TLS versions. Some distributions apply
patches to alter settings. Furthermore default values for settings like
security level, ciphers, TLS version range, and signature algorithms can
be set in OpenSSL config file.

The Python core team lacks resources to test all possible combinations.
This PEP proposes that Python only supports OpenSSL builds that have
standard features enabled. Vendors shall disable deprecated or insecure
algorithms and TLS versions with build time options like
OPENSSL_NO_TLS1_1_METHOD or OpenSSL config options like
MinProtocol = TLSv1.2.

Python assumes that OpenSSL is built with

  • hashlib’s default algorithms such as MD5, SHA-1, SHA-2 family,
    SHA-3/SHAKE family, BLAKE2
  • TLS 1.2 and TLS 1.3 protocols
  • current key agreement, signature, and encryption algorithms for TLS 1.2
    and 1.3 (ECDH, RSA, ECDSA, Curve25519, AES, Poly1309-ChaCha20, …)
  • threading, file I/O, socket I/O, and error messages

Weak algorithms (MD5, SHA-1 signatures) and short keys (RSA < 2024 bits) may
be disabled at runtime. Algorithms may also be blocked when they are
disabled by a crypto policy such as FIPS. The PEP is not more specific on
purpose to give room for new features as well as countermeasures against
vulnerabilities. As a rule of thumb, Python should be able to connect to
PyPI and the test suite should pass.

LibreSSL support

LibreSSL is a fork of OpenSSL. The fork was created off OpenSSL 1.0.1g by
members of the OpenBSD team in 2014 in light of the heartbleed vulnerability.
Since its inception several features deemed problematic or insecure were
removed or replaced (SSL 2.0, SSL 3.0, improved CPRNG) or backported
from OpenSSL and BoringSSL.

At the moment LibreSSL is not fully API compatible with OpenSSL 1.1.1. The
latest release LibreSSL 3.3.2 is missing features and behaves differently
in some cases. Mentionable missing or incompatible features include

  • SHA-3, SHAKE, BLAKE2
  • SSL_CERT_* environment variables
  • security level APIs
  • session handling APIs
  • key logging API
  • verified cert chain APIs
  • OPENSSL_VERSION macro

This PEP proposed to remove any and all LibreSSL related workarounds from
Python. In the future Python will not actively prohibit LibreSSL support
with configure and compile time checks. But Python will not accept patches
that add non-trivial workarounds or disable tests either.

BoringSSL

There are currently no plans to support BoringSSL.

Rejected Ideas

Formalize supported OpenSSL versions

This PEP does not provide a set of formal rules and conditions under which
an OpenSSL version is supported.

In general Python aims to be compatible with commonly used and officially
supported OpenSSL versions. Patch releases of Python may not be compatible
with new major releases of OpenSSL. Users should not expect that a new major
or minor release of Python works with an OpenSSL version that is past its
end-of-lifetime. Python core development may backport fixes for new releases
or extend compatibility with EOLed releases as we see fit.

The new ABI stability and LTS policies of OpenSSL [9]_ should help, too.

Backwards Compatibility

Python 3.10 will no longer support TLS/SSL and fast hashing on platforms
with OpenSSL 1.0.2 or LibreSSL. This PEP is published at the beginning of
the 3.10 release cycles. It gives vendors like Linux distributors or CI
providers roughly 11 months to react.

Disclaimer and special thanks

The author of this PEP is a contributor to OpenSSL project and employed by
a major Linux distributor that uses OpenSSL.

Thanks to Alex Gaynor, Gregory P. Smith, Nathaniel J. Smith, Paul Kehrer,
and Seth Larson for their review and feedback on the initial draft.

References

… [1] https://distrowatch.com/
… [2] https://github.com/libressl-portable/portable/issues/455
… [3] https://hardenedbsd.org/article/shawn-webb/2018-04-30/hardenedbsd-switching-back-openssl
… [4] https://lists.alpinelinux.org/~alpine/devel/<CA%2BT2pCGFeh30aEi43hAvJ3yoHBijABy_U62wfjhVmf3FmbNUUg%40mail.gmail.com>
… [5] https://github.com/void-linux/void-packages/issues/20935
… [6] https://forums.swift.org/t/rfc-moving-swiftnio-ssl-to-boringssl/18280
… [7] https://openports.se/security/openssl/1.1
… [8] https://www.openssl.org/docs/OpenSSL300Design.html
… [9] https://www.openssl.org/policies/releasestrat.html

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.


Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:

5 Likes

You probably meant to put this in the PEPs category. Not sure if you can move it yourself or if a moderator has to do it.

Yes … I was in the PEP category when I created the topic. I’m not sure what went wrong.

Is is possible, legal and feasible to include the current/tested-ok OpenSSL source in the
shipped source tree, possibly with a configure switch to use it or the
OS OpenSSL (if new enough)?

I ask this because I recently built a modern Python (3.8.5 at the time)
on some older Debian hosts, and the OpenSSL part was a massive PITA. To
this day I need to include a special LD_LIBRARY_PATH in the environment
to use that Python, and getting Python to even link against the OpenSSL
1.1.1h at build time I’d built was not easy.

For people building from source on an arbitrary machine it would be a
great boon if this just worked out of the box.

Cheers,
Cameron Simpson cs@cskk.id.au

What you really seem to be asking for is for CPython sources to include Makefile magic to tell a modern OpenSSL version to build a static libssl and link against that when a local recent enough library+headers is not already installed. I do not think we should bundle sources so large ourselves (if even feasible) or even be responsible for such a build process directly.

It seems reasonable enough to detect the lack of modern enough OpenSSL from our ./configure and drop a link to instructions for to people on how to obtain and build a recent enough version so they can use the existing --with-openssl= flag.

Basic instructions to build your own OpenSSL and build a Python interpreter using it are:

# Visit https://www.openssl.org/source/ to find the relevant version.
curl -O https://www.openssl.org/source/openssl-1.1.1h.tar.gz
# Verify that download however you choose
tar -xzf openssl-1.1.1h.tar.gz
cd openssl-1.1.1h
ssldir="$(pwd)/../static-ssl-install"
# /etc/ssl here needs to match where your OS distro puts that, and pick your os-arch.
./Configure no-shared linux-x86_64 --openssldir=/etc/ssl --prefix=$ssldir
make install_sw
cd ../python-build
../Python-3.xx/configure --with-openssl=$ssldir
make

Totally honest answer: Because I’m not willing to put in the extra work. I’m pretty sure that you will get the same answer from other core developers.

I have no incentive and see no reason to include a copy of OpenSSL in CPython. It would be a non-trivial effort to integrate OpenSSL and a serious long-term commitment to maintain the copy. The main goal of the PEP is a reduction of our maintenance overhead. Your proposal would increase our maintenance cost.

Maintaing a source copy of OpenSSL in Python itself would not
really help:

  • It takes a serious effort. I’ve been doing something like this
    with egenix-pyopenssl for a number of years and gave up:
    https://www.egenix.com/products/python/pyOpenSSL/

  • If you have a system OpenSSL lib, which is still maintained,
    this is always the better option.

  • You’d tie OpenSSL updates to Python updates and would have to
    upgrade to the latest Python version just to get the latest
    OpenSSL version.

I think adding more documentation on how to build OpenSSL on
various platforms would be better and pointing users to this
documentation in case ./configure finds that the system doesn’t
ship with OpenSSL 1.1+.

1 Like