With OpenSSL 1.1.1 reaching EOL 20223-09-11 and from what I can gather folks concerned about OpenSSL 3.0 stability, I think it might be time to start a discuss about weaning ourselves off of OpenSSL as a critical dependency that we have. I know this has been a consistent pain on non-Unix platforms like Windows which don’t have OpenSSL out-of-the-box, so this isn’t a new concern.
I could be wrong, but I believe some key places where we use OpenSSL are:
ssl which is used by things like urllib to provide HTTPS support and in general for TLS
In the past there was PEP 543 to rely on the OS to provide TLS support, but it lost a co-author and never had enough support behind it to carry it forward. I know Victor has proposed coming up with a fetch-like API for urllib which would abstract away at least HTTPS concerns, but that doesn’t necessarily cover the TLS use case over a raw socket.
Anyway, OpenSSL seems like one of those things that has been a constant concern, but that we have punted on in general. But with this pending EOL it might be time to figure out what we want to do long-term about this.
I guess the big issue here is what to do with our first-party binary builds? For third-party distributors, they’re already making a variety of decisions about how to manage the ssl module and I guess it’s working OK for them?
In any case, I don’t think it’s realistic to drop openssl entirely. Even if PEP 543 was implemented and shipped today, there would still be so much software out there relying on the ssl module for years, and the ssl module’s API is tightly tied to openssl implementation details.
I think the options are:
Wait and see: half the internet is in the same position as us re: openssl 1.1.1/3.0, so as the planned 1.1.1 EOL date approaches I expect some kind of consensus will develop about what to do – whether that’s pushing back the EOL date, or openssl upstream getting things together to fix 3.0, or a group coalescing to maintain 1.1.1 past the EOL or what.
Fedora uses OpenSSL 3.0 for python and most other packages.
I know stability was a issue a while ago, but thought that 3 was good for prime time use now. But I am just watching OpenSSL from the side lines.
In my work we are still using 1.1
I’ve been following OpenSSL development quite closely - what issues have you heard of, and are there some references? The backports to 3.0 have been working regularly (and where any change in behaviour is required, there’s a pretty heavy-weight voting process by committee, so it happens very rarely and mostly in pretty remote corners of the API) and the maintainers have been receptive/helpful with bug reports.
3.0 has been announced as the next LTS release after 1.1.1, which IMO leaves little choice but to upgrade eventually (EOL of 1.1.1 is in Sept. 2023).
As a reference point: There have been some growing pains in conda-forge with rebuilding everything for OpenSSL 3, but things are turning into home stretch.
I tend to agree. Furthermore, I think there’s value in having python have a unified API[1], though I don’t make claims about whether it’s “worth it” in terms of costs (maintenance effort, etc.).
I think “switch to bundling OpenSSL 3” should definitely be an option here as well (in fact, it should be the default position IMO, barring strong arguments against it).
PEP 543 resolution notes: “this PEP is withdrawn due to changes in the APIs of the underlying operating systems.” ↩︎
I assume that switching to OpenSSL 3 wouldn’t involve any breaking changes to our API? (Potentially to the behaviour of it, but not the shape in a way that forces our users to also change their code.)
If it does, I’d be more inclined to make ssl “optional” and default to platform-native modules for those that can provide other stdlib functionality.
(The main thing holding me back from pushing for this change anyway is honestly the amount of people who already use the OpenSSL environment variables as workarounds for its limitations in CPython. If we hadn’t trained our users on these, a smoother transition might have been possible, but the only painless approach seems to be the one where the ssl API doesn’t change at all.)
FWIW, CPython worked with OpenSSL 3.0 pretty much upon its release (thanks to the tireless work of @tiran), and conda-forge has been publishing CPython 3.7-3.11 builds against OpenSSL 3.0 for over a year. In all that time, I haven’t become aware of any behavioural changes (issues, test failures, crashes, etc.) due to building CPython against OpenSSL 3.0.
This is partly because OpenSSL 3 was developed with maximum compatibility to 1.1.1 in mind (the larger changes were mostly internal, like switching to providers). The biggest issue I’m aware of in the grand scheme of things are projects that are still using APIs which have been deprecated for a long time, some of which got removed in 3.0.
When is the last time we asked them about that? If it’s been a few years maybe it would be good to check again. After all , the desire for “complete freedom to change the APIs” is usually a sign of a young library (or perhaps one with a lot of current activity). BoringSSL may have settled down.
OTOH, if they consider themselves primarily an internal Google library, where indeed they have the freedom to change all code that uses it (because monorepo), why are they even open source? What do they expect their open source users to do? Fork it?
For completeness, this only works if bundling both the default & legacy provider (some functions were moved to the latter in OpenSSL 3). CPython isn’t loading the latter by default, which would address the one bug report that I now happened to remember.
That’s an easy fix though – just load the legacy provider in hashlib – and should be fixed (latest when switching to bundling OpenSSL 3), otherwise users of old hashing functions would see a regression.
I don’t think that’s quite right. IIUC their position is “see that NO WARRANTY sign? yeah we mean it”, i.e. they’re fine with people using their code as long as no-one yells at them if it breaks. E.g. Apple ships BoringSSL as part of their Swift libraries:
(Coincidentally, the author of that blog post author is also one of the PEP 543 authors)
Obviously the ideal would be a TLS library with solid API stability guarantees. But that’s not really on the table. OpenSSL upstream actually breaks stuff all the time, and often has no idea what API contracts their users have been relying on. BoringSSL OTOH reserves the right to break stuff, but they actually test their code against ~everything Google uses, so their stability is much higher than OpenSSL in practice. It’s a pretty weird situation but there it is
My understanding is that they use it as a drop-in replacement for OpenSSL in everything Google uses, so everything in the monorepo, and also Chrome/ChromeOS and Android (which are their own separate massive repos). And all of these repos contain tons of third-party code (like CPython!), so it’s easier to adapt BoringSSL to work for that code than it is to adapt that code to work for BoringSSL.
If we were considering using BoringSSL for real, then the next question would be to articulate exactly what kind of stability we needed. Currently for our binary releases we vendor OpenSSL and we don’t issue new releases when OpenSSL fixes security bugs, so I think we really only care about having one working recent-ish snapshot at release time? (And in particular, we don’t rely on any kind of version-to-version ABI stability?)
Can you provide some examples that affect OpenSSL 3.0?
From what I’ve seen, they take their stability promises very seriously whenever something was raised on their issue tracker:
No API or ABI breaking changes are allowed in a minor or patch release.
No existing public interface can be modified except where changes are unlikely to break source compatibility […]
No existing public interface can be removed until its replacement has been in place in an LTS stable release. The original interface must also have been documented as deprecated for at least 5 years.
Of course that’s no proof that it doesn’t happen, but I’d still be interested to see what got broken recently.
For the record, I’m using OpenSSL 3.0.3 on Gentoo since May and I don’t recall seeing any major issues in CPython or Python packages. Gentoo is planning to switch to OpenSSL 3 by the end of the year (we’re dealing with the last blockers right now).
test_ssl.py needs significant modification to “pass” when linked with boringSSL and various pieces of the _ssl extension code need modifications as well as workarounds to disable unsupported features in some places. We carry patches on our Python runtime internally at Google that does some this, but they wouldn’t be acceptable as is upstream as they disable features that need to work for global users and skip some tests rather than making them work"“”
basically we’d be signing up for work to maintain the ssl module across whatever changes are required due to boringssl needs for the full lifetime of a cpython security release. eek!
3.12 will ship with openssl 3.0. No choice.
The very long term question is how do we deprecate the ssl module in favor of high level native platform APIs. (and should we)
Q: If designed today, why would anyone have a stdlib ssl module? In this day and age all platforms have TLS https stacks built in. Plenty sufficient as a thing to wrap into a battery. Good enough for pip.
This may be the case for Windows and macOS, but for Linux and other Unix systems, OpenSSL (or one of the forks) effectively is the “TLS https stack”, so I guess the answer would be: we’d still have an ssl module, but one with a PEP 543 style interface and several platform specific backends, including OpenSSL for Linux and other Unix platforms.
In other words: OpenSSL will not be going away anytime soon.
Also note that OpenSSL works at the socket level, not at the HTTPS level, so it’s more generic than e.g. a requests type of interface and can be used for other protocols as well.
I know this was true in the OpenSSL earlier days (from working on mxCrypto and egenix-pyopenssl), but this has changed a lot for the better since the project got reorganized. Back then even patch level releases often had API breaking changes.
There’s a OpenSSL 3 project, but a lot of the issues are improvements/additions, or kept open without a clear next step.
What needs to be fixed (how does the amount of work needed compare to switching to something else)?
(Are all the details only in Christian’s head?)
This is what we all imply by the answer that Greg gave. OpenSSL is the native stack for platforms where it’s the native stack, we just wouldn’t have it define the public API within Python (you’d have to test for it - and possibly install it - to use it directly).
This is true, and important. The other native stacks can do it too, but not very easily (though maybe that’s just because HTTPS is such a big, complex protocol and I’ve not tried using their other APIs for simpler ones?). But these may well be cases where the apps that need to do it aren’t going to rely on any platform integration anyway, such as proxy preferences, PKI or cached certificates, and so it would be just fine for them to pull in (e.g.) PyOpenSSL and use it to wrap their socket.
(Probably HTTPS looked this simple 15+ years ago, I wasn’t really looking at the time. It’s only with hindsight that we can now say it would’ve been better to abstract it at the request level rather than the transport level, so no criticism of the original decision.)
Can you clarify? I’m not familiar with other stacks so I’m curious - do other APIs handle STARTTLS differently?
In Python, I would expect to be able to establish a socket (socket.create_connection), communicate over it (recv/send), then send a command and upgrade to an encrypted connection. That seems like a pretty straight-forward use-case (required by various protocols and becoming progressively more common everywhere outside of HTTP), so I’d assume that it’s possible somehow, just that it’s done differently - right?
It means it’d use a urlretrieve style API, rather than a ssl.wrap_socket style API, and would delegate control of the TLS connection to the operating system (which generally also offer a urlretrieve style API that can be pretty easily wrapped).
