Pre-PEP discussion: revival of PEP 543

[alex_Gaynor]

@sethmlarson

Go uses system TLS

Can you expand on what you mean by this? AFAIK Go uses the system X.509 store, but does not use the OS provided TLS impl. Go has its own (written in Go) TLS impl.

[malemburg (Marc-André Lemburg)]

@woodruffw

My understanding is that SMTPS (i.e. TLS-initiated SMTP) is significantly more popular than opportunistic encryption with ESMTP, and has been for several years.

That’s true and the intended goal (see RFC 8314), but only for MUA communication (client-server), not for MTA communication (server-server).

ESMTPS is what is being most commonly used between mail relay servers (MTAs) nowadays.

Also note that IANA encourages port users who want to add encryption to their protocols to reuse the same port for unencrypted and encrypted versions of their protocols - in order to prevent the continued proliferation of port registrations (a limited resource) due to the desire to use implicit TLS. ^[1]

As a result, the need to support TLS upgrades of unencrypted connections is not going to go away soon.

@woodruffw

IMO it’d be a “win” for TLS in CPython even if no non-OpenSSL backends emerge, solely because a fixed interface with fewer abstraction leaks will be easier to maintain and attract ongoing contributor attention to

I’m not opposed to having a new TLS module in Python, but would like to see the functionality of the current ssl module (mainly its wrapping feature) to be retained as a functionality in some form.

It’s also a good idea to try to maintain a compatibility ssl module implementation which then builds on the new TLS module, since there’s a lot of code out there using the ssl module – including for less commonly known protocols.

BTW: My understanding of the idea behind wrapping existing sockets in OpenSSL is the desire to not have to deal with platform specific network stacks, which removes a lot of complexity from the library.

---

1. I was in discussion with IANA about exactly this use case for one of our products and we then agreed to support upgrades on the existing port rather than register a new protocol port for implicit TLS. ↩︎

[sethmlarson (Seth Michael Larson)]

@alex_Gaynor

Can you expand on what you mean by this? AFAIK Go uses the system X.509 store, but does not use the OS provided TLS impl. Go has its own (written in Go) TLS impl.

Oh got it, for some reason I thought they used system TLS instead of writing their own. I’ll update my comment. Thanks Alex!

[ronaldoussoren]

@malemburg

Something else I’d like to draw attention to, which hasn’t come up in the discussion yet (at least as far as I understand it):

The PEP mentions:

…this PEP is withdrawn due to changes in the APIs of the underlying operating systems…

as reason for its withdrawal.

Could you elaborate on the stability of the OS TLS APIs ? If those change every few years, we may end up with a situation where the new TLS abstraction cannot be implemented anymore due to such a change.

AFAIK this question hasn’t been answered yet (nudge, nudge).

How much needs to be changed here? Dropping a dependency on OpenSSL is nice for the CPython project, but on the other hand relying on OpenSSL does give a consistent developer experience across platforms. I’m a but worried that relying on platform TLS stack will result in subtle changes in behaviour between platforms that are confusing to developers, although I cannot substantiate this.

An alternative to reviving PEP 543 is to integrate the truststore project into the standard library as the default SSLContext, which would give us integration with the platform trust stores without otherwise changing the experience (he writes wishfully).

[woodruffw (William Woodruff)]

@malemburg

That’s true and the intended goal (see RFC 8314), but only for MUA communication (client-server), not for MTA communication (server-server).

Got it, thanks for the clarification. I wasn’t aware that MTA traffic encryption was still largely opportunistic, that’s not ideal but understandable – do you have resources on hand for these numbers? In particular, I’m curious whether the server-server full TLS issue is mostly due to holdouts from 1-2 large email providers, or whether it’s completely pervasive.

(To take a step back: I think my concerns around socket wrapping a little moot in the context of a “memory channel” API, since there will no longer be any socket in that case. With that API we get the best of both worlds: users can perform socket wrapping by operating on the socket before handing it off to the TLS layer, and “full” TLS connections can be performed straightforwardly.)

@malemburg

I’m not opposed to having a new TLS module in Python, but would like to see the functionality of the current ssl module (mainly its wrapping feature) to be retained as a functionality in some form.

It’s also a good idea to try to maintain a compatibility ssl module implementation which then builds on the new TLS module, since there’s a lot of code out there using the ssl module – including for less commonly known protocols.

Oh, this may have been poor communication on my part: my thinking here is that these proposed APIs will not immediately replace the ssl module (since it’s pervasive, like you say), but will instead be introduced over a series of phases:

1. New TLS APIs are merged, under a new module or namespace (I’ll call this tlslib purely for example purposes)
2. CPython’s internal uses of ssl are replaced with tlslib over time, external users begin to transition to tlslib as well
3. ssl is marked as deprecated, similar to the optparse/argparse transition in 3.2
4. ssl is turned into _ssl or similar (probably not _ssl itself since that’s the native extension) to mark it as an internal implementation concern
5. ssl (in its private form) is removed outright

My thinking is that this process would probably take (many) years, which should give external users time to perform deprecations and migrations. And of course this doesn’t prevent external users from using e.g. pyOpenSSL or another external binding to perform things that can’t be exposed across disparate underlying implementations.

@ronaldoussoren

AFAIK this question hasn’t been answered yet (nudge, nudge).

I tried to answer that here, but I can elaborate some more :

• Windows’ TLS stack (Schannel) does not change in incompatible ways very frequently: it dates back to 2003 and has supported TLS 1.2 since 2008 and TLS 1.3 since 2021, meaning that the bulk of Windows CPython users (especially on newer CPython releases) should have a fully TLS 1.3 stack.
• macOS has an unfortunately spotty TLS story: Secure Transport is the old (now “legacy”) API, while Network.framework is considered the “modern” API. This poses challenges for CPython and this proposal, since Network.framework does not enable the same kind of memory channel design (it wants to completely own the state machine and TLS lifecycle).

My TL;DR of the situation is “Windows should be straightforward, macOS will be a PITA unless we use a legacy API.” But IMO, per my original comment, there is still significant value in a new TLS API that still uses OpenSSL under the hood on macOS, since an independent motivation here is reducing unnecessary OpenSSL abstraction leaks even if the underlying implementation is still OpenSSL.

@ronaldoussoren

An alternative to reviving PEP 543 is to integrate the truststore project into the standard library as the default SSLContext, which would give us integration with the platform trust stores without otherwise changing the experience (he writes wishfully).

I think there’s a lot of independent value in doing this! Integrating into the platform trust store is arguably more important than using the platform TLS stack; IMO it’d be a “win” if CPython could use the platform trust store and was able to pare down the OpenSSL specifics in the current ssl API without actually having to remove the OpenSSL dep immediately. I believe one current blocker with truststore is that it can’t create SSLContexts for the purposes of client auth, but @sethmlarson may have some thoughts about that

---

As an entirely separate comment, on the topic of an “insecure” flag: @jvdprng pointed out that a less footgun-y version of an “insecure” mode would be to instruct users to construct a trust store containing the peer’s certificate, meaning that the connection would be “insecure” in the “not chained through a root of trust” sense but “secure” in the “not blindly accepting any TOFU’d connection” sense.

I’m curious whether people have any thoughts on that approach – IMO it would make the overall APIs here simpler and would make the most common “insecure” patterns more secure.

Also, to give an update: we’re finalizing the memory channel version today, and we’ll make the repository containing our draft interfaces + an example wrapper over the current ssl module public by EOD tomorrow!

[malemburg (Marc-André Lemburg)]

@woodruffw

do you have resources on hand for these numbers? In particular, I’m curious whether the server-server full TLS issue is mostly due to holdouts from 1-2 large email providers, or whether it’s completely pervasive.

This is not so much about numbers, but rather standards SMTP port 25 is used for communication between email servers (MTAs), while the other ports are used by mail clients (MUAs) to talk to their email servers. See e.g. Simple Mail Transfer Protocol - Wikipedia (the “Ports” section).

Now, whether mail servers use TLS on the port 25 connection via STARTTLS or not depends on their configuration. Many mail servers use opportunistic TLS, i.e. they try to switch to TLS on the connection as soon as possible, but will downgrade to plain text in case the other side doesn’t accept TLS (or doesn’t provide compatible cipher suites).

Fortunately, today, most mail server do accept TLS connections and you can check this by looking at the headers in your emails. If the Received: headers mention ESMTPS the connection will have successfully used STARTTLS to encrypt traffic.

Now, as for numbers I did look around a bit, but could only find this website by Gmail: Google Transparency Report which indicates that 98-99% of the traffic they receive or send uses TLS (that is STARTTLS succeeded).

@woodruffw

my thinking here is that these proposed APIs will not immediately replace the ssl module (since it’s pervasive, like you say), but will instead be introduced over a series of phases:

Sounds like a plan… except that I’d add a

• 0. A new tlslib module is developed and put on PyPI to gather experience and feedback.

[steve.dower]

@woodruffw

As an entirely separate comment, on the topic of an “insecure” flag: @jvdprng pointed out that a less footgun-y version of an “insecure” mode would be to instruct users to construct a trust store containing the peer’s certificate, meaning that the connection would be “insecure” in the “not chained through a root of trust” sense but “secure” in the “not blindly accepting any TOFU’d connection” sense.

I’m curious whether people have any thoughts on that approach – IMO it would make the overall APIs here simpler and would make the most common “insecure” patterns more secure.

Unfortunately, the primary advantage of using the OS stacks (all servers trusted by the client are trusted by Python) means that the situation where your client doesn’t trust a server belongs entirely to the end user. Any app wanting to provide an “insecure” option would have to provide a “configure your trust store manually” option, when the real thing to do is for the user to configure their system to trust the otherwise untrusted server.

The need to ignore verification entirely is an escape hatch. Nobody should really be building it into their app, but we know from experience that there are enough scenarios that need an escape hatch in order to function that we ought to provide it.

[ronaldoussoren]

@woodruffw

• macOS has an unfortunately spotty TLS story: Secure Transport is the old (now “legacy”) API, while Network.framework is considered the “modern” API. This poses challenges for CPython and this proposal, since Network.framework does not enable the same kind of memory channel design (it wants to completely own the state machine and TLS lifecycle).

My TL;DR of the situation is “Windows should be straightforward, macOS will be a PITA unless we use a legacy API.” But IMO, per my original comment, there is still significant value in a new TLS API that still uses OpenSSL under the hood on macOS, since an independent motivation here is reducing unnecessary OpenSSL abstraction leaks even if the underlying implementation is still OpenSSL

I’d expect that at least Network.framework and possibly also SecureTransport suffer from being not safe to use in programs using fork without exec, which is something we ran into with several higher-level APIs. That’s why multiprocessing uses the “spawn” start method on macOS.

Also, Network.framework not only wants to own the state machine, AFAIK it is an async framework itself (layered on top of libdispatch) which doesn’t help.

I wouldn’t use SecureTransport because it is deprecated and might get removed at some point.

So yeah, system TLS libraries on macOS is a PITA ;-).

@woodruffw

IMO it’d be a “win” if CPython could use the platform trust store and was able to pare down the OpenSSL specifics in the current ssl API without actually having to remove the OpenSSL dep immediately.

The OpenSSL dependency is used for more than just ssl, it is also used in hashlib by default. There are fallback implementations for these hash functions, but AFAIK those are less performant.

TBH I don’t fully understand why removing a dependency on OpenSSL would be a good thing overall. On the plus side there wouldn’t have to be emergency security releases when there’s a security patch for OpenSSL, but on the minus side binding to multiple system TLS stacks would increase the load on core devs.

[steve.dower]

@malemburg

That’s true and the intended goal (see RFC 8314), but only for MUA communication (client-server), not for MTA communication (server-server).

I wonder whether this is uncommon enough to justify needing the additional dependency of an actual OpenSSL module?

I fully expect that one of the primary scenarios on servers of the new module will be to install the old module, simply because the subset of functionality needed by the vast majority of clients is relatively narrow, compared to the broad range of functionality needed by servers.

That said, it may also fit nicely enough into the new model to create the new socket-like object with “TLS if you can, unencrypted if not” (which ought to be easy enough if it’s internally linked to a memory channel), and then allow upgrading the unencrypted connection later (by adding the memory channel and starting to pass messages through it).^[1] So it’s possible that this is easily handled, provided the recipient of the first TLS packet knows that it’s coming.

---

1. If I have the wrong idea here it’s entirely my own ignorance, so feel free to correct me. ↩︎

[steve.dower]

@ronaldoussoren

TBH I don’t fully understand why removing a dependency on OpenSSL would be a good thing overall. On the plus side there wouldn’t have to be emergency security releases when there’s a security patch for OpenSSL, but on the minus side binding to multiple system TLS stacks would increase the load on core devs.

At least on Windows, the benefit is perf (without even trying I’ve been able to 2x-20x throughput with native APIs - higher level APIs than we’re talking here, unfortunately) and configuration (OpenSSL simply can’t implement both certificate parsing/verification and integrate with the OS doing the same).

With OpenSSL, users often have to convert and distribute certificates, and configure random (to them) environment variables just in order to access their own sites that their IT department has already deployed certs for. These don’t auto-rotate, while IT probably auto-rotates them, which means periodic downtime. And in some systems the public half of the certificate is also considered a secret - native verification can handle this, but OpenSSL cannot/will not, which means compromising your security configuration, which often means rewriting code in another language that doesn’t depend on OpenSSL…

[pf_moore (Paul Moore)]

@steve.dower

With OpenSSL, users often have to convert and distribute certificates, and configure random (to them) environment variables just in order to access their own sites that their IT department has already deployed certs for.

Is this specific benefit not covered by truststore? I’m not qualified to comment on the other aspects of this proposal, but from my experience the most significant problem with Python networking in a corporate environment was that the IT department had everything set up so that browsers and all the standard tools worked, but Python programs didn’t^[1]. And worse still, because IT handled everything else, getting the information I needed to configure Python to work was difficult, if not impossible. Anything that fixes that issue, whether it’s this proposal or truststore, would be a major improvement IMO.

---

1. This actually goes beyond certificates, and includes things like proxies as well ↩︎

[steve.dower]

@pf_moore

Is this specific benefit not covered by truststore?

Certificates are, provided you’re in an environment where ctypes is allowed (not something I get to assume, these days). I’ve got a private non-ctypes module that does the same thing, which some of my teams use, but both do the not-quite-right thing of entirely replacing OpenSSL’s certificate handling with the Windows API that does the same thing, but without necessarily applying other policy around that.

Unfortunately it’s such a deep rabbit hole of things to do to Get It Right™ that I just can’t imagine managing it with a hybrid TLS stack (e.g. should be we OCSP stapling? For this host? Is leaf certificate trust sufficient today or do we need root trust? I don’t even know where these policy settings live, but they’re all options/addons to the native API, so I have to assume they might be set).

[freakboy3742 (Russell Keith-Magee)]

@sethmlarson

The long-term end goal in my mind is using this functionality in the stdlib automatically and one day not shipping OpenSSL with Windows and macOS installers.

Completely aside from any security implications, relying on system-provided libraries rather than OpenSSL would also be a major benefit to the iOS build. OpenSSL support on iOS exists, but isn’t great; but more importantly, OpenSSL is a major contributor to the size of iOS builds. This is especially problematic on iOS because every app needs its own copy of Python (as there are no shared resources other than the ones provided by the system).

I will admit I’m not especially familiar with the system provided crypto libraries on iOS; but AIUI, they are essentially the same as those provided by macOS, so any solution that works for macOS should (cough) also work for iOS. This sample project was provided at the time the macOS OpenSSL “deprecation” occurred as an example for how to migrate OpenSSL usage patterns to macOS, and the same code is iOS8+ compatible.

[malemburg (Marc-André Lemburg)]

@steve.dower

@malemburg

That’s true and the intended goal (see RFC 8314), but only for MUA communication (client-server), not for MTA communication (server-server).

I wonder whether this is uncommon enough to justify needing the additional dependency of an actual OpenSSL module?

There are a few more protocols using STARTTLS-like upgrades to TLS, see e.g. Opportunistic TLS - Wikipedia, but I’m sure there are more, since the net has slowly been moving towards TLS and there are plenty older protocols which were defined at a time when TLS was not a common thing to have.

XMPP^[1] and LDAP^[2] are certainly protocols which are in wide usage today and they need client side TLS upgrade support to be secure.

It’s also possible that you may have to send emails to a LAN mail relay using port 25 and STARTTLS (e.g. if the relay or firewall doesn’t support port 587 submission).

Using e.g. pyOpenSSL as a fallback solution would remain a possibility (provided this gets continued maintenance), but if possible, I think the new TLS module should try to provide functionality to support such dynamic TLS connection upgrades as well.

---

1. For reference, see RFC 6120: Extensible Messaging and Presence Protocol (XMPP): Core ↩︎

2. LDAP also has a non-standard TLS variant called LDAPS, but this was deprecated in favor of the STARTTLS functionality on the standard LDAP port. See Lightweight Directory Access Protocol - Wikipedia (“Protocol section”). ↩︎

[steve.dower]

My point about “uncommon” is really that our 99%[citation needed] case is HTTPS, and anyone who needs other protocols is going to be aware enough of it to take an additional step on Windows/macOS/iOS to add an additional library (I expect OpenSSL and probably the whole ssl module to remain on platforms where it’s the best way to provide support, and just become optional-but-installable on others).

I know going from zero 3rd-party packages to one is the hardest step, but are we really looking at people maintaining servers on top of these protocols with zero 3rd party packages these days?

@malemburg

I think the new TLS module should try to provide functionality to support such dynamic TLS connection upgrades as well.

I agree, but I don’t think it should be required for the new module to be considered a success^[1] (which you also seem to imply with “try”, but let’s also not over-focus on this aspect or people will get the impression that it’s the most important part )

---

1. Unless of course, we ship without it and everyone hates it because it’s not there. ↩︎

[malemburg (Marc-André Lemburg)]

@steve.dower

I know going from zero 3rd-party packages to one is the hardest step, but are we really looking at people maintaining servers on top of these protocols with zero 3rd party packages these days?

For servers, I agree, but for client protocols in common use (such as SMTP, LDAP, XMPP and probably a few others as well), I think we should continue to provide support for upgrading existing socket connections to TLS.

I’m mostly trying to make the point that outright rejecting the idea of wrap existing sockets for use with TLS is not going in the right direction. The same goes for rejecting the idea to have the user decide whether s/he wants certificate verification or not. Both fall into the “practicality beats purity” category.

To your point, though: I agree that we have hashed this out well enough in this discussion. So let’s see what can be done in a prospective tlslib implementation to address those two aspects and how well the lib can hide OS level differences in how TLS is handled. If OpenSSL is kept as additional cross-platform “plugin” (possibly via the pyOpenSSL module or cryptography), I think we’d be well set up for the future.

[mhsmith (Malcolm Smith)]

@freakboy3742

OpenSSL is a major contributor to the size of iOS builds. This is especially problematic on iOS because every app needs its own copy of Python (as there are no shared resources other than the ones provided by the system).

Android uses BoringSSL internally, but that isn’t part of the public API, and on newer Android versions they’ve actually modified the dynamic linker to prevent you from using it directly.

So the only system-provided SSL API on Android is the Java standard library. I suppose this PEP could be implemented on top of that using JNI, but it probably wouldn’t be worth the effort. OpenSSL is only 4 MB, which isn’t a major problem in the context of a modern phone.

[steve.dower]

One thing that I think I’ve implied a number of times but haven’t said outright is that I’d also like to see this be usable totally independently of the standard socket module, in part because socket on Windows is a stack of about three compatibility layers, but also because I think the more that goes behind the new API the better in terms of portability and optimisation.

So I’m picturing three major sections of an API (and possibly the second can be implementation-independent):

• a memory channel implementation of TLS (local user sends/receives messages to the channel and then either uses the data or posts the message back to the remote)
• “wrapped socket” abstraction that handles this send/receive for an existing socket-like object
• higher-level APIs for common protocols

The third is already available in terms of standard sockets, of course, and I’d expect those to continue to work. But by having a leak-free abstraction we can provide more efficient implementations. (As an analogy, shutil.copy2 can be implemented with “standard” primitives, such as open(), but that’s not specified as part of the API, so our implementation can use a native copy function when available. On Windows at least, this is many times faster, more robust, and behaves more like an OS copy.)

It might be that only HTTPS is valuable enough to provide on its own these days, as most platforms seem to have a dedicated HTTP API (possibly web sockets or FTP as well?). I would really like to be able to take advantage of the dedicated platform APIs for platforms that have them.^[1]

---

1. Without having to maintain a single library that works efficiently on all platforms. Nobody wants to do that, so a pluggable backend model with “universal” fallback to Python+OpenSSL seems a great way to foster development for those platforms that can benefit from it. ↩︎

[sethmlarson (Seth Michael Larson)]

@woodruffw

I think there’s a lot of independent value in doing this! Integrating into the platform trust store is arguably more important than using the platform TLS stack; IMO it’d be a “win” if CPython could use the platform trust store and was able to pare down the OpenSSL specifics in the current ssl API without actually having to remove the OpenSSL dep immediately. I believe one current blocker with truststore is that it can’t create SSLContexts for the purposes of client auth, but @sethmlarson may have some thoughts about that

I also think this would be a big win, it has the same problem with Secure Transport in that if the APIs are removed we have no way forward. For most people this would be the biggest improvement (no more patching certifi!) and if we’re not going to be able to get away from OpenSSL entirely having truststore-like support in the ssl module makes sense to me.

I haven’t looked at any server implementations using SChannel or Secure Transport, we didn’t implement client auth in truststore because we only needed server auth for truststore to be useful for pip

[ronaldoussoren]

@steve.dower

Certificates are, provided you’re in an environment where ctypes is allowed (not something I get to assume, these days). I’ve got a private non-ctypes module that does the same thing, which some of my teams use, but both do the not-quite-right thing of entirely replacing OpenSSL’s certificate handling with the Windows API that does the same thing, but without necessarily applying other policy around that.

The ctypes bit is easily fixed if we’d include the functionality in the standard library. It’s not that hard to rewrite those bits in C ;-).

The hard bit is to verify that the new functionality does indeed follow the expected platform semantics for certificate validation and then implement the missing bits “somehow”. From what I recall from previous discussions the trust store policy bits for Windows and macOS are different from what vanilla OpenSSL does (as you already mention).