Pre-PEP discussion: revival of PEP 543

My thinking here is that the design will be suitable for “drop in” use within asyncio: I will defer to people who are more familiar with async programming here, but my thought is that the current loop.create_connection() API will be able to take either a new tls= kwarg or re-use the existing ssl= one, with additional support for these new interfaces. Under the hood this would then create a nonblocking socket (or memory channel), retaining the current asyncio connection APIs around TLS/SSL.

(Some related abstractions might need to adapt slightly, since one of the proposed design improvements here is reducing the number of abstraction/OpenSSL-specific details that get leaked through Python APIs. So, for example, BaseTransport.get_extra_info() may need to return different/reduced information with these new APIs versus the ssl ones).

My understanding is that SMTPS (i.e. TLS-initiated SMTP) is significantly more popular than opportunistic encryption with ESMTP, and has been for several years. Gmail, for example, has supported SMTPS on port 587 for at least a decade, as has Outlook and other large hosted email providers.

Particularly, I’ve checked that each of the following supports SMTPS:

  • Gmail
  • Outlook
  • Protonmail
  • Fastmail
  • Yahoo
  • Zoho

(Together, I think these constitute over 90% of SMTP handling on the public internet, and I think it’s safe to say that the next ~two dozen largest SMTP hosts also support SMTPS. But I will substantiate this more concretely.)

I see this not as a matter of “can”, but “should” :slightly_smiling_face: – Python is at this point idiosyncratic among general-purpose programming languages when it comes to the “socket wrapping” pattern: the only other major language that I can find that supports it (but does not encourage it) is Ruby, which does so for similar reasons (having a public API that closely reflects OpenSSL internals).

The reasons for this are AFAICT related to the above: the security model of opportunistic encryption is harder to explain to users, is subject to uncontrolled downgrades that compromise transit integrity/privacy, and is also more misuse-prone. It’s also a “speedbump” in turns of current networking idioms: a lot of internet traffic is TLS at this point, so making programmers navigate two layers of abstraction (sockets and TLS) to get to a functional I/O object makes Python harder to use than strictly necessary :slightly_smiling_face:

With that being said, I think the design constraints proposed above would make it relatively straightforward for a user (or library) to build their own socket-wrapping primitive, especially if we settle on a memory channel design. IMO this is the best of both worlds, in that it aligns Python’s TLS APIs with other major languages (e.g. Go), reduces the overall API surface (wins for both security and maintainability), and doesn’t outright prevent the wrapping case (just doesn’t expose it directly as a public API).

Sure! I’ll ask @jvdprng to fill in some more details from his research as well, but my understanding is that the changes in question were primarily macOS’s deprecation of Secure Transport in favor of Network.framework.

The former was OS sockets based while the latter is event-driven and uses abstract “connection” handles that aren’t necessarily tied to a socket or other specific transport interface. My understanding is that this change happened roughly around the same time that the two original PEP 543 authors were no longer able to find time to work on it, so it’s less that the changes are insurmountable and more that they were inopportune and required a larger architectural rethinking (e.g. towards memory channels, which is where we’re trending now).

I agree with the points about risk and adaptation. At the same time, I think it’s important to note that both this proposal and the original PEP don’t require that Python use the OS’s TLS stack: both are intended to put forward a simpler and easier to maintain abstraction, one that’s compatible with OpenSSL without exposing OpenSSL’s internal implementation details like the current ssl module does. This IMO is a sufficiently good reason to consider a new TLS API within the standard library: CPython can choose to continue to use OpenSSL (and only OpenSSL) under the hood, but can do so more confidently knowing that it controls the abstraction presented to users without leaking (potentially changing) implementation details from OpenSSL itself.

I also agree with your point about independent TLS implementations, and that choice is important here! I see this proposal as deepening user choice, since it doesn’t preclude CPython continuing to use OpenSSL but also provides a reusable abstraction that other OSS (or vendor) TLS stacks can satisfy. Per above, IMO it’d be a “win” for TLS in CPython even if no non-OpenSSL backends emerge, solely because a fixed interface with fewer abstraction leaks will be easier to maintain and attract ongoing contributor attention to :slightly_smiling_face:

3 Likes