Pre-PEP discussion: revival of PEP 543

Core Development
41

Something that I’ve encountered in recent years that I believe should be in scope when developing a successor to PEP 543: running OpenSSL in engine mode (where a hardware TPM or software security module is handling the crypto operations, so private keys never appear in the address space of the application itself).

In some use cases (e.g. ISO 15118 Plug & Charge certification with some central CAs) keeping certs out of app memory is a certification requirement. At the moment, using Python in affected use cases is potentially problematic (third party libraries to enable engine mode do exist, but I haven’t seen a trust audit on any of them)

The cross-platform API being discussed here should be compatible with engine mode though, which could bring native support for TPMs and SSMs to the standard library.

5 Likes
42

It should be noted that ENGINE is deprecated. The preferred way to reference private keys in OpenSSL 3 with providers is a URI like (pkcs11-provider) pkcs11:token=xyz;object=SomeKey;type=private or (TPM2 provider) handle:XXXXXXXX.
Some providers (TPM2 pkcs11) also use PEM files with new labels e.g -----BEGIN TSS2 PRIVATE KEY----- or -----BEGIN PKCS#11 PROVIDER URI-----

Maybe the takeaway is that a str that names a private key isn’t necessarily a filename or if a file may not be PKCS#8?

1 Like
43

True, I had forgotten about the introduction of the provider API in OpenSSL 3+ (same general idea as engines, different details in how they’re used and how they interact with the OpenSSL library)

For the purposes of this pre-PEP, I think the key points would be:

  • only OpenSSL 3+ needs to be supported, older versions of OpenSSL can continue to rely on the existing Python TLS APIs
  • the OpenSSL backed implementation of the new API should be defined in terms of explicit algorithm fetching

That way hardware backed providers like TPM2 should be straightforward to enable via the client and server context objects without otherwise altering the way the API is used.

44

We have published an MVP on PyPI[1]. It includes:

  • The items mentioned by Will in the initial post
  • Wrapped buffers as requested
  • An insecure submodule that allows users to disable hostname verification and/or certificate validation for testing purposes. For talking to devices with self-signed certificates it is still recommended to create a custom TrustStore containing this certificate.
  • A partial example modifying asyncio to use this API instead of the stdlib

Any feedback is welcome!

By implementing wrapped buffers in tlslib, supporting asyncio is relatively straightforward [2] as shown by the example. However, the result may not fully support asyncio’s current users, as asyncio allows users to pass in their own SSLContext. Since the corresponding TLSContext is by design more restrictive than an SSLContext (in order to remove OpenSSL-specific concepts), the resulting asyncio is similarly more restrictive. Some examples:

  • Displaying information about peer certificates is not possible. The concept of a certificate is deliberately kept very high level in PEP 543 (it corresponds to a location where the implementation can get a certificate). Exposing the same dict that the ssl module provides is undesirable because the code for this is very complex, and it would force other backends to provide the same.
  • Compression is always disabled (it is a security risk and removed in TLS v1.3).
  • No insecure user choices unless using the insecure submodule.

Some notes on supporting other stdlib modules:

  • http uses wrapped sockets because a HTTPSConnection will first send a “HTTP CONNECT” message in the clear before starting TLS[3].
  • urllib could use the new API, but it uses the http library so http would need to use it as well.
  • ftplib similarly sends some FTP commands in the clear before wrapping the socket. It also makes use of unwrapping the socket to switch back to clear communication.
  • imaplib, nntplib,[4] poplib, and smtplib all use wrapped sockets to implement STARTTLS. In addition, they require the use of blocking sockets, which is currently not supported in the MVP.

  1. I noticed that the links to documentation and source code are currently broken. Please use the following links as a workaround until we fix it:

    https://trailofbits.github.io/tlslib.py/tlslib.html ↩︎

  2. There are a few open items, such as how to deal with ssl.SSLSyscallError, which we do not want to expose in the API if we can get away with it. ↩︎

  3. Connecting via TLS to the proxy first would remove the need for wrapped sockets ↩︎

  4. now removed from stdlib ↩︎

5 Likes
45

This looks great! I assume it’s all intended as part of the specified API right now, so if anything I refer to isn’t meant to be spec, just ignore it.

I wonder if we can get away with less specific enums, looking specifically at CipherSuite, NextProtocol and TlsVersion? There is likely to be system-wide configuration for these, and it may get complicated to resolve specific (even default) requests from the Python side with what the OS allows (well, complicated for the user who has to deal with a random TLSError that we raised :wink: ).

Off the cuff, I’d prefer an enum/factory with “system recommended settings”, “most secure settings”, “least secure settings” (and would seriously consider a noisy warning if the last one is ever used!) At the very least, DEFAULT_CIPHER_LIST should be its own enum value, so that the backend gets to decide it rather than having to deal with whatever default gets into the API.

I’d really like TrustStore, Certificate and PrivateKey to be initialisable from “arbitrary identifier”, and to not offer any kind of introspection (as a required API). This is to allow for things like specifying a certificate by subject or thumbprint/hash and letting the OS locate it in any of its trust stores - in some configurations, this can be the only option on Windows, and extracting any certificate (or a list of certs) from the system is disallowed.

TrustStore.system() would be likely to return a marker object (possibly None) rather than an actual trust store reference on Windows - something that the backend would identify and then handle totally differently from a custom-built trust store. That looks possible with this structure, but I mention it to make sure it remains that way :slight_smile:

Again, it seems implied now, but might be worth specifying that the properties on TLSClientConfiguration and TLSServerConfiguration only return values if they were provided in the initializer, and aren’t meant to retrieve the current settings from the backend. (If they are intended to retrieve settings from the backend, then my suggestion is to not require that, but they appear to be data carriers rather than something the backend would implement.)

In case it’s helpful, I mapped a few of the concepts in the structure to the SChannel APIs that I think would be used to implement them. For the most part, they seem pretty well aligned, which is great! Maybe you’ve already got the mapping somewhere.

On my final look at the current structure, I have a vague concern that we could accidentally push users into having a tricky time matching types, particularly if Certificate and TrustStore are meant to be backend-specific. I may be wrong, but it feels like we want pure data all the way up to ServerContext and ClientContext, and then the backend decides how to handle it.

(A specific example: I see that OpenSSLCertificate.from_buffer creates the temporary file and then returns an instance with reference to the filename. I think it’s better to keep the buffer at that point and have the _configure_context... functions create a file if needed. This way, Certificate can be a concrete type, possibly with a mechanism for smuggling extra key/values to a known backend.)

A hypothetical example: if we made a WindowsCertificate class that creates the CERT_CONTEXT immediately and loads the certificate in, that object is now functionally incompatible with certain backends, despite being type compatible (modulo all the type checking stuff that I just ignored). But a concrete Certificate that doesn’t load the certificate immediately would be fully compatible with either, and it removes the temptation to create incompatible types.

That got longer than I intended, I’ll leave it there for now. Generally, I think we’re on a good path though, and I’m getting excited to actually have a go at implementing this on the Windows side (though I won’t stop anyone else from doing it if they’re more excited).

5 Likes
46

A procedural note that I missed when first reviewing this thread (but picked up when seeing “updated PEP 543 proposal” in the description of tlslib): rather than attempting to make an in-place update to PEP 543, it is procedurally simpler to just write a new PEP (referencing PEP 543 as motivation and inspiration).

It’s a genuinely different proposal, even though it’s tackling the same problem that 543 was looking to address.

3 Likes
47

Looking at tpm2-openssl/docs/keys.md at master · tpm2-software/tpm2-openssl · GitHub and tpm2-openssl/docs/certificates.md at master · tpm2-software/tpm2-openssl · GitHub, wrapping a TPM or SSM based provider in OpenSSL 3+ imposes a similar requirement at least for private keys (so they can be loaded by handle or hardware NV index, rather than from a file).

While it’s less clear to me if tpm2-openssl itself supports the use of opaque certificates (all the examples given in the docs use regular PEM files for the public certificates), TPM hardware and software security modules certainly offer that feature.

48

Yes, thanks for calling this out! We’ve been referring to it as an “updated” PEP 543 during design/discussion internally, but we intend to submit it as an entirely new PEP once ready.

(I left the 543 references in since I figured it would be marginally less confusing than using 9999 as the placeholder :sweat_smile:)

1 Like
49

First of all, great that you’re doing this.

I have a question.

Do you mean that it’s currently not possible, or that by way of policy you don’t want it to be possible?

When I did a mutual TLS project some years back, I had to monkeypatch requests, to get it to show me the peer certificate, just so I could know the CN/DN of who I was talking to.

Would I run into the same problem with tlslib?

1 Like
50

Our thinking here was that exposing a certificate representation at the “tlslib” level is probably a a layering violation: there are a lot of different ways that different backends identify certificates, including solely by SPKI/serial/hash, etc. By exposing a uniform expectation that each backend is able to return an introspectable certificate object, we might run into situations where some backends aren’t sufficiently flexible (or have performance tarpits, since extracting the certificate can be expensive).

OTOH, for just the “end-entity” certificate that completes client or server auth, maybe it would be sufficient to expose bytes(certificate) or similar with the certificate’s raw DER? From there, individual applications that need to introspect the certificate could do so using e.g. cryptography.x509.

2 Likes
51

I think that would be reasonable. Given I have the bytes its then up to my code to keep up with how the certificate world changes.

Will there be a hook that allows me to examine the other ends cert before and veto if I do not like the it? (This might be too advanced a feature).

1 Like
52

That would be perfect. You are absolutely right, extracting information from the certificate is better left to dedicated tools.

I think you should consider introspectability an important use case. Debugging is always hard when cryptography is involved, because the vital clue you need in order to understand the problem tends to be encrypted. And when people run into hard to solve problems with the secure solution, they tend to turn to insecure solutions.

About that uniform expectation, why would it be uniform? When troubleshooting, I’m not dealing with the abstract superset of all possible connection types, I’m dealing with the one specific connection that was made a second ago, and I need to know all about it.

2 Likes
53

My initial thinking here was that this would be part of the client/mutual authentication APIs, but one of the problems with mutual auth is that people do it wildly differently in practice. So a fully generic examine hook might be too advanced of a feature, but there might be a way to make it more tractable (e.g. allowing people to subclass the configuration + define a method that matches the subject + SANs however they please?). I’d be interested in hearing more about common mutual auth use cases to understand if such an API would be useful :slightly_smiling_face:

I phrased that weirdly: the “uniform” expectation in question is that every potential backend has an internal representation of certificates that can be straightforwardly + efficiently dumped into a consistent Python object. I think this is probably true at a lowest common denominator level (i.e. DER), but probably not higher. So it’s not about the connection type, but about minimizing the public API’s commitment to data models that can’t easily/efficiently be fulfilled by different backends.

1 Like
54

I can’t speak for what is common, but I can describe the system that I made.

I use mutual TLS for logon: Clients connect to my company’s web service, using custom software that we provide. They configure the software with their own X.509 certificate, and we get mTLS with this certificate and the server certificate. The server software looks at the parameters of the client certificate, such as the CN, and uses it to restrict access - only those resources in an internal database that are tagged with this CN will be visible to the client.

Without access to the CN for identification, there would be no point to mTLS at all. It wouldn’t help with client authentication anyway, and we might as well use plain TLS.

1 Like
55

Do you check only the CN, or the SANs as well? The CA/B guidelines state that SANs are the authoritative source of identifying information for server certificates, but are unfortunately less prescriptive about client authentication.

(But regardless of the answer: thank you for this datapoint!)

2 Likes
56

I probably should, thanks for the tip. I’m identifying companies, not domain names, so looking at the DN made sense to me. But reading up on it, it looks like subjectAltName should always be consulted when present. I may have to revisit that.

This assumes external CA’s. I have used external CA’s in the past, but I’m moving towards using a private CA instead, and then, as the issuer, I control all the names and don’t have these problems. For that, TrustStore.from_pem_file in PEP 543 looks great to me: It seems a very straightforward way to set up your own private little universe, where you don’t have to deal with system trust stores. Also for unit tests.

Though, the PEP says “A given TLS implementation is not required to implement all of the constructors”, so it doesn’t seem guaranteed that there will always be any backend that implements TrustStore.from_pem_file, or how is that?

2 Likes
57

If you’re using a separate software or hardware security module (including OS crypto APIs), the trust store configuration might not be under the application’s control.

That’s OK though - if an application needs a feature that a given backend doesn’t provide, then that application won’t be compatible with that backend. There’s no getting around that, and the API doesn’t need to magically make such cases work, it just needs to try to make it fail in a way that makes it reasonably clear what the underlying problem is (i.e. the app needs a feature the backend doesn’t provide).

Introspection APIs may fall into the same category: there’s potentially value in saying “if your backend provides access to full certificates, expose them in this fashion”, even if providing access to full certificates in the first place is an optional backend behaviour.

2 Likes
58

It makes sense that the API has this flexibility. What I’m unclear on is what backend or backends a normal CPython install is to be expected to have out of the box, and what options they will implement. I presume OpenSSL is still going to be an included battery, at least in the foreseeable future? So there’s going to be an OpenSSL PEP 543 implementation from day one?

59

That was our thinking, yeah – one of the things that came up with the original (PEP 543) effort was that some OSes are either too opinionated about their TLS APIs or simply don’t have official ones, and so Python will likely need an OpenSSL implementation for those.

(This connects to one of the original two rationales: even if this doesn’t allow CPython to eliminate its OpenSSL dep, it does remove public APIs that are tied to OpenSSL implementation details that cause compatibility and usability issues.)

3 Likes
60

Now that the PoC exists, what are the next steps?