Pre-PEP discussion: revival of PEP 543

Core Development
61

Sorry for the delayed response here – I took a bit of a vacation after PyCon :slightly_smiling_face:

I think there are two next steps here:

  1. @jvdprng and I need to go through the discussion here, and file issues for various ideas/API surfaces discussed, including evaluating the feasibility of mapping the current interfaces to native TLS APIs like SChannel.
  2. One discussion that came out of PyCon was the importance/priority of a new “thick” TLS API, versus a “thin” PEP for just system trust stores, i.e. something closer to @sethmlarson’s truststore. I think there’s a lot of separate value in providing independent TLS APIs, but a strong argument has been made that starting with just trust stores is both easier and higher priority (in terms of resolving pip et al.'s long-standing root of trust problems).
2 Likes
62

The thing is, this “thin” API is really just an OpenSSL integration/hack/feature, so I don’t think it’s in any way a substitute for a proper API. My goal/hope is to drop dependence on OpenSSL entirely, making it something that users can pull in optionally if they need OpenSSL-specific APIs, but otherwise don’t need it at all for “normal” (probably mostly client) TLS.

I’m happy to scope down “normal TLS” to a smaller range of functionality, but just replacing OpenSSL’s core verification process with a native implementation isn’t a substitute. It’s also a good thing, and if people weren’t reliant on OpenSSL’s verification process already then I’d go ahead and do it by default, but for the sake of compatibility it has to somehow be opt-in anyway. So I’m not even sure it becomes less complicated than offering a totally new API.

5 Likes
63

Regarding the next steps, I am currently looking into a backend based on Windows SChannel to confirm that the API can be implemented by other backends.

Afterwards, I am planning to look into the other suggestions in this topic and try to implement them.

The eventual goal is still to submit a new PEP for this work.

3 Likes
64

I have been looking into the documentation, and encountered the following:

  • HCERTSTORE or hRootStore in the SCH_CREDENTIALS struct is valid for server applications only according to the documentation. It is meant to store self-signed root certificates for client auth.
  • Similarly, the CERT_CONTEXT array paCred is for specifying certificates that contain a private key to be used in authenticating the application. So neither of these can be used to set the trust store for clients if the documentation is to be believed.
  • The flags can be used to turn server certificate validation and hostname verification on and off, which will be useful to implement the insecure variant.
  • There is an array of TLS_PARAMETERS that can be used to configure the TLS protocols and cryptographic algorithms. Unlike the previous way of configuring this, these settings disable particular protocols and algorithms. The way to disable cryptographic algorithms is a bit cumbersome, as evidenced by the curl implementation where ~200 lines of code are required to choose the correct ciphers from the five TLS v1.3 ciphers.

My initial attempts at implementing a backend suggest that it is not completely straightforward to lift a working Schannel implementation of TLS v1.2 to work with TLS v1.3. According to this blog post you need to make sure that the code correctly handles SEC_I_RENEGOTIATE, which is interesting as TLS v1.3 does not have renegotiation. Even when handling it according to their instructions I cannot get message decryption to work with TLS v1.3. Did you have a chance to look into this, @steve.dower ?

65

Yeah, I haven’t had time to try anything yet. More than enough other work piling up :sweat_smile:

Seems believable. It looks like server credentials need to be validated manually (I didn’t see exactly where in the process you should do this, but I didn’t look too deep. Hopefully it’s pre-auth.)

What can I say, they want you to use the system defaults :smiley: The whole intent of this API is to move that configuration to the user/system level rather than having apps do it, and as a result, when an app wants to do it they have to work harder. This is partly why I’m keen to have less configuration in the API, and for “use the default” to be an option, rather than specifying the default ourselves.

I suspect this is a different renegotiation than what TLS may specify - it’s the same result code for all Schannel modules, so any time any of them need to refresh credentials/keys they’ll have to use it. It’s just part of the state machine, ultimately, which is why I think this whole area is complex enough that we want as much of a “do what I mean” API rather than a highly customisable one.

2 Likes
66

Yeah, this looks like it will be gross. It may be possible to do it during the handshake, but I have not tested this yet.

I agree, it will be a good idea to include system defaults as an option. For the current MVP I would even consider leaving out configuring the cipher suites until later.

Indeed, after some more digging I concluded that this is what you get when receiving a Session Ticket. The reason it is not working out of the box seems to be that you can receive more than one Session Ticket, and processing more than one using a single call to DecryptMessage seems to cause it to process the first one and mangle the second one. Since the second ticket is now lost, further calls to DecryptMessage will fail as it does not use the correct keys.

2 Likes
67

In terms of problems that a new TLS API needs to solve, “Application knows exactly what it wants to do, and wants full control over how it is done”, using OpenSSL with an application provided certificate bundle (i.e. the status quo) actually has that covered pretty well. Sure, it’s OpenSSL centric, but “bring your own TLS library” is the only real way to get that kind of control.

It’s the “Application wants to ask the OS to handle everything, and not worry about the details” use case that isn’t currently well served (because the OpenSSL layer in the standard library gets in the way).

So maybe the right way to think about this TLS API proposal is less as a “new” TLS API, and more as a “platform managed TLS API” that fits better into an increasingly more sandboxed world?

The existing TLS API would then be retained indefinitely as the “customisable TLS API” rather than just being the “old” TLS API.

Edit: looking at this way also aligns with seeing the updated API as an opportunity to better expose the OpenSSL provider APIs, since those are also about “make TLS someone else’s problem” where TPM hardware and software security modules are concerned.

2 Likes
68

Hey all,

I’ve cut a new release of the design, which includes a bunch of the feedback above that @jvdprng has addressed: tlslib · PyPI

We’re getting close to the end of the period we have scoped with the Sovereign Tech Fund to complete the PEP draft, so our next step (if I’m reading PEP 1 correctly) is to take our internal draft and make a public draft for review here. I’ll follow up with that shortly!

3 Likes
69

For this particular case, you can also go straight to submitting a PR to the PEPs repo and requesting assignment of a PEP number. That way the next discussion will be able to refer to it directly rather than describing it as the evolution of PEP 543.

I’d be happy to be listed as the PEP sponsor (I’m sure you could find a few willing sponsors for this, but you only need one).

4 Likes
70

I’ve created a draft PR containing the body of the PEP here: pep-9999: A Unified TLS API for Python by woodruffw · Pull Request #3853 · python/peps · GitHub

@jvdprng and I need to clean it up a bit and fill in the code-blocks (they’re all in tlslib and just need to be copied over), but I’ll undraft once that’s complete!

2 Likes
71

Hello there,

We’ve been talking with @jvdprng over the github issues of tlslib and he suggested I write a message here :slight_smile:

I’ve been working on my own sans-io pure-python crypto-agnostic implementation of TLS 1.3 for the past two years (for fun): siotls. The latest stable-ish version (0.0.5) doesn’t implement this PEP and contains my own vision of what a TLS lib for python could look like.

I knew about PEP 543 but it seemed stale so I didn’t care to implement it. I discovered this revival not too long ago and I am eager to adapt siotls to use tlslib.

I’ve got many questions, I did read the whole forum but kinda diagonally, sorry if I’m asking anything that has been decided already.

Ok here it goes.

Socket Wrapping and Blocking Socket

There something I really like about the current ssl library, it is that it offers a wrap_socket function to wrap an existing socket. That’s something that doesn’t exist inside tlslib, the only two public functions are connect() and create_buffer(). Server-side the connect() function of tlslib.stdlib creates a non-blocking IPv4 + TCP socket. In my very first attempt, I tried to create an IPv6 server on ::1 which didn’t work because the underlying socket was AF_INET and not AF_INET6, see issue #68. The second thing I tried (using 127.0.0.1 instead) was to call accept() on the created TLSSocket object, I was expecting it to block until I could connect a client, but it instead rose an exception because socket had been set non-blocking by tlslib, see issue #71.

So in my first attempt, I got two assumptions wrong: that it would work with an IPv6 address, and that I could use it as a regular blocking socket.

I understand the desire to expose a simple interface to users. tlslib.stdlib could maybe do a dns lookup on the address to determine the socket family to use, but it won’t work with everything, e.g. Unix Domain Sockets. I don’t understand why it returns a non-blocking socket. Is it part of the spec? Is it for async? Aren’t async libraries gonna use TLSBuffer? What should people in a sync environment do? Is setting the socket blocking ok? Should we instead implement our own logic on top of TLSBuffer?

It is annoying that there is nothing to secure a regular blocking socket.

Configuration

In my own lib I have the following:

  • TLSConfiguration(side=...), unified for client and server
  • No Context factory
  • TLSConnection(config, server_hostname=None), which is TLSBuffer in tlslib terms
  • tls_connection.wrap(tcp_socket) that returns a object close to TLSSocket in tlslib terms

A ton of ValueErrors / warnings for everything erroneous / insecure.

The split client vs server and the context factories are nice. I wasn’t too fond at first, but they indeed make the user experience better, I’ll adopt the structure :smiley:

I’ve got a whole set of questions/remarks regarding the configuration.

Client vs Server certificate_chain

The TLSServerConfiguration object is similar to the client one, except that it takes a Sequence[SigningChain] as the certificate_chain parameter.

Why? I mean I understand why it should be Sequence[SigningChain] server-side, but why is it a single SigningChain client-side?

All the Nones

There something else I don’t understand, why all the Nones? I get it that a user might not want to decide what ciphers to instantiate the configuration with, to let the system decides on a sane default value, but why can the property returns None as well?

I have a pretty strong feeling that the two configuration classes should perform some sanity checks, like to assert lowest_supported_version <= highest_supported_version, or that the private key and certificate match (but that requires access to crypto primitives, so maybe not for the stdlib). I think it should also decide on sane default values when the user provides nothing.

Maybe the configuration should be abstract in tlslib, to let every implementation decides on its own checks and to possibly narrow down the returned types. We could have some checks in the base class for the easy stuff (e.g. lowest_supported_version <= highest_supported_version). It would mean users must import and instantiate the concrete TLS(Client|Server)Configuration class from the implementation they are using. Shouldn’t be a problem as this is already the case for (Client|Server)Context.

Remove default for client truststore and server certificate_chain

Unless we want to support TLS 1.2 unsafe ciphers DH_anon_*, the ServerCertificate.certificate_chain must be set and non-empty (it’s a requirement of TLS), it should be a mandatory parameter server-side.

The same goes for ClientCertificate.truststore, it is set None by default at the moment, making the connection insecure by default. It should be a mandatory parameter client-side. Users who wish for an insecure connection are free to explicitly set the param None.

Ciphers

At the moment the only cryptographic primitives that are configurable with tlslib are the cipher suites (AES). I don’t know for TLS 1.2, but in TLS 1.3 the key exchanges (DH/x25519) and signature schemes (RSA/ECDSA) are negotiated as well. TLS 1.3 can even use a configuration to select the peer’s public key, and another to select the peer’s certificate chain (not implemented in siotls yet).

Negotiated Options

At the moment all the negotiated options end up on the TLSBuffer/TLSSocket class as negotiated_... attribute. In siotls I’m using a dedicated class for all the negotiated options, which is saved as attribute of my TLSBuffer.

tlslib:

tlssocket.negotiated_protocol

siotls:

tlsconn.nconfig.alpn  # am renaming it inner_protocol, for // with tlslib Config

(there’s also the peer_certificate that is saved in my nconfig object, but I’ll move it on the socket, makes more sense to have it on the socket like tlslib does)

Like the configuration, I’ve got more negotiated stuff than tlslib, hence the dedicated object. I have no strong opinion but I fear there’s gonna be lots of negotiated_... attributes, just wanna open the discussion on what you think seems best?

Raw Public Key

siotls allows to setup a TLS connection between a client and server certificate-less using a public key server-side, and a list of trusted public keys client-side. That’s RFC 7250 and that’s pretty much like your everyday SSH server. I haven’t seen it deployed anywhere, but IMO it is great for IoT and outside the web.

For this to work with tlslib, I would need a PublicKey class pretty much like the existing PrivateKey one. There should also be an entry for a Collection[PublicKey] in the config, or could it be something in TrustStore? Same goes for SigningChain.

Anyway, I would like to open the discussion on this matter as well.

Certificate Chain, Certificate Revocation List

It seems that the current Certificate class only is about loading a single certificate. Certificate chains and certificate revocation lists (CRLs) are typically stored as multiple concatenated PEM-encoded certificates, how do we load those?

As a user, should I open and parse those files myself? Using Certificate.from_buffer for every PEM-certificate I find in the files? Or can I use Certificate.from_file directly?

Same goes for the other side, in siotls, should I expect a Certificate.buffer to hold multiple concatenated PEM certificates?

IMO there should be two other constructors:

@classmethod
def many_from_buffer(cls, buffer: bytes) -> Collection[Certificate]

@classmethod
def many_from_file(cls, path: os.PathLike) -> Collection[Certificate]

or maybe chain_from_buffer(...) -> Sequence[Certificate] and crl_from_buffer(...) -> Set[Certificate] (idem for from_file).

Please note that even if I have had my mind in TLS stuff for two years, I still don’t know if the leaf certificate appears first or last in a certificate chain. I expect other users will struggle with this as well. It would be nice if we don’t have to extract the leaf certificate from the chain ourselves for SigningChain, and just throw the fullchain and private key and let the class deal with it.

Save DER in-place

Is it ok if we change in-place the Certificate and PublicKey objects provided by the user to always guarantee buffer is set, and is DER? I am using asn1crypto in various places of siotls to extract information out of the cert/privkey, for doing sanity checks while loading the config, but also for communicating with backend cryptographic libraries. I don’t want to read and decore a PEM-encoded file every time I need to access this peer’s private key. At the moment I read and decode everything while instantiating the TLSConfiguration object and save it in-place in the certificate/private-key object. It is ok to do that? Should I make a copy instead?

This is my first time contributing to a PEP. Hopefully this message, the content and tone are appropriate, if not please tell me.

Thank you for this PEP. Let’s make something great!

Regards,
Julien

3 Likes
72

Hello again :slight_smile:

I’ve been reading the difference between TLS 1.2 and TLS 1.3 a bit and I can expand on my previous message.

Cipher Suites

In TLS 1.2 the “cipher suite” concept is a tuple (key exchange, authentication, encryption, hash) e.g. TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

TLS 1.3 reuse the name “cipher suite” and slot in ClientHello/ServerHello but the concept changed, it only is a tuple (encryption, hash) e.g. TLS_AES_256_GCM_SHA384. The authentication and key exchange algorithms are negotiated via dedicated ClientHello/ServerHello extensions.

It means that there are two ways of configuring the same thing.

Either we do it (like now) the TLS 1.2 way, i.e. TLS(Client|Server)Configuration.ciphers holds suites like TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 and TLS 1.3 implementations (e.g. siotls) need to parse those to extract ECDHE with SHA384 => NamedGroup.secp384r1 and ECDSA with SHA384 => SignatureScheme.ecdsa_secp384r1_sha384. I’ve a feeling that’s what OpenSSL is doing.

Either we configure it like siotls, i.e. a list for each, and folks doing TLS 1.2 will need to combine them together to form the long cipher suite.

Or maybe we could support both, i.e. let users instantiate the configuration giving either TLS-1.2 long suites, either all the primitives, and have the other automatically computed inside __init__.

Close Notify

There is currently only one way to close a socket in tlslib and it is the following:

@abstractmethod
def close(self, force: bool = False) -> None:
    """Shuts down the connection and mark the socket closed.
    If force is True, this method should send the close_notify alert and shut down
    the socket without waiting for the other side.
    If force is False, this method should send the close_notify alert and raise
    the WantReadError exception until a corresponding close_notify alert has been
    received from the other side.
    In either case, this method should return WantWriteError if sending the
    close_notify alert currently fails."""

The way I read the docstring, I understand that we want to close both sides of the connection, sending and receiving ends. force=True basically is “don’t actively wait for the other to close the connection”.

This is a violation of TLS 1.3 which states the following (emphasis mine):

Each party MUST send a “close_notify” alert before closing its write side of the connection, unless it has already sent some error alert. This does not have any effect on its read side of the connection. Note that this is a change from versions of TLS prior to TLS 1.3 in which implementations were required to react to a “close_notify” by discarding pending writes and sending an immediate “close_notify” alert of their own. That previous requirement could cause truncation in the read side. Both parties need not wait to receive a “close_notify” alert before closing their read side of the connection, though doing so would introduce the possibility of truncation.

https://tlswg.org/tls13-spec/draft-ietf-tls-rfc8446bis.html#section-6.1-4 (8446bis is rfc8446 (TLS-1.3) with all merged erratas, a nice navigation, and a way to pin-down a specific paragraph)

Only fatal alerts in TLS 1.3 ought to close both sides of the connection, and just discard all further messages with no other alerts. Sending a close notify only means we must close the sending end, and receiving a close notify only means we must close the receiving end.

I’ve been trying to think of a solution, but it is getting late (2AM). All I can say is that I’m pretty confident the current close() will prove problematic.

Regards,
Julien

73

Hello,

To expand further on the cipher suites difference between TLS 1.2 and TLS 1.3.

In TLS 1.2 there exists no cipher that combine DHE (key-exchange) and ECDSA (signature), but such a combination is possible in TLS 1.3: (NamedGroup.ffdhe2048, SignatureScheme.ecdsa_secp256r1_sha256).

This means that if we keep the current configuration scheme, i.e. a single entry for list a TLS 1.2-like cipher suites, we cannot express all the combinations possible under TLS 1.3.

The contrary, if we were to have 3 different entries in the configuration (key exchange, signature and encryption), then it’ll be possible to craft the TLS 1.2 cipher suite name for all combinations, and keep those that exist.

Regards,
Julien

1 Like