Sorry for the delayed response here – I took a bit of a vacation after PyCon
I think there are two next steps here:
@jvdprng and I need to go through the discussion here, and file issues for various ideas/API surfaces discussed, including evaluating the feasibility of mapping the current interfaces to native TLS APIs like SChannel.
One discussion that came out of PyCon was the importance/priority of a new “thick” TLS API, versus a “thin” PEP for just system trust stores, i.e. something closer to @sethmlarson’s truststore. I think there’s a lot of separate value in providing independent TLS APIs, but a strong argument has been made that starting with just trust stores is both easier and higher priority (in terms of resolving pip et al.'s long-standing root of trust problems).
The thing is, this “thin” API is really just an OpenSSL integration/hack/feature, so I don’t think it’s in any way a substitute for a proper API. My goal/hope is to drop dependence on OpenSSL entirely, making it something that users can pull in optionally if they need OpenSSL-specific APIs, but otherwise don’t need it at all for “normal” (probably mostly client) TLS.
I’m happy to scope down “normal TLS” to a smaller range of functionality, but just replacing OpenSSL’s core verification process with a native implementation isn’t a substitute. It’s also a good thing, and if people weren’t reliant on OpenSSL’s verification process already then I’d go ahead and do it by default, but for the sake of compatibility it has to somehow be opt-in anyway. So I’m not even sure it becomes less complicated than offering a totally new API.
Regarding the next steps, I am currently looking into a backend based on Windows SChannel to confirm that the API can be implemented by other backends.
Afterwards, I am planning to look into the other suggestions in this topic and try to implement them.
The eventual goal is still to submit a new PEP for this work.
I have been looking into the documentation, and encountered the following:
HCERTSTORE or hRootStore in the SCH_CREDENTIALS struct is valid for server applications only according to the documentation. It is meant to store self-signed root certificates for client auth.
Similarly, the CERT_CONTEXT array paCred is for specifying certificates that contain a private key to be used in authenticating the application. So neither of these can be used to set the trust store for clients if the documentation is to be believed.
The flags can be used to turn server certificate validation and hostname verification on and off, which will be useful to implement the insecure variant.
There is an array of TLS_PARAMETERS that can be used to configure the TLS protocols and cryptographic algorithms. Unlike the previous way of configuring this, these settings disable particular protocols and algorithms. The way to disable cryptographic algorithms is a bit cumbersome, as evidenced by the curl implementation where ~200 lines of code are required to choose the correct ciphers from the five TLS v1.3 ciphers.
My initial attempts at implementing a backend suggest that it is not completely straightforward to lift a working Schannel implementation of TLS v1.2 to work with TLS v1.3. According to this blog post you need to make sure that the code correctly handles SEC_I_RENEGOTIATE, which is interesting as TLS v1.3 does not have renegotiation. Even when handling it according to their instructions I cannot get message decryption to work with TLS v1.3. Did you have a chance to look into this, @steve.dower ?
Yeah, I haven’t had time to try anything yet. More than enough other work piling up
Seems believable. It looks like server credentials need to be validated manually (I didn’t see exactly where in the process you should do this, but I didn’t look too deep. Hopefully it’s pre-auth.)
What can I say, they want you to use the system defaults The whole intent of this API is to move that configuration to the user/system level rather than having apps do it, and as a result, when an app wants to do it they have to work harder. This is partly why I’m keen to have less configuration in the API, and for “use the default” to be an option, rather than specifying the default ourselves.
I suspect this is a different renegotiation than what TLS may specify - it’s the same result code for all Schannel modules, so any time any of them need to refresh credentials/keys they’ll have to use it. It’s just part of the state machine, ultimately, which is why I think this whole area is complex enough that we want as much of a “do what I mean” API rather than a highly customisable one.
Yeah, this looks like it will be gross. It may be possible to do it during the handshake, but I have not tested this yet.
I agree, it will be a good idea to include system defaults as an option. For the current MVP I would even consider leaving out configuring the cipher suites until later.
Indeed, after some more digging I concluded that this is what you get when receiving a Session Ticket. The reason it is not working out of the box seems to be that you can receive more than one Session Ticket, and processing more than one using a single call to DecryptMessage seems to cause it to process the first one and mangle the second one. Since the second ticket is now lost, further calls to DecryptMessage will fail as it does not use the correct keys.
In terms of problems that a new TLS API needs to solve, “Application knows exactly what it wants to do, and wants full control over how it is done”, using OpenSSL with an application provided certificate bundle (i.e. the status quo) actually has that covered pretty well. Sure, it’s OpenSSL centric, but “bring your own TLS library” is the only real way to get that kind of control.
It’s the “Application wants to ask the OS to handle everything, and not worry about the details” use case that isn’t currently well served (because the OpenSSL layer in the standard library gets in the way).
So maybe the right way to think about this TLS API proposal is less as a “new” TLS API, and more as a “platform managed TLS API” that fits better into an increasingly more sandboxed world?
The existing TLS API would then be retained indefinitely as the “customisable TLS API” rather than just being the “old” TLS API.
Edit: looking at this way also aligns with seeing the updated API as an opportunity to better expose the OpenSSL provider APIs, since those are also about “make TLS someone else’s problem” where TPM hardware and software security modules are concerned.
I’ve cut a new release of the design, which includes a bunch of the feedback above that @jvdprng has addressed: tlslib · PyPI
We’re getting close to the end of the period we have scoped with the Sovereign Tech Fund to complete the PEP draft, so our next step (if I’m reading PEP 1 correctly) is to take our internal draft and make a public draft for review here. I’ll follow up with that shortly!
For this particular case, you can also go straight to submitting a PR to the PEPs repo and requesting assignment of a PEP number. That way the next discussion will be able to refer to it directly rather than describing it as the evolution of PEP 543.
I’d be happy to be listed as the PEP sponsor (I’m sure you could find a few willing sponsors for this, but you only need one).
@jvdprng and I need to clean it up a bit and fill in the code-blocks (they’re all in tlslib and just need to be copied over), but I’ll undraft once that’s complete!
The whole intent of this API is to move that configuration to the user/system level rather than having apps do it, and as a result, when an app wants to do it they have to work harder. This is partly why I’m keen to have less configuration in the API, and for “use the default” to be an option, rather than specifying the default ourselves.