Our future with OpenSSL

Core Development
61

Heh, it was a legit question rather than a point, but your response answers the question :slight_smile: Thanks.

62

One thing I’d love to see is integration with system certificate validation on all platforms, not just platforms that use OpenSSL as the system TLS stack. As a user I don’t particularly care if that’s done by wrapping a system TLS API in the ssl module or by teaching OpenSSL to access the system validation APIs. I do not yet have an opinion on what would be better for CPython, that amongst others depends on which option gives the least load on the development team.

Using the system CA store is likely non trivial, especially given features that don’t 100% overlap with how OpenSSL does things (e.g. trusting certificates for specific use-cases on macOS). And as someone noted earlier some system validation APIs might block, which is problematic in asyncio.

63

You can get system trust store integration on PyPI, if you’re using the ssl module from 3.10+: https://github.com/sethmlarson/truststore

4 Likes
64

Interesting. But this is something that should be part of the stdlib. In particular, on macOS users of our installers now have to manually install (and update) the certifi package to be able to use the ssl module because openssl doesn’t use the system trust store. That’s confusing for users, and may lead to users disabling certificate validation.

We could, and probably should, tweak the installers to automatically install this package but in the end that’s a stopgap measure.

2 Likes
65

The truststore module shows that it is possible to use the system trust store. Adding the feature in python would be better as C code?
Aside from developer time to write the code is there a technical reason that would stop this being added to python?

1 Like
66

Yeah, this is why I keep coming back to wanting to start managing parts of the stdlib as separately-updateable packages, like Ruby does (see also). We can’t put certifi in the stdlib because it needs regular updates. We could put truststore (or equivalent) in the stdlib, but even if someone shows up to do the work in time to ship it in 3.12, pip won’t be able to rely on it until 3.11 hits EOL in 2027 – and it’s entirely possible that by 2027 the truststore approach will be obsolete and we’ll be having this conversation again about whatever the new thing is.

OTOH if we could ship ssl patches through PyPI, independently from cpython releases, all these issues would be so much more tractable. Our overly-rigid stdlib is the root cause of a lot of problems.

4 Likes
67

In the alternative, the problem is that we’re in a situation where users have high degrees of flexibility in their PyPI package versions, and very little in their Python version. Combined with the need/expectations/desire to support large parts of the ecosystem on older Python versions (including, often, those no longer supported by the CPython team.)

I don’t know that this changes anything you said, but I think it’s worth drawing it out.

4 Likes
68

I don’t think this is a blocking issue. As packages like tomli or asyncio show, putting a backport for a new stdlib module on PyPI works pretty well.
And as importlib_resources or compileall2 show, an alternate package can provide newer features for older Python versions to those that need the new features or just want to test them.
(Of course, an alternate PyPI package works for new features/approaches/API only. Bugs, especially security ones, need to be fixed in stdlib too.)

If we don’t expect the thing we put in stdlib to completely break down by 2027 to the point where it’s unmaintainable/unimplementable, I think it’s reasonable to put it in.

69

Agreed. I created a dataclasses backport for 3.6, and it worked well. The only downside is that people thought it should get updates for features in 3.8+, but that wasn’t its purpose.

I can’t decide if it was a good decision to use the same name for the backport as the stdlib name. On the one hand it was confusing and the module could not be used in 3.7+, on the other hand no client code had to change when moving to newer pythons.

70

I wouldn’t hold up asyncio as a shining example. The version on PyPI that you linked to is 7 years old and yet we recently had a debate with a user who believed they needed it with a recent Python version.

1 Like
71

Looking at the documentation for the ssl module we already have support for loading the CA chain from the system store on Windows in SSLContext.load_default_certs. Someone would “just” have to implement this for macOS as well.

I might have some time to look into this around Christmas.

Btw. IMHO the fact that pip cannot rely on this functionality for a long time is no reason for not trying to find a solution for this issue.

1 Like
72

The problem with this is that it will only load already-cached certificates from the store. Windows downloads (and updates) CA certificates on demand, but OpenSSL does not know how to demand it. I regularly advise people who get SSL validation errors to visit the site in a browser first, which will ensure the necessary CA cert has been cached and will be found by that function.

The easiest way to fix this is to replace the entire OpenSSL certificate validation function with a call into the Windows one (WinVerifyTrust), though this of course breaks all the OpenSSL handling of its environment variables, and unfortunately leaves the parsing functions in use so we are still vulnerable to the regular series of misparsed certificate issues.

The truststore project (on Windows) appears to simply disable OpenSSL’s validation and injects its own (correct) call to the Windows API as part of wrapping the socket, not replacing OpenSSL’s validation function (so maybe it doesn’t work on renegotiation or something? maybe it’s totally sufficient, but I don’t know enough to be sure).

2 Likes
73

I don’t know if the “already-cached” problem is relevant for macOS. It would still be better to find a way to explicitly use system certificate validation because the system has features that might not map nicely onto OpenSSL’s CA model (such as being able to override the system policy for trusting certificates).

1 Like
74

Hey all, author of truststore here. Would be very happy with having truststore-like code merged into the standard library in addition to OS-native APIs for establishing a TLS connection and managing certificates. What’s the best way I can help move that forward?

4 Likes
75

Talk to me at PyCon US after the language summit. :sweat_smile: Otherwise, right now you’re looking at a PEP to propose a new module.

2 Likes
77

The probably doesn’t need to be a new module, but could likely be a new type in ssl or with some luck even be integrated into SSLContext.

My current plan is to spend some time on looking into the macOS bits next week and the week after, which will likely result in questions for @sethmlarson.

78

If there is a newer module, it’d be nice imo if we could call it tls

1 Like
79

Is there any possibility that TLS will be replaced by something else in the future? It might be better to avoid using the name of any specific library or specification.

80

extreme programming teaches that its a waste of effort writing code for future undefined requirements.

TLS it it for now and the foreseeable future snd has replaced ssl well and truly.

2 Likes
81

OpenSSL was almost called OpenTLS at the beginning of the project, so the name has been around for a while. And even the fanciest QUIC / HTTP 3 still uses TLS, so I doubt the name is going to disappear anytime soon (on the order of decades).