I think there are two additional flags that (likely) make sense to include as defaults:
VERIFY_X509_STRICT: this performs more strict X.509 certificate validation, which is what users probably generally expect (insofar as the average user is probably talking to TLS server that’s supposed to be conforming with the CA/B Basic Requirements anyways).
VERIFY_X509_PARTIAL_CHAIN: despite the confusing name, this flag makes OpenSSL’s path building behave more consistently (and more like other path building implementations): it terminates the chain building process as soon as a trust anchor is encountered, rather than continuing to search the trust store for a self-signed trust anchor.
Together, these additional flags would make the ssl module’s default path building/certificate validation rules more consistent with how Go, GnuTLS[1], etc. behave. At the same time, making them stricter shouldn’t affect the vast majority of users, since the Web PKI already enforces (as a matter of policy) the things OpenSSL considers “strict.”
I’m raising this to get opinions from others, particularly with respect to compatibility/consequences that I might be oblivious to.
CC @sethmlarson, who may have opinions about this as the SecDevInRes
[1]: This is not otherwise an endorsement of these implementations; just that they perform my default the standard behavior that OpenSSL chooses to tuck behind VERIFY_X509_PARTIAL_CHAIN
Changing them in 3.13 is possible if the new defaults make more sense. I’m interested to hear what might be considered the TLS certificate verification strictness best practices here from someone deep in the TLS world such as @davidben.
My own presumption is that our certificate verification default behaviors would ideally match what major browsers like Chromium and Firefox do?
This isn’t exhaustive, but from a quick look: without X509_V_FLAG_X509_STRICT (the equivalent on the OpenSSL C API side), OpenSSL fails to enforce the basicConstraints, keyUsage, and issuer/subject/SAN rules (among others) required by RFC 5280.
VERIFY_X509_PARTIAL_CHAIN makes sense to me, and if the user-perceived difference when that default is changed is potentially shorter trust chains that feels like an acceptable difference to have without a deprecation period? (Would that only be accessible through the private certificate chain APIs on SSLObject?)
I am worried about unintended breakages for VERIFY_X509_STRICT, it’s likely totally fine for public-facing websites for the reasons you mentioned but enterprise configurations are more varied having experience with changing default certificate verification behaviors in the past. Is there a way we can detect when a given certificate would have caused problems and emit a DeprecationWarning or something like this for folks so we can get an early signal to what magnitude of breakage we can expect from changing the default there?
Changing default is always scary. What would be the option for users affected by stricter default? Is there a way to configure the Python crypto policy?
Fedora has a nice system-wide update-crypto-policies command which can be used by the admin for that:
When RHEL 7 was modified to enable HTTPS certification validation, a new configuration was added to give the choice to opt-in for the old unsafe default (don’t check certificates):
I vaguely recall that when we fixed the stdlib to prevent sending invalid HTTP requests, we added an opt-in API for people who want to send invalid HTTP requests on purpose. But I failed to find a reference to that
In order to do this, I think we’d effectively need to run every X.509 path validation twice: once with the flag, and once without. That’s probably possible since CPython controls all of the inputs here, but I have no idea whether it’s advisable or not – it’d be a bit of a performance hit and we’d need to be careful about ownership/not misusing OpenSSL’s APIs across two verification contexts.
Those users could explicitly configure a weaker set of SSLContext.verify_flags, e.g.:
If by “crypto policy” you mean something more global than that (e.g. an environment variable that sets the verify_flags), then I don’t believe that currently exists (but I might be wrong!)
IMO adding that kind of global switch here might cause more problems than it solves: even users who need weaker X.509 verification settings probably don’t need them on a per-interpreter basis, but probably only need them for a specific handful of uses of ssl. Giving them a big flag that makes the error go away might result in a lot of people opting into a worse overall security posture for just a single use of ssl that actually needs it
So if you tried the more secure way first and it worked, you wouldn’t
have to do it twice. But if you got a validation error, you could try
it again with the current default settings, and report the warning?
The protocol, options, cipher and other settings may change to more restrictive values anytime without prior deprecation. The values represent a fair balance between compatibility and security.
In other words, create_default_context was designed to represent an evolutive default choice adequate for common TLS usage. It is perfectly fine to change those defaults to stricter values in a new CPython release, and no deprecation period is needed.
And as the doc goes on to say:
If your application needs specific settings, you should create a SSLContext and apply the settings yourself.
So this is semi-complicated, but it is one potential way we could offer a deprecation period that folks could opt-in to non-strict verification or be alerted to their non-standard certificate situation. This explanation leaves out the difficulty in implementing the behavior of OpenSSL’s strict verification, I looked briefly but didn’t do a deep-dive into feasibility there. Folks that know more should weigh in on this aspect.
During the deprecation period:
Track whether SSLContext.verify_flags has been set after being put into user control (ie, after ssl.create_default_context())
On the handshake do a normal path validation without VERIFY_X509_STRICT enabled.
After building a chain, if SSLContext.verify_flags hasn’t been modified since creation:
Inspect the certificates in the chain for the conditions where OpenSSL would have failed on if VERIFY_X509_STRICT were enabled.
Emit a DeprecationWarning if any of those cases are found with links to documentation.
Documentation shows way to suppress the deprecation (ie SSLContext.verify_flags &= ~ssl.VERIFY_X509_STRICT which would unset the “unmodified verify_flags” flag on SSLContext)
Then after the deprecation period we can add VERIFY_X509_STRICT as a default.
Based on the change notes below that, it looks like quite a few other “breaking” changes have been made to create_default_context under the same policy (e.g., dropping 3DES).
Given that, IMO it makes sense to add VERIFY_X509_STRICT with no deprecation period: it’s the same sort of explicitly allowed “breaking” change.
Again, changing that would break a lot of projects, but we should do it at some point.