Python maillibs don't verify server certificates by default, which is documented behavior, but several open source projects failed to do this right and I like to see this fixed

The Python mail libraries such as smtplib, imaplib and pop3lib do not verify server certificates per default, when a client based on these mail libraries connects to a server via TLS. Any server certificate is accepted per default. This means, a client can’t ensure that it connects to the server to which the connection was intended. This allows an active attacker in a machine-in-the-middle postion to intercept communication, read mail contents, credentials or may abuse an SMTP server for spamming.

While it is documented behavior, several open source projects failed to recognize this. I reported this to the projects, I found and wrote a blog post about this problem.

It would be good to change the default behavior and activate certificate verification. Therefore I like to start a discussion here, if such a change is wanted at all, if it requires a PEP (second attempt, first attempt is in pull request 3537), or if it it is just a security bug that can be fixed without much discussion.

So, what do you think?

6 Likes

FWIW I think this is a good idea, but I don’t have time to sponsor. And a migration plan is needed (see the GitHub issue).

1 Like

I don’t think this is a good idea.

The mail system typically tries to use TLS when possible, but will fall back to plain text in case it can’t establish a TLS session (*).

By enabling certificate checks per default, it’s more likely that you’ll end up with an unencrypted connection, rather than an encrypted one which may be using a non-validating certificate (e.g. a self-signed one). You’d certainly not want the latter. Encryption with a non-validating certificate is much better than no encryption at all.

If you do know that the server does support TLS and provides compatible TLS settings, you can enable TLS with certificate checks by explicitly passing in an SSL context which does this (e.g. the default one used by the ssl module): ssl — TLS/SSL wrapper for socket objects — Python 3.12.1 documentation

(*) The most common (and recommended) configuration for popular mail servers such as Postfix, still is to use opportunistic TLS: https://www.postfix.org/TLS_README.html and Postfix Configuration Parameters

2 Likes

I think one has to differentiate between mail clients to server and SMTP to other SMTP server communication. I am not aware of any real-world MUA which silently falls back to clear-text communication and I think it should not be used then. Even if a self-signed cert is accepted, the MUA should warn, when the cert changes to another cert that does not refer back to a known trust anchor.

4 Likes

Is it possible today to prevent the fallback?

If an ssl context is used does that disable the fallback?

If a TLS socket is used, it does not fallback automatically to clear-text. It must be programmed to do this. This is independent from the an SSL/TLS context.

With SMTP server to server communication, the MTA connects via clear-text and tries to level up to TLS via STARTTLS, if the target server does not support this, then there is no TLS context. So, “fallback” is maybe misleading here and can be read as “no-upgrade”.

Then my question becomes is it possible to ensure that STARTTLS must work or do not continue?

I’d like to summarize (STARTTLS in SMTP):

  • server to server: not continuing after an unsuccessful STARTLS means in many cases not delivering the message to the recipient
  • client to server: The SMTP protocol itself makes sure the communication will not continue if STARTTLS fails (unless the server is misconfigured)

In detail:

The privacy of e-mail messages must be ensured by end-to-end encryption. The SMTP infrastructure does not guarantee this. Messages on servers are stored on disks (SMTP is a store and forward protocol). Messages in transit from server to server can be protected by STARTTLS on a best-effort basis as mentioned already.

In the client to server case, i.e. during the initial message submission is the STARTTLS important for login credentials’ protection. In a typical setup a client is permitted to submit a new message:

  • if it has a trusted IP address (“trusted” can mean localhost only or also an office LAN or VPN).
  • or if it authentificates itself with its name/password. In this case a normally configured server will not allow a login before STARTTLS. A server responds to initial EHLO with its capabilities and AUTH is omitted there. Only after a successful STARTTLS a second EHLO takes place and AUTH becomes enabled.

Default configurations should either do the right thing or at a minimum warn users when insecure compromises are being made. Especially when choosing between two APIs: IMAP and IMAP_SSL, you would expect having made the choice for “SSL” that it would be secure, I can see having this distinction “catching” folks expecting that after making this choice that they’re good to go.

My preference is to switch the default SSLContext to verify, even knowing that this will cause breakage, if only because we made the distinction in the APIs.

My thoughts and questions on this:

  • Expand this proposal to include all insecure default standard library uses of SSLContext, not only email modules. This would add ftplib to the list of imaplib, poplib, smtplib.
  • Do we think complexity like “fallbacks”, retries, or behavior distinctions between local connections/remote connections belong in the standard library? If not, then we have fewer options for a graceful migration period.
  • Maybe we add configuration options to the classes that allow disabling cert/hostname verification without instantiating your own SSLContext? I know it’s changing a few lines of code into a parameter being set, but might have a positive effect on folks keeping TLS enabled which seems like our primary concern with this migration? Certainly makes examples less daunting.
  • Provide examples in the docs for each module for common issues folks might run into and how to keep TLS enabled while making things “work” (in addition to the caveats and linking to “Security Considerations”).

Looking at how other languages handle SMTP in particular, Go goes even further by disallowing authentication on non-encrypted channels and default TLS configuration verifies certificates. Ruby’s SMTP library also verifies certificates by default.

3 Likes

+1 Agreed, as long as we explicitly document the way for people to ask the variety of protocol APIs to opt out of certificate verification (in a way that’d work with existing library versions today, so they can write code compatible with all versions) we should be able to go forward with cert verification changes without a deprecation period given its security nature.

We did this belatedly with https TLS verification by default in PEP 476 – Enabling certificate verification by default for stdlib http clients | peps.python.org due to the protocols importance. We’ll likely find that it is time to do it for the others.

  • Do we think complexity like “fallbacks”, retries, or behavior distinctions between local connections/remote connections belong in the standard library? If not, then we have fewer options for a graceful migration period.

This may depend on what the library APIs provide (I haven’t looked at all 4+ of them…), but it seems potentially reasonable to say that validation be skipped when no fully qualified server hostname was provided. So that people using raw IP addresses or local names continue to work as-is. (a warning could still be raised if we believe it would be seen by someone able to do anything about it)

2 Likes

I like to add that skipping verification on some cases is a dangerous. It could introduce new surprises. In the specific case of IP addresses, there could be IP addresses in a certificate as an alternative name, which allows a “hostname verification” now with IP addresses, i.e. if a check is skipped, then it would be yet another surprise. I would vote for a programmer-readable version, like verify_cert=False. Then it is pretty clear what the expectation is, but even passing a check-disabling TLS context would be fine, although less readable.

Over at the oss-security mailing list, Hanno brought up the topic of certificate verification and refers to RFC 8314 “Cleartext Considered Obsolete: Use of Transport Layer Security (TLS) for Email Submission and Access”. Section 5.3 requires:

MUAs MUST validate TLS server certificates according to RFC7817 and PKIX.

So, changing the default behavior would not only be a step towards more security, but also to more RFC-conformity.

RFC 8314 “Cleartext Considered Obsolete: Use of Transport Layer
Security (TLS) for Email Submission and Access”. Section 5.3
requires:

MUAs MUST validate TLS server certificates according to RFC7817
and PKIX.

So, changing the default behavior would not only be a step towards
more security, but also to more RFC-conformity.

MUA is an abbreviation for Mail User Agent (colloquially referred to
as a “mail client”), but SMTP is used by a variety of programs not
all of which are MUAs. RFC7817 is about submission of and access
to E-mail, so some specific use cases for the protocol (and hence
the smtplib module), so really the proposed change would make
certain kinds of programs written using that module RFC-compliant,
but that RFC is not necessarily relevant to all kinds of programs
smtplib might be used to create.

I’m in favor of smtplib growing the ability to verify certificates,
and even having that feature enabled by default, but RFC compliance
is a thin argument here because smtplib is not itself an MUA any
more than urllib is a Web browser.

Yes, I agree. My use-case in mind was writing a client, which is likely 90 % of what people do, but of course, it is not the only use-case.