Smtplib/starttls/ssl do_handshake error in Python 3.7 but not Python 3.6

Hello all,

I am developing an application for use in my organization. In porting from a development server to a production server, I’ve run into problems using smtplib. The development server, which is running Ubuntu 18.04 and natively uses Python 3.6, works fine. The production server is running Debian Bullseye (the current testing distribution), which is shipped with Python 3.7. The issue is with the portion of the script that connects to my enterprise email server for the purpose of sending email notifications. The server, which is running Microsoft Exchange, requires authentication to access; which I initiate via a smtplib.starttls() call in the script.

On the development server, I am able to authenticate with the Microsoft Exchange server.

On the production server, I receive the following trace:

Traceback (most recent call last):
  File "./notify.py", line 470, in <module>
    server.starttls ()
  File "/usr/lib/python3.7/smtplib.py", line 771, in starttls
    server_hostname=self._host)
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

Turning on SMTP logging doesn’t reveal anything unusual:

send: 'STARTTLS\r\n'
reply: b'220 2.0.0 SMTP server ready\r\n'
reply: retcode (220); Msg: b'2.0.0 SMTP server ready'

(The next line, after the above, forms the start of the traceback reported above.) Based on the above, it would not appear to be an issue with the server since the server is responding identically in each case.

I’ve done the following to troubleshoot the issue:

  • Created new virtual machines for Ubuntu 18.04, Debian Stable, and Debian Testing
  • Followed identical steps to install the dependencies for my script

If the virtual machine ships with Python 3.6, smtplib.starttls() will succeed. If Python 3.7 is supplied by the distribution, smtplib.startls() will produce the error above.

I have tried setting up a virtual environment to make isolating the issue easier, but I’m not greatly experienced with them. After compiling Python 3.6 from source and setting up a virtual environment with it, the environment was still importing the OS-supplied version of the ssl.py library (i.e., the one in /usr/lib/python3.7/ssl.py). Presumably I need to separately download these library modules?

At this point I’m looking for advice on how best to isolate the issue so that I can either find a workaround, gain advice on how best to run a newer development version of Python to see if that solves the issue, or (if appropriate) file a bug report.

1 Like

I performed additional investigation. This script will reproduce the error:

import smtplib

SMTP_SERVER = 'XXXXX'
SMTP_USER   = 'YYYYY'
SMTP_PASS   = 'ZZZZZ'

server = smtplib.SMTP(SMTP_SERVER)
server.set_debuglevel (1)
server.starttls ()
server.login (SMTP_USER, SMTP_PASS)
print ("login successful")

I also found the “altinstall” flag and compiled and installed different versions of Python on one of the virtual machines and discovered that the version of Python has no impact on whether the script runs. Therefore it must be one of the C libraries called by Python.

I also installed Python 3.8.1 on Windows and found my script runs without error in that environment.

I thought I should update my post with this information before commencing an investigation into differences in C library versions. I am open to suggestions on what to compare.

1 Like

Sounds similar to https://bugs.python.org/issue31122

Maybe check OpenSSL version:

$ python -c 'import ssl; print("\nOPENSSL_VERSION: " + ssl.OPENSSL_VERSION + "\nOPENSSL_VERSION_INFO: " + repr(ssl.OPENSSL_VERSION_INFO) + "\nOPENSSL_VERSION_NUMBER: " + repr(ssl.OPENSSL_VERSION_NUMBER))'

It seems a workaround is to catch the exception and ignore it if exception.args == (0, 'Error') and ssl.OPENSSL_VERSION_INFO >= (1, 1).

1 Like

Thank you Peter, that bug does sound very similar.

I tried catching OSError and it results in an authentication error. I believe not successfully completing starttls () causes the library to continue without SSL. (The socket never gets wrapped, if I’m reading the smtplib starttls function correctly.) Since the next step is authentication and that is not being performed with encryption, the server rejects it (“SMTP AUTH extension not supported”).

I checked versions of software and interestingly, Windows and Debian Testing are running identical versions of TLS OpenSSL down to the raw version number. So perhaps the problem is with another library. I’m happy to keep digging.

As things currently stand, I can’t seem to catch the error and continue with TLS (unless I’m doing something wrong with that or missed something obvious), and I don’t yet know which library is causing the issue. I appreciate any other suggestions that any may have.

1 Like

For reference, both platforms (Windows and Debian Testing) report:

OPENSSL_VERSION: OpenSSL 1.1.1d  10 Sep 2019
OPENSSL_VERSION_INFO: (1, 1, 1, 4, 15)
OPENSSL_VERSION_NUMBER: 269488207L

For completeness, the test script completes in Windows and does not complete on Debian Testing.

1 Like

Issue 10808: ssl unwrap fails with Error 0 - Python tracker and
Issue 33808: ssl.get_server_certificate fails with openssl 1.1.0 but works with 1.0.2g for self-signed certificate - Python tracker and
Issue 31320: test_ssl logs a traceback - Python tracker and maybe others also mention ssl OSError with Errno 0 and some possible reasons:

This is an issue of cipher support

this condition happens when the local end calls unwrap(), but the low-level socket connection has already been shut down from the remote end

you tend to get this when the connection is shut down insecurely at the TCP level

Thank you Peter. I read through all the links you sent. The most promising idea therein is that the two ends can’t agree on a cipher. To pursue this further, I learned that I can run openssl from the terminal to test starttls. I also learned I can essentially get openssl to list what ciphers the email server supports using this command:

openssl s_client -starttls smtp -connect XYZ -security_debug_verbose

Along the way, I found a script:

https://33hops.com/send-email-from-bash-shell.html

That sends an email using STARTTLS from the terminal by calling openssl. This script, too, fails on the same platforms as the Python script, and successfully sends an email from the platforms that Python script works on; so the problem appears to be in openssl (whether configuration or otherwise). In fact, the error is right there in the output of the bash script: write:errno=0

I’m currently looking for openssl related posts that discuss write:errno=0. I’m also trying to figure out whether I can configure openssl to work. Listing supported cyphers using:

openssl enc -ciphers

Produces identical output (verified with diff) on the working and non-working Linux platforms, so just picking out a new cipher to try doesn’t seem like it would be fruitful.

I appreciate your help thus far, Peter.

1 Like

On a hunch (based on noting that one of the posts I saw today said there was an issue with TLS version 1.2 generating the OSError), I decided to try different versions of TLS and was successul in sending an email from Debian using essentially this command (borrowed from the script linked in my last post):

openssl s_client -tls1_1 -starttls smtp -connect ${smtpsrv}:${smtpport} 

To try to get my workaround working, my Python script now looks like this:

import smtplib
import ssl

# Create context (to specify TLS version)
sc = ssl.create_default_context ()
sc.options |= ssl.OP_NO_TLSv1_2 | ssl.OP_NO_TLSv1_3
sc.minimum_version = ssl.TLSVersion ["TLSv1_1"]

server = smtplib.SMTP(SMTP_SERVER)
server.set_debuglevel (1)
server.starttls (context=sc)
server.login (SMTP_USER, SMTP_PASS)
print ("login successful")

I had a certificate problem as well, which I got around by specifying:

sc.check_hostname = False
sc.verify_mode = ssl.CERT_NONE

I will re-visit these security bypass provisions later.

As of this writing, I have a workaround which is specifying TLS 1.1 in my script, and also allowing it. The above hopefully will serve to guide the next person who runs into this issue. I appreciate the help, Peter.

1 Like