Is is possible to use CGI with HTTPS?

Python Help
1

I have a CGI script that works fine with the built-in HTTP server (run as python -m http.server --cgi). However, when I try to use the same CGI handler in a server that supports HTTPS, I get a TLSV1_ALERT_RECORD_OVERFLOW.

I have not been able to find any information about what this error is or what it might mean. I can’t find any examples of Python’s CGI being used over HTTPS either, so I don’t even know if they can be used together.

This is the CGI script that I’m trying to run:

#!/bin/bash
inotifywait --event modify .
echo Content-type: text/plain
echo
echo Watched file modified

This is the error the server throws:

127.0.0.1 - - [05/Feb/2023 16:30:49] "GET /cgi-bin/inotifywait HTTP/1.1" 200 -
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 57506)
Traceback (most recent call last):
  File "/usr/lib64/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python3.10/http/server.py", line 651, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/lib64/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/lib64/python3.10/http/server.py", line 425, in handle
    self.handle_one_request()
  File "/usr/lib64/python3.10/http/server.py", line 413, in handle_one_request
    method()
  File "/home/lunariel/.local/lib/python3.10/site-packages/uploadserver/__init__.py", line 219, in do_GET
    else: http.server.CGIHTTPRequestHandler.do_GET(self)
  File "/usr/lib64/python3.10/http/server.py", line 655, in do_GET
    f = self.send_head()
  File "/usr/lib64/python3.10/http/server.py", line 993, in send_head
    return self.run_cgi()
  File "/usr/lib64/python3.10/http/server.py", line 1158, in run_cgi
    if not self.rfile.read(1):
  File "/usr/lib64/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib64/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib64/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: TLSV1_ALERT_RECORD_OVERFLOW] tlsv1 alert record overflow (_ssl.c:2548)

This is a minimal server that reproduces the issue (run with no arguments and it serves HTTP and the CGI works, pass a .pem file in and it uses HTTPS and the CGI fails):

import http.server, http, sys, ssl, socket

class ThreadingHTTPServer(http.server.ThreadingHTTPServer):
    def server_bind(self):
        bind = super().server_bind()
        if len(sys.argv) > 2:
            context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
            context.load_cert_chain(certfile=sys.argv[2])
            self.socket = context.wrap_socket(self.socket, server_side=True)
        return bind

http.server.test(
    HandlerClass=http.server.CGIHTTPRequestHandler,
    ServerClass=ThreadingHTTPServer,
    port=8000,
    bind=None,
)

I tried removing the inotifywait call and using a CGI script that just sent a text string, but that had the same error. The server is able to server regular files (.html, .js, etc.) over HTTPS, so I don’t think it’s a certificate issue.

2

What is the OS?
I see its python 3.10
What is the version of openssl?

What program do you use to access the server? Browser? curl?
What is the URL used?

3

After checking, I found the OpenSSL version was out of date. Trying with a current version resulted in a new error. Here’s the details:

Original test system:
Fedora 35, Python 3.10.5, OpenSSL 1.1.1q

Second test system:
Fedora 37, Python 3.10.9 and 3.11.1 (tried both), OpenSSL 3.0.5

On both systems, I accessed the server via localhost:8000. I did most testing with Firefox, though I also tried Chrome, mobile Safari, and curl.

Additionally, I simplified the CGI script down to:

#!/bin/bash
echo Content-type: text/plain
echo
echo Hello

(at the path cgi-bin/test).

In all cases, the server worked correctly over plain HTTP (using python -m http.server 8000 --cgi. The directory listing at http://localhost:8000/ was sent correctly, and requesting http://localhost:8000/cgi-bin/test received the hello message.

In all cases, the directory listing at https://localhost:8000/ was served correctly over HTTPS using the Python script above (after clicking through the expected warning about self-signed certificates).

However, when accessing https://localhost:8000/cgi-bin/test via HTTPS, several different errors occurred depending on setup:

  • On the Fedora 35 system, opening https://localhost:8000/cgi-bin/test with a browser caused the browser to display an SSL error SSL_ERROR_RX_RECORD_TOO_LONG, and the server to throw the error above.
  • On the Fedora 37 system, opening https://localhost:8000/cgi-bin/test with a browser caused the browser to display a generic “Secure Connection Failed” message. The server did not report an error.
  • On both systems, requesting https://localhost:8000/cgi-bin/test with curl https://localhost:8000/cgi-bin/test -k returned curl: (56) OpenSSL SSL_read: error:1408F10B:SSL routines:ssl3_get_record:wrong version number, errno 0. On the Fedora 35 system the server additionally reported another stack trace ending with ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:2548).
4

Thanks for the details. I just tried to repoduce what you are doing and know why it does not work.

The code you are running is an HTTP server not a HTTPS server!
Notice that the documentation does not mention HTTPS anywhere.

You cannot just connect to port 8000 and try to use TLS.
That does not work as the server has no idea that is what you wish to do.
There is no message defined in HTTP that allows you to upgrade to HTTPS.

I did find this link Simple HTTPS server in python - No One Is Perfect that does one way to do HTTPS.
I have not tested the code shown.

The errors you are seeing are the TLS code in curl trying to intepret the ASCII HTTP headers as the binary TLS protocol and failing

5

On fedora I use apache httpd to serve files and run cgi for http and https.

6

The http.server module I used as a CGI test.

The HTTPS serving is done by the Python script I linked earlier (see the context.wrap_socket() call. It works fine for serving regular documents, but breaks on CGI.

7

I will try your original code later and report what I find.

8

I will look at your code, but I test all my CGI using httpd, you could do the same?

9

I took a look at Apache and it looks like a bit of a project to set it up. I’ll post my results if I find time to try it.

In any case, the CGI script I tested worked over HTTP, so I don’t think it’s an issue with the script.

10

Setting up httpd for simple stuff is not that complex.
Its off topic for here but you can get help for fedora on its user list.
https://lists.fedoraproject.org/archives/

11

Your original code does not start an HTTPS server.
Its still a HTTP server.

I do suggest that you use httpd - ask on the fedora user list for help setting it up if you do not guides a search.

12

It is an HTTPS server:

image
image697×380 35.9 KB

HTTP requests to this server are rejected, but HTTPS requests are accepted. It works fine for serving static files over HTTPS. It’s just CGI scripts that don’t work.

13

Sorry it is not. You are only replacing some of the HTTP with HTTPS.
Start the server that browser to http://localhost:8000/ it will list the files in the current directory.

14

Ah, I found your confusion: It reads the certificate from the second argument (even though the first argument is ignored). I think that’s because I copied it from some other code.

Supply a certificate and it serves HTTPS.