Self-signed ssl certs verification

I’ve generated a private key and a self-signed certificate 3 different ways: using command-line openssl, pyopenssl and pyca/cryptography. Then I’ve used them to create ssl context for a simple flask app.
The flask app itself runs fine, but when I try to send a file to it using requests.post(url, files={"file": my_file}, verify=my_cert), I get an ssl error:

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1007)'))

Trying to mess around with certificate extensions hasn’t solved the issue so far. Using verify=False would be equal to giving up.
The surprising part (to me) is that all the methods of self-signed cert generation lead to the same error: the cert being recognised as self-signed.

I can add further details if needed (python version, pyopenssl version, system ssl version and so on).

Any pointers’d be appreciated.

Here’s a function that creates the self-signed certificates:

from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
import datetime
import ipaddress

def create_self_signed_cert(cert_file, key_file, ip_address):
    # Generate private key - same as OpenSSL's -newkey rsa:4096
    key = rsa.generate_private_key(public_exponent=65537, key_size=4096)
    
    # Simple subject/issuer like OpenSSL
    subject = issuer = x509.Name([
        x509.NameAttribute(NameOID.COMMON_NAME, ip_address),
        x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Some Name"),
    ])
    
    # Build certificate - match OpenSSL's basic self-signed cert
    cert_builder = x509.CertificateBuilder().subject_name(
        subject
    ).issuer_name(
        issuer
    ).public_key(
        key.public_key()
    ).serial_number(
        x509.random_serial_number()
    ).not_valid_before(
        datetime.datetime.utcnow()
    ).not_valid_after(
        datetime.datetime.utcnow() + datetime.timedelta(days=365)
    )

    # Add the exact extensions that OpenSSL adds for server certs
    cert_builder = cert_builder.add_extension(
        x509.SubjectAlternativeName([
            x509.DNSName(ip_address),
            x509.IPAddress(ipaddress.ip_address(ip_address))
        ]),
        critical=False
    )
    
    # Basic Constraints marked as critical - this is a CA cert
    cert_builder = cert_builder.add_extension(
        x509.BasicConstraints(ca=True, path_length=None),
        critical=True
    )

    # Key Usage extension
    cert_builder = cert_builder.add_extension(
        x509.KeyUsage(
            digital_signature=True,
            content_commitment=False,
            key_encipherment=True,
            data_encipherment=False,
            key_agreement=False,
            key_cert_sign=True,
            crl_sign=True,
            encipher_only=False,
            decipher_only=False
        ),
        critical=True
    )

    # Extended Key Usage
    cert_builder = cert_builder.add_extension(
        x509.ExtendedKeyUsage([
            x509.oid.ExtendedKeyUsageOID.SERVER_AUTH,
            x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH
        ]),
        critical=False
    )
    
    # Subject Key Identifier
    cert_builder = cert_builder.add_extension(
        x509.SubjectKeyIdentifier.from_public_key(key.public_key()),
        critical=False
    )

    # Authority Key Identifier
    cert_builder = cert_builder.add_extension(
        x509.AuthorityKeyIdentifier.from_issuer_public_key(key.public_key()),
        critical=False
    )
    
    # Sign with SHA512 
    cert = cert_builder.sign(private_key=key, algorithm=hashes.SHA512())
    
    # Write certificate and key in PEM format
    with open(cert_file, "wb") as f:
        f.write(cert.public_bytes(serialization.Encoding.PEM))
    
    with open(key_file, "wb") as f:
        f.write(key.private_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PrivateFormat.TraditionalOpenSSL,
            encryption_algorithm=serialization.NoEncryption()
        ))

create_self_signed_cert("selfsigned.crt", "private.key", "192.168.1.172")

(the whole experiment is setup in a local network)

The flask app is pretty basic:

from flask import Flask, request
from pathlib import Path

from crypto_common import verify_descriptor
from utils.networking_utils import get_ip_address_2


app = Flask(__name__)

@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

@app.route("/tor", methods=["GET", "POST"])
def accept_descriptor():
    if request.method == "GET":
        return "<p>You're supposed to send descriptors here</p>"
    if request.method == "POST":
        # print(f"we've got this POSTed to us: {request.data}")
        print(f"the files we've got are {request.files}")
        print("Saving our file...")
        file = request.files["file"]
        our_folder = Path("received_descriptors/")
        our_filepath = our_folder / file.filename
        our_folder.mkdir(exist_ok=True, parents=True)
        descriptor_content = file.read().decode("utf-8")
        if verify_descriptor(descriptor_content) == 0:
            print("Saving a valid descriptor to file...")
            with our_filepath.open("w", encoding="utf-8") as f:
                f.write(descriptor_content)
            return "OK"
        else:
            print("Descriptor verification failed;\nnot saving.")
            return "Not OK"    



app.debug = False


def run_flask_app(dir_port=10330, 
                  cert="selfsigned.crt", 
                  key="private.key"):
    #interface = "wlp4s0"
    ip = get_ip_address_2()[1]

    app.run(host=ip, port=dir_port, ssl_context=(cert, key))

And then setup a request:

import requests

files = {"file": b"example"}

requests.post("192.168.1.172", files=file, verify=selfsigned.crt)

And there you go.

The line using openssl is pretty standard:

openssl req -x509 -newkey rsa:4096 -nodes -out cert.pem -keyout key.pem -days 365

A web search for “ python openssl self signed certificate error” leads to this stackoverflow answer How to get Python requests to trust a self signed SSL certificate? - Stack Overflow which suggests this

r = requests.post(url, data=data, verify='/path/to/public_key.pem')

Does that work for you code?

In so many words: no.

Please see above for the part where I do basically the same

This ssl - Can't verify an openssl certificate against a self signed openssl certificate? - Super User suggests that there are important config settings that are needed for the self-signed cert to work.

But I’m not sure this is the problem.

As an aside I side stepped these issues by running my own CA using the easy-rsa software. So each of my systems has a certificate that can be verified against my root CA cert.

1 Like

Well, sure, the problem can be sidestepped, but I’d like to find out why it occurs in the first place, and how it can be fixed

One thing you could try to make sure your Python code is not the culprit is to generate the same certs using the openssl command line tools. If it works using cli generated tools it might help you figure out what’s wrong with the Python generated ones.

True, and I did. My python code recognizes the certs generated with console openssl as self-signed as well.
My current guess is that there’s some check that fails somewhere deep inside the C/Cpython part of requests. But I don’t know enough C/C++ to find it in _ssl.c

1 Like

The rules for valid certificates keep being tighten up.
I suspect your self sign cert is missing one of the newer requirements.

I use curl to test a url that is failing as curl provides lots of information about what it is doing and what the errors are.

Try curl, does it work?

1 Like

Nope. Here’re the logs

curl    --data "stuff" --verbose https://192.168.1.172:10330/tor
*   Trying 192.168.1.172:10330...
* Connected to 192.168.1.172 (192.168.1.172) port 10330 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self-signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Guess I should visit that page.
ETA:

If the remote server uses a self-signed certificate, if you do not install a CA cert store, if the server uses a certificate signed by a CA that is not included in the store you use or if the remote host is an impostor impersonating your favorite site, the certificate check fails and reports an error.

Okay, that makes sense, but that basically dooms self-signed certs forever, and that’s not cool

Okay, I’ve just created a fresh conda environment, and there everything worked like a charm.
This is getting interesting, if not really any clearer