Best way to handle both server and client parts in one app

Python Help
1

I’m trying to write a simple p2p app; as such, it would need to create nodes that would both listen (run a server socket.accept() loop) and connect to other nodes (with something like socket.connect()).
So far I’ve discovered the use of select for the first part, and I have a script that uses a callback to exchange messages with itself:

from socket import (socket, 
                    AF_INET, 
                    SOCK_STREAM, 
                    create_connection, 
                    SOL_SOCKET, 
                    SO_REUSEADDR)
from ssl import (SSLContext, 
                 PROTOCOL_TLS_SERVER, 
                 PROTOCOL_TLS_CLIENT)


import select

import threading

import time
from tqdm.auto import tqdm


import random


hostname = "example.org"
ip = "127.0.0.1"
port = 8443
client_counter = 0


def server(idle_callback):
    server_socket = socket(AF_INET, SOCK_STREAM)
    server_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    server_socket.bind((ip, port))
    server_socket.listen(5)
    # server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    # server_socket = server_context.wrap_socket(server_socket, server_side=True)
    server_context = SSLContext(PROTOCOL_TLS_SERVER)
    server_context.load_cert_chain('cert_ex1.pem', 'key_ex1.pem')
    server_socket = server_context.wrap_socket(server_socket, server_side=True)




    clients = []

    while True:
        sockets = [server_socket, *clients]
        readable, writable, exceptional = select.select(sockets, [], sockets, 0.1)
        for s in readable:
            if s is server_socket:
                connection, client_address = server_socket.accept()
                connection.setblocking(False)
                clients.append(connection)
                print(f"new connection from {client_address}")
            else:  # must be a client socket
                try:
                    msg = s.recv(1024*8)
                    print(f"{s}: received {msg}")
                    if msg.startswith(b"The time is"):
                        s.sendall(b"The eagle flies at midnight...\n")
                    elif msg == b"":
                        s.sendall(b"cool")
                    else:
                        s.sendall(f"Sorry, I don't understand {msg}\n".encode())
                except ConnectionError as exc:
                    print(f"Exception on {s}: {exc}")
                    s.close()
                    clients.remove(s)
                    continue
        for x in exceptional:
            print(f"exceptional condition on {x}")
            x.close()
            clients.remove(x)
        idle_callback()


def client():
    global client_counter
    client_counter += 1
    client_id = client_counter
    print(f"[{client_id}] Starting client...")

    with create_connection((ip, port)) as client:
        # client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
        client_context = SSLContext(PROTOCOL_TLS_CLIENT)
        client_context.load_verify_locations('cert_ex1.pem')
        client = client_context.wrap_socket(client, server_hostname=hostname)
        # with client_context.wrap_socket(client, server_hostname=hostname) as client:
        print(f'Using {client.version()}\n')

        while True:
            if random.random() < 0.1:
                print(f"[{client_id}] Time to go...")
                break

            if random.random() < 0.5:
                client.sendall(f"The time is {time.asctime()}\n".encode("utf-8"))
            else:
                client.sendall(b"Hello, server\n")

            data = client.recv(1024 * 8)
            if not data:
                print(f"[{client_id}] Client received no data")
                break
            print(f"[{client_id}] Server says: {data}")
            time.sleep(0.5)


def idle_callback():
    if random.random() < 0.1:
        threading.Thread(target=client).start()


def main():
    server(idle_callback)


if __name__ == "__main__":
    main()

A version of this script works equally well when run on two nodes in a LAN (provided proper SSL certs are used).
The problem with this is that I find this use of callbacks rather limiting, but without it only the server loop works and it never comes to the client part.

So, what would be the best way to run such a server listening loop that would not block execution and allow other parts of the script to run as well?

Some ideas I have so far:

  1. in the example above a callback is being called from an endless while True loop, which spams threads that are started, which in some situations is annoying. Perhaps I should learn more about threading and use some sort of lock
  2. the server loop above looks a lot like a simplified version of the serve_forever loop of socketserver.TCPServer. On their own both work great, but both are “blocking” in the sense of waiting forever for an incoming connection
  3. the general solution as I imagine it now should run the server loop and the client side separately. If you have two scripts (server.py and client.py) and run them in two separate terminals, everything works fine, so this is the behaviour we should aim for. Perhaps something like subprocess.run can be used for the server side?
  4. another way to have a non-“blocking” server loop might be using something like asyncio, but I know too little about it to be sure of the details it might be used here.
2

You could use threads.
Have a thread run the listening side of your p2p.
Have other threads do the client side.

3

Sorry, what would that look like?
Let’s say I have a server function which contains the server listening loop, and a client function that uses socket.connect().
Do I then start threads for both, something like

def main():
    server_thread = threading.Thread(target=server)
    client_thread = threading.Thread(target=client)
    server_thread.start()
    client_thread.start()

?

UPD: just tried basically this, and it works, both in the “ping yourself” and the “two nodes talking” versions