Hi All,
I’m working on an inventory script that generates a dictionary (~ 3000 iterations), json.dumps() it and sends it to a pool of servers /random.shuffle(pool_list)/.
Currently the function creates a connection via socket.create_connection(), then socket.sendall() , next socket.shutdown(SHUT_WR) and then check for any data returned and finally socket.close() .
What worries me is that the code keeps creating and closing sockets. Do you see any problem in that ?
Depending on your platform, and how fast this churns through
sockets, you sometimes find that the TCP/IP stack starts refusing to
open new sockets. When a socket is closed, the system may still want
to listen for any wayward packets associated with that prior
connection (stray FIN/ack and RST for example which can sometimes
arrive after a delay), so you’ll see a sort of cooldown period
imposed before an identical source/destination address+port can be
opened again. The ephemeral ports range is pretty big, but it’s not
infinite, so definitely possible to put all of your source ports
into cooldown if you’re burning through new sockets fast enough.
By Strahil Nikolov via Discussions on Python.org at 06Aug2022 08:05:
I’m working on an inventory script that generates a dictionary (~ 3000
iterations), json.dumps() it and sends it to a pool of servers
/random.shuffle(pool_list)/.
Currently the function creates a connection via socket.create_connection(), then socket.sendall() , next socket.shutdown(SHUT_WR) and then check for any data returned and finally socket.close() .What worries me is that the code keeps creating and closing sockets. Do
you see any problem in that ?
Not inherently. It should be fine, and is what normally happens.
A socket is essentailly a file descriptor on the client (Python) side
and a small data structure on the OS side. There’s not much state when
it isn’t connected.
Cheers,
Cameron Simpson cs@cskk.id.au
@fungi, that’s a good point. The system is quite idle most of the time, but I was wondering if I can set s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) with socket.create_connection() . Any clue on that ?
I though SO_REUSEADDR was only useful for listening sockets.
I doubt you can use it to by pass the timeout period as they would break TCP.
According to socket — Low-level networking interface — Python 3.12.1 documentation :
On POSIX platforms the
SO_REUSEADDRsocket option is set in order to immediately reuse previous sockets which were bound on the same address and remained in TIME_WAIT state.New in version 3.8.
Also, there is a very detailed explanation at
StackOverflow
It seems with the option, the ports that are in TIME_WAIT will be treated as available and this way the actual “consumed” count will be lower.
I would be concerned that overriding the TIME_WAIT may lead to problems with the new connection.
Thanks. I 'm also hesitant to leave it.
I will test the code with SO_REUSEADDR and then I’ll just comment it out (with proper comments why I didn’t remove it ).
If someone needs it, he/she will uncomment and go with it .
I read up onTIME_WAIT and the risk is that your new connection will be closed because of a retransmitted FIN from the old connect.
It’s discussed here What is the purpose of TIME WAIT in TCP connection tear down? - Network Engineering Stack Exchange
I’m more worried about other Apps not being able to connect due to port exhaustion.
My code has a “retry” logic, so it will just establish a new con to one of the remaining hosts in the pool.
If you run out of ports then add more source addresses to your server to send connections from. Each connect is the tuple of srcaddr, srcport, dstaddr, dstport. Having more srcaddr will expand the poll of possible connections.
(I always wondered why IPv6 did not increase the port range).
Without SO_REUSEADDR the sockets with 0.0.0.0 create a conflict with other IPs ( 0.0.0.0:60000 blocks 10.10.10.2:60000).
I’m also considering to create a list of all dictionaries and then push it in a single connection, but I’m not sure how much memory it will use . Today it detects and generates around 3000 dictionaries, but tomorrow they can be 10000.
A quick calculation of
10,000 * AverageDictionarySize
will tell you how much RAM will be needed.
If RAM is short, you could accumulate your dictionaries by appending to a JSON or HTML file (which imports/exports with stdLib) or YAML file (which doesn’t) and then open the file and forward the dictionaries to the TCP port in packets that easily fit into available memory. This will also reduce the frequency of port creation.
It is certainly not a good idea to shorten the TIME_WAIT state of a TCP connection. If you do this you are asking for problems on network devices which work with the TCP state. They will interpret the state not the way you want. For example firewalls will drop your new connections on the too quickly reused 5-tuple (addresses, protocol, ports).
The preferred solution would be to send multiple requests over a single TCP connection. You certainly do not need to generate all the data in memory in advance before sending them! Just generate one JSON dictionary before you need to send it. This approach also avoids the overhead of the repeated TCP handshakes.
If you want the whole TCP data to be a JSON (a list of dictionaries), generate the top-level list yourself and use json.dump() individually for each dictionary inside the list. If you are implementing the receiving side too there is a streaming JSON decoder which is more memory efficient than the standard one:
Another (not optimal) approach is to use multiple source IP addresses as Barry suggested.
The receiving side is out of my reach. I know that it accepts JSON on that TCP port and the rest is black box for me.
Do you think that I can get away with not shuting down and closing the socket until I finish generating and sending the data (for example call the socket shutdown() & close() with an atexit.register() ?)
Do you know the complete specification of the communication protocol? I.e. is it capable of accepting just a single JSON dictionary per connection or can it accept a list of multiple dictionaries?
If it can accept just a single dict then I see these options sorted by speed efficiency of the solution:
If the accepting side is implemented correctly you certainly do not need to prepare the data to be sent in advance. You can keep the connection open and generate the data to be sent as needed. There are protocols which are know for keeping a TCP connection open for hours without sending any data but such long periods are certainly not a good practice, I think a sane protocol should send some communication at least every 10 minutes (otherwise you again have problems with stateful network devices).
Why complicating the program with calling shutdown() & close() using atexit.register()? Use the recommended way with a context manager. The example below is derived from an example in the documentation socket — Low-level networking interface — Python 3.12.1 documentation
import socket
...
HOST = 'daring.cwi.nl'
PORT = 50007
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sending_socket:
sending_socket.connect((HOST, PORT))
for input_data_part in input_data:
data_to_send = process_input_data_part(input_data_part)
sending_socket.sendall(data_to_send)
It seems that the remote system supports newline-delimited JSON.
I think I misunderstood the whole socket logic and thus I used the shutdown() in order to guarantee that the remote system will process the data.
I just tested generating the whole payload (single string value) and then sendall(payload) and the data was received.
I will try to get rid of this payload preparation and give it a try.
I am not sure what all does shutdown() do but I would be afraid it can discard the data which were not sent yet or that it will cancel possible re-submission of the data. Unfortunately i have no experience with that and the documentation does not say much:
By Václav Brožík via Discussions on Python.org at 08Aug2022 10:21:
I am not sure what all does
shutdown()do but I would be afraid it can discard the data which were not sent yet or that it will cancel possible re-submission of the data.
shutdown closes one side of the bidirectional TCP communication. In
the OP’s case, the sending side. All the sent data will arrive, and the
receiver will then see EOF on the dtaa stream.
Unfortunately i have no experience with that and the documentation does
not say much:
socket — Low-level networking interface — Python 3.12.1 documentation
Like os, the socket module is a wrapper for the OS level socket
system. You need to go read man 2 socket on a POSIX system or the
equivalent elsewhere. So the Python docs describe the Python API layer,
its arguments, etc. The detailed semantics are in the OS (and of course
the UDP and TCP etc specs).
IIRC, sockets are unbuffered objects - if you send() or sendall(),
the data are on their way (there will of course be a little buffering in
the OS as the data make their way to the ethernet cable). So there isn’t
really a notion of flush().
When you shutdown the send side you’re closing that side of the TCP
connection and saying “there will be no more data”.
Cheers,
Cameron Simpson cs@cskk.id.au
There is no buffering in python, but there is buffering and a delay to sending in the kernel. The Nagel algorithm delays sending packets I. The hope that shortly the program will send more data.
In the case where this will harm the performance of the app it is usual to turn of Nagel aka TCP_NODELAY.
I found this that seems to explain what is going on: