Hello all !
I’m new to python and working on data streaming for anomaly detection.
As input data i use tshark to capture network paquets, send them to a kafka queue, to train neural networks to detect anomalies. I already have anomaly detection model that works on static data. I’m stuck on the data streaming ingestion stage.
I use tshark to capture paquets in json format. So this code works
json_str = sp.check_output(“tshark -c 5 -T json -e ip.src -e ip.dst -e ip.flags.df -e ip.flags.mf -e ip.flags.rb -e ip.len -e ip.ttl”.split(’ ‘)).decode(‘utf-8’)
tshark_pkts = json.loads(json_str)
data = [pkt[’_source’][‘layers’] for pkt in tshark_pkts]
pprint.pprint(data[0])
while(True):
producer.send(topic, data)
time.sleep(sleep_time)
but i have to wait for tshark process to stop before the data are sent to kafka topic.
I’m wondering how can i send the data in real-time to kafka topic.
thanks for your help