I have a rather large multi-threaded (using PyQt5’s threading provisions; Qrunnable, QThreadPool, etc.) Python script which is performing at least 10 repetitive/concurrent functions, running in perpetuity. For each of these threaded functions, I pass class variables from the PyQt GUI parent thread as arguments into the function, put the function into the queue/QThreadpool, and when the function runs it emits signals which I copy back into class variables. When the function ends, repeat.
The latest function that I added to the script is one where I can send an email to request the status of various things (the values of various class variables) and it will reply with whatever I asked for. There are hundreds of class variables, probably >1,000. Formatting this data for email is proving to be quite a chore. I have to deliberately add every single variable that I want to send. For example, let’s say I request the state of variables related to the “override” function by sending an email containing “OVERRIDE INFO”:
if self.emailReceived:
emailStatusData = {}
if self.dataRequested == "OVERRIDE INFO":
emailStatusData["systemsEnabled"] = self.systemsEnabled
emailStatusData["waitingOnApproval"] = self.waitingOnApproval
emailStatusData["overrideMode"] = self.overrideMode
emailStatusData["overrideState"] = self.overrideState
(etc. many more variables)
elif self.dataRequested == "CLEANOUT INFO":
emailStatusData["hopperEmpty"] = self.hopperEmpty
emailStatusData["cleanoutRequired"] = self.cleanoutRequired
emailStatusData["cleanoutPerformed"] = self.cleanoutPerformed
emailStatusData["cleanoutRunning"] = self.cleanoutRunning
(etc. many more variables)
elif....
(etc., many more options)
self.sendReply(emailStatusData)
I would get the values of [systemsEnabled, waitingOnApproval, overrideMode, overrideState, et. al.].
As I expand the options for these email requests, I have to go scour through my code, collect all the variables I want to be part of the groups, and add them to the appropriate cluster of “emailStatusData[“thing”] = self.thing”. It’s a pain. And I’m adding new class variables all the time (this script is under perpetual development), and I have to remember every time a create I new one, to go add it to one of these clusters.
So to make things easier, what I’m thinking of doing, is replacing all of my class variables with a single (nested) dictionary, so:
self.systemsEnabled becomes self.systemVariables["overrideData"]["systemsEnabled"]
self.waitingOnApproval becomes self.systemVariables["overrideData"]["waitingOnApproval"]
self.overrideMode becomes self.systemVariables["overrideData"]["overrideMode"]
self.overrideState becomes self.systemVariables["overrideData"]["overrideState"]
(etc. many more variables)
self.hopperEmpty becomes self.systemVariables["cleanoutData"]["hopperEmpty"]
self.cleanoutRequired becomes self.systemVariables["cleanoutData"]["cleanoutRequired"]
self.cleanoutPerformed becomes self.systemVariables["cleanoutData"]["cleanoutPerformed"]
self.cleanoutRunning becomes self.systemVariables["cleanoutData"]["cleanoutRunning"]
(etc. many more variables)
I could write a quick script to automate the changeover, and then from there, my variables are all grouped logically, and sending the right data in the email reply becomes as easy as:
if self.emailReceived:
if self.dataRequested == "OVERRIDE INFO":
self.sendReply(self.systemVariables["overrideData"])
elif self.dataRequested == "CLEANOUT INFO":
self.sendReply(self.systemVariables["cleanoutData"])
elif....
(etc., many more options, INCLUDING...)
elif self.dataRequested == "ALL SYSTEM INFO":
self.sendReply(self.systemVariables)
It seems like a fine idea to me, but what I don’t know (among most other things) is the performance characteristics of Python’s dictionary objects vs class variables. These class variables are being read by and written by multiple threads at once, some of them change once per day, some of them change thousands of times per second, etc.
By putting them all into a single dictionary would I be creating a bottleneck where only one variable can be written at a time?
(or is that how it is already?)
By putting them all into a single dictionary would I be creating a bottleneck where only one variable can read at a time?
(or is that how it is already?)
Would I be adding some delay each time a variable is accessed, as now it has to be looked up in the dictionary?
(or is that how it is already?)
I don’t really know what’s going on beneath all the layers of the Python onion, so I don’t know what to expect.
I don’t even know if these are the right questions to be asking.
What else is there to consider?