Respected Fellow Python team, Need some of your insights please to see if this would be a good solution to be implemented using Python in the networks to avoid outages.
Introduction
With all new technologies in place, we still see latency across the board with medium scale servers and at times large scales too. Competitive business in the IT domain demands continuous server service. The present load balancers do balance the incoming load. But, has a threshold in balancing the load. Identifying a load controller and placing the device in the CLOUD infrastructure/ or for an independent server and configuring a logical way to streamline the ingress load (ensuring the source is alerted) and kept the queue to avoid further damage will ensure latency is avoided.
Problem statement:
With versatile load balancers and elasticized server methodologies for an application, we still witness and discuss about throughput/latency issues caused by an ad hoc load across the physical /cloud servers.
Having critical sources invoking a complex server/application, an unpredictable load from various interfaces, the elasticity of server/application will always be in stake.
Identifying a solution within the cloud of AZ or AWS or IBMs, to keep a low-latency server for any burst of loads thus ensuring a low-latency business is inevitable.
Proposed Solution:
A robust shielding system/Service which ‘gates’ the interfaces and allows the ‘priority’ and stable traffic to the target system thus controlling the load rather than balancing.
Parallelly, sending a planned, periodic alarm for a specific span of time to the overloading interfacing system and pausing temporarily only that interface when unanswered.
By side, storing the requests, processing them asynchronously using surge queues, thus not shedding them right away.
Allocating enough listeners/queues to pick the load for each interface and , maintaining surplus listeners in this system and using them for the over loaded interface will be an added advantage.
Further Details:
At the time of a burst of load:
The controller ensure it gates/filters the load based on the pre-set limit before it reaches the target server, passes only the handleable/threshold limit.
While scanning/preventing, the highest priority tagged system should be considered / prioritized ahead of a low priority invokes.
Parallelly, sending a planned, periodic alarm for a specific span of time to the overloading interfacing system.
By side, storing the requests, processing them asynchronously in surge queues, thus not shedding them right away.
Eventually, pausing only that specific interface, if the alarm is unnoticed (depending on the priority weightage we can configure the alarming counts for that interface)
In case of a high priority system is overloading, the threads used for the low priority systems can be used for the high priority system to quickly process more requests in the surge queue.
The surplus threads will be used to provide additional support the ingress from the overloading interface.
A robust shielding system/Service which ‘gates’ the interfaces and allows the ‘priority’ and stable traffic to the target system thus controlling the load rather than balancing.