Secure Scalable Cloud Architectures with Anomaly Detection Using Kubernetes and AKS

Designing data architectures for the cloud that are scalable requires elasticity, stable performance, fault tolerance, and cost efficiency. These measurements must be taken in both a dynamic and randomized way. This paper presents a summation-centric, cloud-native framework for scalable data pipelines that have been deployed on Kubernetes and Azure Kubernetes Service (AKS). This strategy attempts to address the issue of global workload aggregation for data ingestion and processing. This enables estimations of arrival rate, service rate, queue length, and resource consumption that are all mathematically estimable. A control system that operates in a closed loop integrates summation workload aggregation, queue length stabilization, predictive control of the system, inverse load distribution, and resiliency with checkpoints in order to maintain the equilibrium of the system and equilibrium of the system in a proactive way. Also, the framework presents a more-efficient approach to addressing challenges in the system. It does so by avoiding the use of reactive autoscaling by using stability, cost, and reliability considerations directly in the control scheme and operational decision-making to eliminate oscillation and overprovisioning of system resources.