Cloud Infrastructure Maintenance

99% of teams are overengineering their Kubernetes deployments. They choose the wrong tool and pay for it later lol After managing 100+ Kubernetes clusters and debugging 100s of broken deployments, I’ve seen most teams picking up Helm, Kustomize, or Operators based on popularity, not use case. (1) 𝗜𝗳 𝘆𝗼𝘂’𝗿𝗲 𝗱𝗲𝗽𝗹𝗼𝘆𝗶𝗻𝗴 <10 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 → 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗛𝗲𝗹𝗺 ► Use public charts only for commodities: NGINX, Cert-Manager, Ingress. ► Always fork & freeze charts you rely on. ► Don’t template environment-specific secrets in Helm values. Cost trap: Over-provisioned replicas from Helm defaults = 25–40% hidden spend. Always audit values.yaml. (2) 𝗪𝗵𝗲𝗻 𝘆𝗼𝘂 𝗵𝗶𝘁 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 → 𝗦𝘄𝗶𝘁𝗰𝗵 𝘁𝗼 𝗞𝘂𝘀𝘁𝗼𝗺𝗶𝘇𝗲 ► Helm breaks when you need deep overlays (staging, perf, prod, blue/green.) ► Kustomize is declarative, GitOps-friendly, and patch-first. ► Use base + overlay patterns to avoid value sprawl. ► If you’re not diffing kustomize build outputs in CI before every push, you will ship misconfigs. Pro tip: Pair Kustomize with ArgoCD for instant visual diffs → you’ll catch 80% of config drift before prod sees it. (3) 𝗦𝘁𝗮𝘁𝗲𝗳𝘂𝗹 𝘄𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 & 𝗱𝗼𝗺𝗮𝗶𝗻 𝗹𝗼𝗴𝗶𝗰 → 𝗢𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 𝗼𝗿 𝗯𝘂𝘀𝘁 ► Operators shine when apps manage themselves: DB failovers, cluster autoscaling, sharded messaging queues. ► If your app isn’t managing state reconciliation, an Operator is expensive theatre. But when you need one: Write controllers, don’t hack CRDs. Most “custom” Operators fail because the reconciliation loop isn’t designed for retries at scale. Always isolate Operator RBAC (they’re the #1 privilege escalation vector in clusters.) 𝐌𝐲 𝐇𝐲𝐛𝐫𝐢𝐝 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 At 50+ services across 3 regions, we use: ► Helm → Install “standard” infra packages fast. ► Kustomize → Layer custom patches per env, tracked in GitOps. ► Operators → Manage stateful apps (DBs, queues, AI pipelines) automatically. Which strategy are you using right now? Helm-first, Kustomize-heavy, or Operator-led?

As I grow as a DevOps engineer, here’s a simple way I finally understood Kubernetes… Because let’s be honest: Most people learn Kubernetes like this: Pod today. Service tomorrow. Deployment next week. And at the end? Still confused. Because no one explains how all the pieces connect. Meet Alex again. She already: ✔ Built her app ✔ Dockerized it ✔ Has it ready Now her company says: 👉 “Deploy this on Kubernetes.” And that’s where confusion usually starts. Kubernetes: Not just one thing… but a system Think of Kubernetes like a city. Each file you write is like a set of instructions telling the city what to do. 1. Deployment “Run my app” Alex starts here. She writes a Deployment file. This tells Kubernetes: • What container to run (Docker image) • How many copies (replicas) • How to update the app safely 👉 Example: “I want 3 copies of my app always running.” If one crashes? Kubernetes replaces it automatically. 2. Pod “Where the app lives” A Pod is the smallest unit in Kubernetes. It’s where your container actually runs. But here’s the catch: 👉 You don’t usually create Pods directly. Deployment manages Pods for you. 3. Service “Make it reachable” Now Alex has her app running… But no one can access it. That’s where a Service comes in. It: • Gives the app a stable IP • Allows communication inside the cluster • Can expose the app to users Types: • ClusterIP (internal) • NodePort (external via node) • LoadBalancer (public access) 4. Ingress “Control traffic like a pro” Instead of exposing many services… Alex uses an Ingress. It acts like a smart gate: 👉 “If user goes to /login → send to this service” 👉 “If user goes to /api → send somewhere else” Clean URLs. Better control. 5. ConfigMap “Non-secret settings” Her app needs configs: • Environment = production • API URLs Instead of hardcoding… She uses a ConfigMap. 👉 Keeps config separate from code. 6. Secret “Sensitive data” Passwords. Tokens. Keys. These go into Secrets. 👉 Not exposed like normal configs. 7. Persistent Volume “Keep data safe” Containers are temporary. If they restart… data disappears. So Alex uses: • Persistent Volume (PV) • Persistent Volume Claim (PVC) This keeps data safe even if containers die. 8. ReplicaSet “Keep the right number running” Behind every Deployment… There’s a ReplicaSet. Its job: “Make sure exactly X pods are running.” So how everything connects: 1️⃣ Deployment creates Pods 2️⃣ ReplicaSet ensures the right number stays running 3️⃣ Pods run your containers 4️⃣ Service exposes Pods 5️⃣ Ingress manages external access 6️⃣ ConfigMap + Secret provide configuration 7️⃣ PV/PVC stores persistent data The truth most people miss: Kubernetes is not about memorizing files. It’s about understanding how they work together. Real takeaway: When you understand this flow… You stop being confused by YAML files. And start thinking like: “How do I want my system to behave?” #Kubernetes #MTN

I’ve spent 7 years obsessing over the perfect Kubernetes Stack. These are the best-practices I would recommend as a basis for every Kubernetes cluster. 1. Implement an Observability stack A monitoring stack prevents downtime and helps with troubleshooting. Best-practices: - Implement a Centralised logging solution like Loki. Logs will otherwise disappear, and it makes it easier to troubleshoot. - Use a central monitoring stack with pre-built dashboards, metrics and alerts. - For microservices architectures, implement tracing (e.g. Grafana Tempo). This gives better visibility in your traffic flows. 2. Setup a good Network foundation Networking in Kubernetes is abstracted away, so developers don't need to worry about it. Best practices: - Implement Cilium + Hubble for increased security, performance and observability - Setup a centralised Ingress Controller (like Nginx Ingress). This takes care of all incoming HTTP traffic in the cluster. - Auto-encrypt all traffic on the network-layer using cert-manager. 3. Secure your clusters Kubernetes is not secure by default. Securing your production cluster is one of the most important things for production. Best practices: - Regularly patch your Nodes, but also your containers. This mitigates most vulnerabilities - Scan for vulnerabilities in your cluster. Send alerts when critical vulnerabilities are introduced. - Implement a good secret management solution in your cluster like External Secrets. 4. Use a GitOps Deployment Strategy All Desired State should be in Git. This is the best way to deploy to Kubernetes. ArgoCD is truly open-source and has a fantastic UI. Best practices: - Implement the app-of-apps pattern. This simplifies the creation of new apps in ArgoCD. - Use ArgoCD Autosync. Don’t rely on sync buttons. This makes GIT your single-source-of-truth. 5. Data Try to use managed (cloud) databases if possible. This makes data management a lot easier. If you want to run databases on Kubernetes, make sure you know what you are doing! Best practices - Use databases that are scalable and can handle sudden redeployments - Setup a backup, restore and disaster-recovery strategy. And regularly test it! - Actively monitor your databases and persistent volumes - Use Kubernetes Operators as much as possible for management of these databases Are you implementing Kubernetes, or do you think your architecture needs improvement? Send me a message, I'd love to help you out! #kubernetes #devops #cloud

AWS EKS Traffic Flow in Real Production: From Route 53 to Pods and RDS Explained In real-world AWS production environments, Kubernetes is not just about running containers. It is about designing a secure, scalable, and predictable traffic flow that can survive failures, scale under pressure, and remain observable at every layer. The architecture shown here represents a proven AWS EKS production pattern used across SaaS platforms, fintech systems, and enterprise applications. The request lifecycle starts at the user browser and reaches Amazon Route 53, where DNS resolution maps the application domain to an Application Load Balancer. Route 53 enables health-based routing, failover strategies, and multi-region expansion without changing application logic. The Application Load Balancer is deployed in a public subnet and acts as the only internet-facing entry point. It terminates TLS, applies security group rules, and evaluates listener conditions. Host-based and path-based routing allow multiple applications or microservices to share the same load balancer while remaining logically isolated. Traffic is forwarded to ALB target groups associated with EKS worker nodes running inside private subnets. These nodes have no direct internet exposure. Outbound connectivity, such as container image pulls or third-party API access, is handled through a NAT Gateway, preserving isolation while maintaining operational flexibility. Within the cluster, the NGINX Ingress Controller translates external traffic into Kubernetes-native routing rules. It forwards requests to ClusterIP services, which load balance traffic across healthy pods. This layer enables blue-green deployments, canary releases, and zero-downtime rollouts commonly required in production environments. Application pods communicate with Amazon RDS hosted in a dedicated database subnet with no internet gateway access. Security groups strictly allow traffic only from application nodes, protecting the database from unintended exposure. Security is enforced through multiple layers. Network ACLs protect subnets, security groups control traffic between components, and Kubernetes policies govern pod behavior. This defense-in-depth approach ensures resilience, compliance, and operational confidence. Final Note: In mature production environments, this architecture works because every layer has a clearly defined responsibility and failure boundary. Internet traffic is terminated at the ALB, compute remains isolated in private subnets, and data persistence is strictly confined to database networks. NAT, security groups, and NACLs collectively reduce blast radius during incidents. When combined with proper observability, incident response, and controlled release strategies, this model enables teams to operate Kubernetes on AWS with predictability, resilience, and long-term operational confidence.This design reflects how real production EKS platforms balance scalability, security, and operational control.

Routing traffic into Kubernetes? You’re not just choosing a tool, you’re choosing a paradigm. 𝐈𝐧𝐠𝐫𝐞𝐬𝐬 got us started. 𝐆𝐚𝐭𝐞𝐰𝐚𝐲 API is where we’re headed. Let’s talk about why this shift isn’t just about new YAML, it’s a mindset change. 𝐈𝐧𝐠𝐫𝐞𝐬𝐬 𝐰𝐚𝐬 𝐠𝐫𝐞𝐚𝐭… 𝐮𝐧𝐭𝐢𝐥 𝐢𝐭 𝐰𝐚𝐬𝐧’𝐭. It gave us a simple way to handle HTTP routing through a controller. But once clusters scaled and teams grew, it began to show its age: 🔸 Hard-to-manage configs 🔸 No native multi-tenancy 🔸 Limited protocol support 🔸 Inconsistent behavior across vendors It did the job, until the job got too complex. 𝐆𝐚𝐭𝐞𝐰𝐚𝐲 𝐀𝐏𝐈 𝐢𝐬 𝐛𝐮𝐢𝐥𝐭 𝐟𝐨𝐫 𝐰𝐡𝐚𝐭 𝐜𝐨𝐦𝐞𝐬 𝐧𝐞𝐱𝐭. Designed by Kubernetes SIG-Network, it brings: 🔸 Native support for HTTP, TCP, and UDP 🔸 True multi-tenant gateway deployments 🔸 Clean separation of infrastructure (Gateways) from routing logic (Routes) 🔸 Extensibility and cloud-provider awareness baked in 𝐇𝐞𝐫𝐞’𝐬 𝐭𝐡𝐞 𝐛𝐨𝐭𝐭𝐨𝐦 𝐥𝐢𝐧𝐞: 𝐈𝐧𝐠𝐫𝐞𝐬𝐬 is a resource. 𝐆𝐚𝐭𝐞𝐰𝐚𝐲 is a framework. And in modern Kubernetes environments, that difference matters. As apps go multi-protocol, clusters scale out, and teams demand better boundaries. Gateway API isn’t a nice-to-have. It’s the standard Kubernetes has been waiting for. So… Still patching Ingress? Or already designing with Gateways? #Kubernetes #GatewayAPI #CloudNative #DevOps #Ingress #PlatformEngineering #SystemDesign

This is one of the most frustrating Kubernetes moments. You check everything: Pods are running. Deployments are successful. Logs look clean. Still… users cannot access the application. After spending hours debugging, you realize: •The problem was never the app. •It was how you exposed it. Here is the clarity most people wish they had earlier: 📦 ClusterIP: - Your app works perfectly inside the cluster. - But from outside, it simply does not exist. 🌐 NodePort: - You can access it using node IP and port. - Works for testing, but feels messy and limited. ⚖️ LoadBalancer: - Now your app is reachable with a public IP. - This is what most production setups rely on. 🚪 Ingress: - Not just exposure, but control. - Routing with domains, paths, and HTTPS, all in one place. The real problem: Most developers focus on making the app run. But Kubernetes requires you to also design how it is accessed. Running ≠ Reachable A simple way to think about it: Internal traffic → ClusterIP Quick access → NodePort Production access → LoadBalancer Controlled routing → Ingress Once you understand this, you stop guessing and start solving. #Kubernetes #DevOps #CloudComputing #CloudNative #Networking #SRE #PlatformEngineering

Azure Private AKS with External Access: A reference architecture implemented in Terraform. One of the trickiest and hardest topics in Kubernetes on Azure: you want your cluster locked down, but you still need the outside world to reach your apps. ✅ Here's an architecture pattern that solves this elegantly, built with Azure best practices and battle tested for production. Private AKS clusters are great for security, no public API server exposure. But "private" can also mean "isolated" if you're not careful about how external traffic gets in. 📌 The Solution: Hub & Spoke with strategic public touch points. This architecture uses a hub-spoke network model where: • The hub VNet centralizes your security controls (Azure Firewall, Bastion, jumpbox). • The spoke VNet hosts your AKS workloads in isolation. VNet peering connects them privately. • External access comes through an Application Gateway with WAF. This is your single, controlled entry point. Everything else stays internal. 🚀 What makes it production-ready 1/ Security layers that actually work together: • Private endpoints for ACR, Key Vault, and Storage (no public blob URLs floating around) • Azure Firewall controlling egress (your nodes can't phone home to unexpected places) • Bastion + jumpbox for management access (no SSH exposed, ever) Managed identities throughout (no secrets to rotate) 2/ Operational foundations: • Log Analytics integration from day one • Proper RBAC with least-privilege role assignments • Separate node pools for workload isolation 3/ IaC: The entire architecture is implemented in Terraform (automatically generated and tested for policies, naming conventions, and costs) and can easily be deployed in Brainboard.co or in your own CI/CD solution. ⚠️ Most teams skip the private DNS zones, because they're usually not easy to set up, but they're what makes private endpoints actually work → This architecture includes them for AKS, ACR, Key Vault, and Storage, because partial private networking is often worse than none at all. This reference architecture is ideal for: • Regulated industries requiring network isolation • Multi-tenant platforms where blast radius matters • Any production workload where "secure by default" isn't optional ❤️ Besides that, the architecture is modular enough to strip out what you don't need. Not everyone needs Traffic Manager across regions or the full firewall setup for dev environments. That's why it is highly flexible. Get it here for free: https://lnkd.in/eZYJKgJx What's your experience been with private AKS? #Azure #Kubernetes #AKS #Terraform #CloudArchitecture #DevOps #InfrastructureAsCode

Many API gateways were built before cloud native and GitOps became the standard. Here’s what they tend to get wrong. They were created in a pre–cloud native, pre–GitOps world. As infrastructure evolved, 𝘁𝗲𝗮𝗺𝘀 𝗼𝗳𝘁𝗲𝗻 𝗵𝗮𝗱 𝘁𝗼 𝗹𝗮𝘆𝗲𝗿 𝗼𝗻 𝘀𝗰𝗿𝗶𝗽𝘁𝘀, 𝗲𝘅𝘁𝗿𝗮 𝘁𝗼𝗼𝗹𝘀, 𝗮𝗻𝗱 𝘄𝗼𝗿𝗸𝗮𝗿𝗼𝘂𝗻𝗱𝘀 instead of having a gateway designed for declarative, automated workflows from the start. I went through how legacy gateways operate vs a modern, cloud-native API gateway to see the pattern. Let me show you what I mean (with two small examples): 𝗔𝗣𝗣𝗥𝗢𝗔𝗖𝗛 🅰️: The Legacy, UI-Driven Gateway This is the classic gateway that predates cloud native and GitOps, where everything is configured through UIs, proprietary workflows, and manual updates. 𝗦𝘁𝗮𝗴𝗲 𝟭 - Operations 🅰️ UI-centric operations that don’t align with declarative configs, so teams glue together extra tools and scripts just to get basic GitOps flows working. 🅰️ Limited support for Kubernetes, multi-cloud, and hybrid setups, often tied to specific environments or stacks. 🅰️ Service discovery and config changes require full reloads, dropping open connections and risking SLAs every time you ship. 𝗥𝗲𝘀𝘂𝗹𝘁: ➡️ Slow, brittle changes. ➡️ Fragmented tooling around the gateway. ➡️ Every deployment feels risky. 𝗔𝗣𝗣𝗥𝗢𝗔𝗖𝗛 🅱️: The Modern, GitOps-Driven API Gateway This is a cloud native gateway that is fully declarative, GitOps‑first, and designed to span Kubernetes, multi-cloud, and hybrid environments. 𝗦𝘁𝗮𝗴𝗲 𝟭 – Configuration: 🅱️ Routing, load balancing, security policies, and even custom objects are all managed declaratively, so they plug directly into CI/CD and GitOps workflows. 𝗦𝘁𝗮𝗴𝗲 𝟮 – Environments and scale: 🅱️ The same gateway model works across hybrid cloud, multi-cloud, multi-orchestrator, and on‑prem, covering any Kubernetes distribution and even non‑Kubernetes orchestrators. 𝗦𝘁𝗮𝗴𝗲 𝟯 – Change management: 🅱️ New services, routing rules, certificates, and security hardening apply dynamically without full reloads, keeping HA and SLAs intact. 🅱️ Intelligent routing can use host, path, HTTP method, headers, JWT claims, and more, instead of just a couple of static fields. 𝗥𝗲𝘀𝘂𝗹𝘁: ➡️ Safe, continuous changes instead of disruptive releases. ➡️ One gateway layer that actually keeps up with your platform. ➡️ Operational overhead drops while control and flexibility go up. 𝗙𝗶𝗻𝗮𝗹 𝘀𝗰𝗼𝗿𝗲: Approach 🅰️: Change debt, manual ops, and reload‑driven outages. Approach 🅱️: Cloud native, GitOps‑aligned, and built for dynamic, multi‑cloud platforms. If your API gateway still behaves like a monolith front-end, it will eventually slow down everything behind it. — If you’re rethinking your API gateway strategy, let me know of any questions you have, and I’ll be happy to help.

𝗚𝗮𝘁𝗲𝘄𝗮𝘆 𝗔𝗣𝗜 𝗶𝗻 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 : In this setup, both teams are using the same domain name. Everything comes through google com. But control is not shared. Each team owns its 𝗼𝘄𝗻 𝗻𝗮𝗺𝗲𝘀𝗽𝗮𝗰𝗲. The dev team manages routes like /iphone and /payment. The devops team manages /android and /order. Every route forwards traffic only to pods inside its own namespace. The important part is separation of control. Application teams define only their routing rules. They do not manage the load balancer. They do not change 𝗧𝗟𝗦 𝗰𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗮𝘁𝗶𝗼𝗻. They do not interfere with other teams’ paths. The Gateway is defined once. It acts as the single entry point. Routes from 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗻𝗮𝗺𝗲𝘀𝗽𝗮𝗰𝗲𝘀 attach to it only if allowed. This keeps things clean and secure. Now compare this with Ingress. Ingress is 𝗻𝗮𝗺𝗲𝘀𝗽𝗮𝗰𝗲𝗱 𝘀𝗰𝗼𝗽𝗲. If we have /payment in app1-ns and /order in app2-ns, we usually depend on controller specific features to manage advanced routing. Kubernetes Ingress only supports basic host and path routing out of the box. Features like traffic splitting, rewrites, redirects or fine grained control come from annotations and depend on the controller implementation. They are not part of the core Ingress API. With Gateway API, 𝗶𝗻𝗳𝗿𝗮 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝗮𝗻𝗱 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗿𝗼𝘂𝘁𝗶𝗻𝗴 are clearly separated. So we get one public domain, one gateway, but multiple isolated teams working safely inside the same cluster. 𝗦𝗵𝗮𝗿𝗲𝗱 𝗱𝗼𝗺𝗮𝗶𝗻 does not mean shared ownership. That is the real power of Gateway API in 𝗺𝘂𝗹𝘁𝗶 𝘁𝗲𝗮𝗺 Kubernetes environments. ➕ Follow Sai P. for more insights ♻ Repost to help others 📩 Save for later #Kubernetes #GatewayAPI #DevOps #docker #K8s #TLS #multitenancy #isolation #routes #gatewayclass #applicationroutes

Step-by-Step Workflow: Step 1: DNS Configuration DNS provider maps *.devopscube.com domain to a Load Balancer IP address Users access applications via subdomains like app1.devopscube.com, app2.devopscube.com Step 2: Load Balancer Creation Load Balancer is automatically created by the Gateway Service The LB routes incoming traffic to the Kubernetes cluster Step 3: Gateway API Configuration Gateway Resource is created in the K8s cluster HTTPRoute resources define routing rules for different services This uses the newer Gateway API (not the older Ingress API) Step 4: Istio Gateway Controller The istio-gateway-controller watches for Gateway and HTTPRoute resources It configures the Istio Gateway Proxy based on these resources This follows the Kubernetes Gateway API standard Step 5: Traffic Routing Istio Gateway Proxy receives traffic from the Load Balancer Based on HTTPRoute configurations, it routes traffic to appropriate backend services Example shows traffic split 50/50 between two backend services Step 6: Backend Services backend-v1-svc and backend-v2-svc receive traffic from the gateway Destination Rules (Istio CRD) are applied to these services for advanced features: Circuit breaking (prevent cascading failures) mTLS (mutual TLS for service-to-service encryption) Rate limiting (control request flow) Step 7: Application Deployment backend v1 deployment and backend v2 deployment run the actual application containers These are standard Kubernetes deployments with Istio sidecar proxies injected Key Components Summary: Gateway API - Modern Kubernetes API for north-south traffic Istio Gateway Controller - Bridges Gateway API with Istio data plane Istio Gateway Proxy - Actual proxy handling inbound traffic (Envoy-based) Istio Destination Rules - Advanced traffic management policies Service Mesh Features - mTLS, circuit breaking, observability Traffic Flow: External User → DNS → Load Balancer → Istio Gateway Proxy → HTTPRoute Rules → Backend Services (with Istio features) → Application Pods This architecture provides a standardized, portable ingress configuration using Gateway API while leveraging Istio's advanced service mesh capabilities for security and traffic management. Kubernetes Kubernetes Certification Hub Docker and Kubernetes Security