Kubernetes Upgrades: Zero‑Downtime Strategies and Runbooks

TL;DR

Kubernetes Upgrades: Zero‑Downtime Strategies and Runbooks without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“If every cluster upgrade still feels like a one-off stunt, you do not have a process—you have recurring bravery.”

Upgrade Goals

• No customer impact; maintain SLOs
• Repeatable process; minimize manual steps
• Clear rollback levers and checkpoints

What “Zero Downtime” Actually Means

“Zero downtime” does not mean “nothing changes.” It means:

• Requests continue to succeed within your SLO boundaries.
• Deployments and platform operations may be slower, but user journeys remain healthy.
• Any disruption is limited to maintenance windows with clear comms and rollback.

If you can’t measure user impact during upgrades, you’re flying blind.

Preflight Checklist

• Cluster health: API latency, etcd health, controller errors
• Workload readiness: PDBs, readiness probes, multiple replicas
• Capacity: extra headroom for rescheduling during drain
• Backups: etcd snapshot and manifest backups
• Add-on compatibility: CNI, CSI, ingress controllers, service mesh versions
• Policy checks: admission policies and PodSecurity compatibility for new version
• Critical dependencies: DNS, registry, secrets manager, identity provider health

“Most ‘upgrade outages’ are not Kubernetes bugs. They’re your hidden assumptions about capacity, readiness, and add-ons.”

Strategies

• Blue‑green clusters: Stand up a new cluster; migrate via GitOps and DNS
• In‑place upgrades: Control plane then nodes; cordon/drain with PDB protection
• Rolling node pools: Replace nodes gradually; validate workload SLOs

Runbook: Safe Upgrade in Practice

1. Plan: Pick target version; read release notes; check add-on compatibility matrix (CNI/CSI/ingress/mesh).
2. Freeze: Announce window; pause risky deploys; pin image tags/digests; stop noisy experiments.
3. Preflight: Verify control plane health, etcd latency, and node headroom. Validate that critical workloads have PDBs and multiple replicas.
4. Control plane first: Upgrade control plane (managed service or kubeadm) and watch API latency, controller errors, and etcd health.
5. Nodes next: Roll node pools (surge nodes), cordon/drain with respect to PDBs. Watch workload error rate and latency.
6. Validate: Run synthetic checks (login/checkout) and platform checks (image pulls, DNS queries, GitOps sync).
7. Unfreeze: Resume deploys; publish release notes; capture metrics (time, errors, rollback events).

“The upgrade is not complete when nodes report Ready. It’s complete when your synthetic journeys pass and your error budgets stop burning.”

Disruption Controls

• PDBs protecting availability
• Topology spread constraints for zone/node diversity
• MaxUnavailable/Surge in deployments for headroom
• PriorityClasses to ensure critical services schedule first during churn
• NodeLocal DNSCache to reduce DNS flakiness during node drains
• Surge node pools for temporary headroom during rolling upgrades

Upgrade Readiness Scorecard (Before You Touch the Version)

Treat upgrades like a product launch: you can’t “hope” your way to safety. A simple readiness scorecard keeps you honest:

• Workload redundancy: Critical services run ≥2 replicas across zones; no single-node or single-zone hot spots.
• Disruption tolerance: Every tier-1 workload has a sane PDB and passes a real drain rehearsal in staging.
• Dependency health: DNS, ingress, registry pulls, and identity are healthy and observable; you have dashboards ready before the window.
• Admission survivability: Webhooks are horizontally scalable; timeouts and failure policies are explicit (no accidental cluster-wide “deny”).
• Rollback rehearsal: You have practiced the rollback path in non-prod within the last quarter and captured the steps in a runbook.

If you cannot score yourself quickly, the upgrade is already telling you it will be risky.

Sample Config Snippets (Make Safety Concrete)

A minimal PDB that protects availability while still allowing node drains:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

And a topology spread constraint that reduces “all replicas on one node/zone” surprises during churn:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

These aren’t magic. They’re just the minimum set of intent you want Kubernetes to respect when the platform is changing underneath you.

Communications (Don’t Surprise Anyone)

Even with perfect engineering, upgrades fail socially if stakeholders learn about them after the fact. A lightweight comms plan reduces risk:

• Publish the window, expected “blast radius,” and rollback trigger (usually SLO burn) to a shared channel.
• Call out which deploys are frozen, which are allowed, and who approves exceptions.
• During execution, post timed checkpoints: “control plane complete,” “node pool 1/3,” “synthetics green,” “unfreeze.”
• Afterward, write a short upgrade note: duration, regressions found, and one improvement for the next run.

That last step matters: you’re building trust that upgrades are routine and that surprises are rare.

GitOps Runbook (Outline)

1. Freeze window and change ticket
2. Validate preflight checks; snapshot etcd
3. Upgrade control plane (managed service or kubeadm)
4. Rotate node pools; monitor golden signals
5. Validate workloads; synthetic checks pass
6. Unfreeze; document and publish release notes

Observability During Upgrades

• Control plane: API server latency, etcd fsync, controller-manager queue depth, scheduler latency.
• Workloads: Error rate, p95/p99 latency, saturation (CPU/memory), restart counts.
• Platform components: Ingress health, DNS latency, registry pull success, GitOps sync lag.
• Alerts: Temporary thresholds tuned for upgrades, but never silenced blindly. Use SLO burn alerts as the “stop” signal.

Common Failure Modes (and Fixes)

• PDB deadlocks: Drains stuck because PDBs are too strict or replicas too low. Fix by increasing replicas, adjusting PDBs, or adding surge nodes.
• Add-on mismatch: CNI/CSI/ingress not compatible with new version. Fix by upgrading add-ons first in staging.
• Capacity crunch: Insufficient headroom causes evictions and latency spikes. Fix by adding temporary nodes and tuning autoscalers.
• DNS flaps: CoreDNS overloaded during churn. Fix with NodeLocal DNSCache, tune CoreDNS resources, and add more replicas.
• Webhook timeouts: Admission webhooks become slow and block scheduling. Fix by scaling webhooks and setting timeouts/failurePolicy intentionally.

Tooling Stack

• Cluster lifecycle: Managed services (EKS/GKE/AKS), Cluster API, or Terraform modules.
• Node rolling: Node groups with surge; Karpenter for right-sized replacement nodes.
• Validation: Synthetic checks (k6, Playwright), kubectl conformance checks, and canary workload tests.
• GitOps: Argo CD/Flux to apply desired state and rollback by commit.
• Observability: Prometheus/Grafana + OpenTelemetry for traces; alerts wired to SLO burn.

30/60/90 Plan

• 30 days: Document upgrade runbook; add preflight checks; ensure all critical workloads have PDBs and readiness probes; create staging upgrade cadence.
• 60 days: Automate upgrades for non-prod; add synthetic checks; implement surge capacity approach and CoreDNS hardening.
• 90 days: Make upgrades routine with quarterly cadence; measure MTTR/incident rates during upgrades; publish platform release notes and post-upgrade retros.

Observability

• Dashboards for API server, etcd, controller, scheduler
• Error budget burn alerts during upgrade windows
• Traces for critical paths to detect latency regressions

Rollback Plan

• Revert cluster version (managed) or restore etcd snapshot (self‑managed)
• Roll back node pools; revert Git changes; re‑point DNS

Conclusion

Upgrades become routine with the right guardrails. Bake the plan into GitOps, protect workloads with PDBs, and validate health with SLOs and synthetic checks.