Event‑Driven Autoscaling with KEDA: From Queue Depth to Custom Metrics

TL;DR

Event‑Driven Autoscaling with KEDA: From Queue Depth to Custom Metrics without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“Scaling on CPU for queue workers is like measuring traffic by how loud the engines are revving instead of how many cars are on the road.”

Why KEDA?

The HPA shines for CPU and memory, but real workloads are driven by external signals—queue depth, Kafka lag, HTTP RPS. KEDA bridges that gap with scalable triggers and integrates with HPA behind the scenes.

Core Concepts

• ScaledObject ties a deployment to triggers
• Triggers: SQS, Kafka, RabbitMQ, Prometheus, HTTP, and more
• Scaling: KEDA calculates desired replicas and feeds HPA

Example: Scale on SQS Queue Depth

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
  namespace: billing
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 1
  maxReplicaCount: 30
  cooldownPeriod: 120
  pollingInterval: 15
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.us-east-1.amazonaws.com/1234567890/billing
      queueLength: "100"
      awsRegion: us-east-1

Example: Scale on Kafka Lag

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-worker
  namespace: ingest
spec:
  scaleTargetRef:
    name: kafka-worker
  pollingInterval: 10
  cooldownPeriod: 120
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka:9092
      consumerGroup: ingest
      topic: events
      lagThreshold: "1000"

Safeguards and Tuning

• Use minReplicaCount to keep latency SLOs
• Set reasonable maxReplicaCount and validate capacity
• Tune pollingInterval/cooldownPeriod to avoid thrash
• Add rate limits in app code; protect downstreams

Observability

• Track backlog, processing rate, and saturation
• Alert on sustained backlog growth and scaling failures
• Include KEDA/HPA metrics in dashboards

Pitfalls

• Missing permissions for cloud triggers (IRSA/Workload Identity)
• Underprovisioned nodes; cluster autoscaler must keep up
• Scaling on noisy metrics; prefer stable aggregated signals

Conclusion

KEDA enables scaling on business work, not just CPU. Choose robust triggers, add safeguards, and watch the whole pipeline to keep latency predictable.

Long‑Form Addendum: A Practical Operating Model for Event‑Driven Autoscaling with KEDA: From Queue Depth to Custom Metrics

“Resilience is a behavior, not a topology.”

1) Define Your Non‑Negotiables

Start with the constraints you cannot violate: customer impact, regulated data boundaries, and acceptable recovery windows. Write them down as measurable targets:

• RTO and RPO per system of record
• An SLO for the top user journeys (login, checkout, API success)
• A definition of “tier-1”: what pages the on-call, what can wait

2) Runbook (Repeatable)

1. Preflight: verify capacity headroom, DNS and ingress health, controller errors, and that tier‑1 workloads have sane replicas and a PDB.
2. Execute: make one change at a time (control plane, then node pools, then add-ons). Publish timed checkpoints to a shared channel.
3. Validate: run synthetics per region/cluster, confirm burn-rate alerts are stable, and ensure the control plane (API, scheduler) latency has not regressed.
4. Rollback: revert the last change (node pool, add-on, or traffic steering) before you start debugging. Debugging is easier when the blast radius is shrinking.
5. Document: update the runbook with the 2–3 pivots that actually worked.

3) Concrete Guardrails

• No manual drift: changes go through Git; break-glass is time-bound and then codified.
• Standardized components: one ingress pattern, one policy stack, one logging/tracing convention per fleet.
• Controlled disruption: test drain behavior in staging; ensure you can drain a node without violating PDBs or taking SLO hits.
• Ownership clarity: every platform component has an on-call and an escalation path.

4) Metrics That Prove This Works

• Change failure rate during maintenance windows
• Median time to complete a safe node pool rotation
• % tier‑1 services with rehearsed runbooks in the last 90 days
• Alert quality: pages that include dashboard links, owners, and a next action

5) A Small Checklist

• One “go/no‑go” dashboard exists (SLO burn + synthetics + platform signals).
• Every tier‑1 service has a tested rollback and a failover decision tree.
• Policies and configs are versioned and enforced (GitOps + RBAC).
• Quarterly drills produce measurable improvements and updated runbooks.

Appendix 1: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).

Appendix 2: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).