Data Plane SRE: Operating Kafka, Redis, and Postgres On—and Off—Kubernetes

TL;DR

Data Plane SRE: Operating Kafka, Redis, and Postgres On—and Off—Kubernetes without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

Build vs. Buy: Decision Factors

• Latency & locality: In‑cluster reduces hops; managed adds cross‑VPC hops
• Operational maturity: Backups, upgrades, failover, security posture
• Costs: Infra + people; managed premiums vs toil
• Blast radius: Shared cluster failures vs dedicated managed service SLOs

Kafka

• On K8s: Use operators, dedicated node pools, fast disks; plan ZooKeeper/KRaft migrations
• Managed: Leverage provider SLOs; align partitions/throughput to pricing
• SRE: Lag SLOs, ISR counts, controller health; runbooks for rebalancing and broker loss

Redis

• On K8s: StatefulSets with AOF, anti‑affinity, and eviction control; sentinel/cluster mode
• Managed: Quick HA with automated failover and TLS
• SRE: Evictions, latency, hit rate; runbooks for failover and memory pressure

Postgres

• On K8s: Patroni or operators; WAL archiving; storage classes with IOPS guarantees
• Managed: Backups, replicas, point‑in‑time restore built‑in
• SRE: Replication lag, deadlocks, bloat; runbooks for failover and vacuuming

Common Runbook Elements

• Backups and restores (tested); DR drills and RTO/RPO
• Capacity: IOPS, throughput, memory; alerts on saturation
• Change management: version upgrades, schema changes, partitioning

Security and Compliance

• TLS/mTLS; encryption at rest; secrets via KMS/Vault
• Access: least privilege; audit logs; data masking where needed

Conclusion

Pick the right tool for the job: managed where it saves toil and risk; in‑cluster when latency or control wins. Either way, SRE metrics and runbooks make it reliable.

“Make Data Plane SRE: Operating Kafka, Redis, and Postgres On—and Off—Kubernetes boring: repeatable, measurable, and rehearsed.”

Long‑Form Addendum: A Practical Operating Model for Data Plane SRE: Operating Kafka, Redis, and Postgres On—and Off—Kubernetes

“Resilience is a behavior, not a topology.”

1) Define Your Non‑Negotiables

Start with the constraints you cannot violate: customer impact, regulated data boundaries, and acceptable recovery windows. Write them down as measurable targets:

• RTO and RPO per system of record
• An SLO for the top user journeys (login, checkout, API success)
• A definition of “tier-1”: what pages the on-call, what can wait

2) Runbook (Repeatable)

1. Preflight: verify capacity headroom, DNS and ingress health, controller errors, and that tier‑1 workloads have sane replicas and a PDB.
2. Execute: make one change at a time (control plane, then node pools, then add-ons). Publish timed checkpoints to a shared channel.
3. Validate: run synthetics per region/cluster, confirm burn-rate alerts are stable, and ensure the control plane (API, scheduler) latency has not regressed.
4. Rollback: revert the last change (node pool, add-on, or traffic steering) before you start debugging. Debugging is easier when the blast radius is shrinking.
5. Document: update the runbook with the 2–3 pivots that actually worked.

3) Concrete Guardrails

• No manual drift: changes go through Git; break-glass is time-bound and then codified.
• Standardized components: one ingress pattern, one policy stack, one logging/tracing convention per fleet.
• Controlled disruption: test drain behavior in staging; ensure you can drain a node without violating PDBs or taking SLO hits.
• Ownership clarity: every platform component has an on-call and an escalation path.

4) Metrics That Prove This Works

• Change failure rate during maintenance windows
• Median time to complete a safe node pool rotation
• % tier‑1 services with rehearsed runbooks in the last 90 days
• Alert quality: pages that include dashboard links, owners, and a next action

5) A Small Checklist

• One “go/no‑go” dashboard exists (SLO burn + synthetics + platform signals).
• Every tier‑1 service has a tested rollback and a failover decision tree.
• Policies and configs are versioned and enforced (GitOps + RBAC).
• Quarterly drills produce measurable improvements and updated runbooks.

Glossary (Tooltips)

• PDB: A primary guardrail for “safe” node operations.
• RTO: How fast you must recover.
• RPO: How much data you can lose.
• SLO: Your stop/go signal for risky operations.
• RBAC: Prevents dangerous manual changes and bypasses.
• IaC: The sibling discipline to GitOps.

Appendix 1: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).

Appendix 2: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).