Secrets Management for Kubernetes: Vault, KMS, and CSI Drivers

TL;DR

Secrets Management for Kubernetes: Vault, KMS, and CSI Drivers without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“If someone can kubectl get pod -o yaml and see your production passwords, you do not have secrets management—you have wishful thinking.”

Core Requirements

• Least‑privilege access; short‑lived tokens
• Rotation without redeploys; minimal blast radius
• Auditability and policy enforcement

Options and Tradeoffs

• Vault: Dynamic secrets, rich policies, audit logs; more ops overhead.
• Cloud KMS: Simpler ops, great for encryption and key management; pair with CSI for delivery.
• CSI Secret Store: Mount secrets from external managers into pods at runtime.

Patterns

• Workload identity (IRSA/Workload Identity) for cloud KMS access
• Dynamic DB creds from Vault for short lifetimes
• Certificate issuance via cert-manager; mesh cert rotation

Example: CSI Secret Store + KMS

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: db-password
        objectType: secretsmanager

Policy Guardrails

• Enforce non‑root, read‑only mounts, and no plaintext secrets in env vars
• Admission policies to block pods without identity or with hardcoded secrets

Operations

• Rotation playbooks; secret versioning and rollback
• Audit trails; alerts on secret access anomalies

Conclusion

Secrets are safer with identity, rotation, and runtime delivery. Pair KMS/Vault with CSI and policy‑as‑code to reduce risk without slowing teams.

Long‑Form Addendum: A Practical Operating Model for Secrets Management for Kubernetes: Vault, KMS, and CSI Drivers

“Resilience is a behavior, not a topology.”

1) Define Your Non‑Negotiables

Start with the constraints you cannot violate: customer impact, regulated data boundaries, and acceptable recovery windows. Write them down as measurable targets:

• RTO and RPO per system of record
• An SLO for the top user journeys (login, checkout, API success)
• A definition of “tier-1”: what pages the on-call, what can wait

2) Runbook (Repeatable)

1. Preflight: verify capacity headroom, DNS and ingress health, controller errors, and that tier‑1 workloads have sane replicas and a PDB.
2. Execute: make one change at a time (control plane, then node pools, then add-ons). Publish timed checkpoints to a shared channel.
3. Validate: run synthetics per region/cluster, confirm burn-rate alerts are stable, and ensure the control plane (API, scheduler) latency has not regressed.
4. Rollback: revert the last change (node pool, add-on, or traffic steering) before you start debugging. Debugging is easier when the blast radius is shrinking.
5. Document: update the runbook with the 2–3 pivots that actually worked.

3) Concrete Guardrails

• No manual drift: changes go through Git; break-glass is time-bound and then codified.
• Standardized components: one ingress pattern, one policy stack, one logging/tracing convention per fleet.
• Controlled disruption: test drain behavior in staging; ensure you can drain a node without violating PDBs or taking SLO hits.
• Ownership clarity: every platform component has an on-call and an escalation path.

4) Metrics That Prove This Works

• Change failure rate during maintenance windows
• Median time to complete a safe node pool rotation
• % tier‑1 services with rehearsed runbooks in the last 90 days
• Alert quality: pages that include dashboard links, owners, and a next action

5) A Small Checklist

• One “go/no‑go” dashboard exists (SLO burn + synthetics + platform signals).
• Every tier‑1 service has a tested rollback and a failover decision tree.
• Policies and configs are versioned and enforced (GitOps + RBAC).
• Quarterly drills produce measurable improvements and updated runbooks.

Appendix 1: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).

Appendix 2: Checklists, Gates, and a 30/60/90 Plan

A Minimal “Go/No‑Go” Gate

Before you execute a risky operation, confirm:

• You have a clear stop signal (SLO burn + a synthetic journey).
• You can roll back within minutes (node pool revert, traffic revert, or Git revert).
• You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).

30/60/90 (Operating Improvements)

• 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
• 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
• 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).

Common Investigations (What On‑Call Actually Does)

• “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
• “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
• “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.

Checklist

• One owner per platform component and an escalation path.
• One canonical runbook per operation (upgrade, failover, restore).
• One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).