Policy‑as‑Code for Kubernetes: Kyverno and OPA Guardrails that Scale

TL;DR

Policy‑as‑Code for Kubernetes: Kyverno and OPA Guardrails that Scale without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“If your security model lives in a wiki page, your real policy is whatever someone typed into kubectl at 3 a.m.”

Why Policy‑as‑Code?

Documents don’t stop incidents—guardrails do. Policy‑as‑code brings enforceable rules into pull requests and cluster admission. You get consistent security and reliability without manual reviews.

Core Principles

• Expressive: Policies describe desired state (non-root, signed images) in declarative code.
• Shift-left + guard: Checks run in CI (admission dry-run) and again at admission; the last line of defense enforces decisions already reviewed.
• Versioned and reviewable: Every change is a PR with owners and audit trail.
• Observable: Violations generate metrics/logs; teams see impact and fix fast.
• Exception-aware: Temporary exceptions documented, approved, and expired automatically.

Approaches: Kyverno vs. OPA/Gatekeeper

• Kyverno: Purpose‑built for Kubernetes; policies are YAML; simpler onboarding.
• OPA/Gatekeeper: Rego language; powerful and flexible; steeper learning curve.

Use either—choose based on team skills and complexity of rules.

When to Use Kyverno

• Platform teams unfamiliar with Rego want YAML-based policies.
• Need features like mutate policies, image verification, JSON Patch operations.
• Favor Pod Security Standard-style controls and templated best practices.

When to Use OPA/Gatekeeper

• Require policy reuse across systems (APIs, Terraform, CI).
• Need complex logic or data sources (e.g., check against external allow list).
• Already have Rego expertise or centralized OPA service.

Many platforms run both: Kyverno for Kubernetes-native controls, OPA for cross-cutting policies. Just ensure governance so rules don’t conflict.

Runbook: Rolling Out Guardrails

1. Inventory: Document baseline requirements (Pod Security Standards, image signing, labels). Map to existing incidents (privileged pod, missing labels, etc.).
2. Prototype: Write Kyverno/OPA policies in audit mode for critical rules. Run against live clusters to gauge violations.
3. CI integration: Add kubectl kyverno apply or conftest to pipelines. Fail PRs on violations with actionable errors.
4. Enforce: Flip validationFailureAction: enforce (Kyverno) or enforcementAction: deny (Gatekeeper) for controls vetted in audit mode.
5. Exceptions: Provide CRDs or ConfigMaps for temporary allowlists with expiry metadata. Track in dashboards.
6. Iterate: Add advanced policies (signed images, network policies, resource quotas) once basics are stable. Review metrics monthly.

“Audit mode that never flips to enforce is just passive-aggressive logging.”

Core Guardrails to Enforce

• Non‑root containers; drop capabilities; read‑only rootfs
• Image provenance: signed images and pinned digests
• Resource requests and limits; quotas per namespace
• Network policies: default‑deny + explicit allow lists
• Labels and annotations: ownership, environment, SLO, version

Kyverno Examples

Require non‑root and drop capabilities:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pod-security-basics
spec:
  validationFailureAction: enforce
  rules:
    - name: non-root
      match:
        resources:
          kinds: [Pod, Deployment]
      validate:
        message: "Pods must run as non-root and drop all capabilities"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
            - securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]

Require signed images:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  rules:
    - name: signed-images
      match:
        resources:
          kinds: [Pod, Deployment]
      verifyImages:
        - image: "registry.acme.io/*"
          keyless: true

OPA/Gatekeeper Example

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPNonRoot
metadata:
  name: non-root
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["prod", "stage"]

Operate Policies as Code

• Store policies in Git; review via PRs.
• Run policy checks in CI to catch issues early.
• Use admission controllers to enforce at runtime.
• Maintain exceptions via annotated allow lists with expiry.
• Tag policies with owners, documentation links, and severity (blocker vs warn).
• Version policies; release notes for teams (e.g., “psp-baseline v1.3 now enforces readOnlyRootFilesystem”).

Tooling Stack

• Kyverno (Kubernetes-native YAML policies) or OPA Gatekeeper (Rego language).
• CI integration: kyverno apply, kubectl kyverno test, conftest, or opa eval.
• GitOps: Argo CD/Flux to deploy policies; ensure reconciliation and alerting on drift.
• Exception management: CRDs/ConfigMaps storing justifications, owners, expiry dates; dashboards that show outstanding exceptions.
• Developer tooling: IDE plugins or pre-commit hooks to lint manifests before pushing.

Rollout Strategy

1. Start with audit mode; measure impact and false positives.
2. Switch to enforce for baseline controls.
3. Expand rules gradually; add image signing and network policies.
4. Document remediation steps in runbooks.
   • Pair enforcements with messaging: share sample PRs, remediation guides, and office hours.
   • For high-risk clusters, enforce sooner; for dev sandboxes, maintain warn-only to encourage experimentation.

Example Rollout Timeline

• Week 1: Deploy Kyverno/Gatekeeper; enable non-root and label policies in audit. Report top offenders.
• Week 2-3: Work with app teams to patch manifests; provide scripts and Helm patch examples. Document common fixes.
• Week 4: Flip non-root and label policies to enforce in staging; monitor metrics. Add CI checks so violations fail earlier.
• Week 5: Enforce in prod; add image verification and resource-request policies in audit mode. Repeat cycle.

The key is predictability: teams know when enforcement is coming, what to do, and who approves exceptions.

Observability and Maintenance

• Export policy violation metrics; alert on spikes.
• Track exceptions and expirations; review quarterly.
• Provide developer linting tools to validate manifests locally.
• Log violations with context (namespace, team owner, severity). Feed to SIEM for correlation.
• Track mean time to remediate policy violations; treat prolonged violations as incidents.
• Dashboards should highlight top offending namespaces and policy types.

Metrics That Matter

• Violations per deploy: Trend down as teams adopt paved roads.
• Time to enforce: Days/weeks between audit detection and enforce mode.
• Exception count and age: Keep backlog low; stale exceptions indicate policy fatigue.
• Coverage: % of namespaces/workloads protected by baseline policies (non-root, network policies, resource limits).
• Security incidents prevented: Map policies to incidents they would have prevented to show value.
• CI vs admission ratio: Aim for most violations caught in CI (cheap) versus admission (expensive). Track improvements.
• Policy drift: Alert when controllers or Git repos diverge; treat as SRE issue.

30/60/90 Plan

• 30 days: Inventory requirements; deploy policies in audit mode; set up CI checks; publish docs/runbooks.
• 60 days: Flip key policies (non-root, labels, resource limits) to enforce; integrate image signing checks; add dashboards + alerts.
• 90 days: Introduce advanced policies (network policies, egress restrictions, config validation). Formalize exception workflow and quarterly reviews; tie policy metrics to security OKRs.

Case Study (Hypothetical)

A fintech company suffers repeated incidents from privileged pods and missing labels. They deploy Kyverno in audit mode, revealing 600 violations. Over two sprints, app teams fix configs with help from templated patches. Non-root and label policies move to enforce, reducing violations to <20 per week. Next, they add image-signing verification; two unsigned images are blocked before reaching prod. Compliance audits now point to Git history and metrics, saving weeks of manual evidence gathering.

Developer Enablement Checklist

• Provide Helm/Kustomize snippets that satisfy policies by default (non-root, labels, resource requests).
• Publish “policy packs” per service type (web API, batch job, cron) so teams copy, not guess.
• Offer CLI/IDE plugins (Kyverno CLI, opa eval) so developers can lint manifests locally.
• Document exception request template with owner, justification, expiry, and SLO impact.
• Run office hours/demo sessions when new policies launch; treat feedback seriously.

Guardrails succeed when developers feel supported, not ambushed.

Conclusion

Policy‑as‑code creates paved roads that prevent misconfigurations and security drift. Start with non‑root, resource sizing, and default‑deny networking, then grow into supply chain and provenance. Keep it in Git, enforce in CI and admission, and iterate safely.