Zero‑Trust Networking for Kubernetes: A Practical Guide

TL;DR

Zero‑Trust Networking for Kubernetes: A Practical Guide without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

Why Zero‑Trust for Kubernetes?

Kubernetes makes east–west traffic cheap, which is great for microservices—and bad for attackers. A zero‑trust model assumes the network is hostile, identities are verified continuously, and access is least‑privilege. This guide provides a practical blueprint: network policies, service identity, mTLS, service mesh, and guardrails that teams can live with.

“If any pod can talk to any other pod just because they share a cluster, you have a flat network with extra steps—not zero‑trust.”

Foundations

• Identity: Workload identity via service accounts and SPIFFE/SPIRE or mesh certificates.
• Encryption: mTLS for service‑to‑service; TLS for ingress/egress.
• Segmentation: Namespaces and network policies enforce boundaries.
• Policy: Declarative controls (OPA/Kyverno) + admission to prevent risky configs.

Runbook: Implement Zero-Trust in Phases

1. Inventory + labels: Tag workloads by tier, sensitivity, owner. Document required flows (app → DB, app → third-party).
2. Default deny: Apply namespace-wide default-deny network policies; add allow rules iteratively.
3. Identity + mTLS: Deploy service mesh or SPIRE; enforce STRICT mTLS; reject unknown certs.
4. Egress control: Route outbound traffic via egress gateways; approve destinations; log DNS/HTTP.
5. Layer 7 + ingress: AuthorizationPolicies, rate limits, WAF, and DDoS protections.

“Zero trust isn’t a flag you flip; it’s the absence of implicit trust at every layer, enforced one control at a time.”

Step 1: Default‑Deny Everywhere

Start by denying all ingress/egress inside namespaces, then allow explicit flows.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Create allow‑lists for required communications (e.g., app → database, app → cache). Keep rules as tight as possible using labels.

Step 2: Strong Service Identity + mTLS

Use a mesh (Istio/Linkerd) or SPIRE to issue workload identities and enforce mTLS for all east–west traffic. Reject plaintext and unknown identities.

• Istio: PeerAuthentication and DestinationRule ensure STRICT mTLS.
• Linkerd: Automatic mTLS with service accounts.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT

Step 3: Layer 7 Policy (Optional but Powerful)

Service mesh lets you define intents beyond IP/port—methods, paths, and auth claims.

• Block dangerous verbs, rate‑limit, and require JWT claims for sensitive APIs.
• Use AuthorizationPolicy with custom conditions tied to identity.

Step 4: Egress Control and DNS

• Restrict egress to known endpoints; audit unexpected external calls.
• Prefer egress gateways; block direct internet traffic from pods.
• Validate DNS policies to prevent data exfil via DNS.

Step 5: Protect Ingress

• WAF for public endpoints; validate content types and request sizes.
• Terminate TLS at ingress with strong ciphers; forward mTLS inside the mesh.
• Rate‑limit and enable DDoS protections at the edge.

Operational Practices

• Labels: Consistent labels drive policies; enforce via admission controls.
• Testing: Chaos testing for policies—what breaks when rules tighten?
• Observability: Mesh telemetry, flow logs, and alerts on denied connections.
• Runbooks: Fast rollback of policy changes; staged rollout with canaries.

Observability and Metrics

• Mesh/iptables telemetry to show allow/deny counts, TLS versions, and identities.
• Alert on policy denials, failed TLS handshakes, and egress anomalies.
• Track “time to codify” manual firewall changes; goal is hours, not days.
• KPIs: % workloads under default-deny, % east-west traffic with mTLS, number of egress destinations per namespace.
• Feed these metrics to SLOs: e.g., “99.9% of East-West traffic must be mTLS” or “Denied connections resolved within 15 minutes.” When SLOs burn, treat them like any reliability issue—slow change velocity and fix controls.

30/60/90 Adoption Plan

• 30 days: Label workloads; apply default-deny in dev/stage; enable mesh mTLS in non-prod; add drift alerts for policies.
• 60 days: Roll out mTLS + network policies in prod; add egress gateways; integrate admission checks for labels and sidecars; run chaos tests.
• 90 days: Layer-7 policies for sensitive APIs; WAF + DDoS protections at ingress; monthly policy drills; dashboard for coverage/denied flows.

Case Study (Hypothetical)

A fintech platform labels workloads by PCI scope, applies namespace default-deny, and uses Istio STRICT mTLS. An egress gateway allows only payment processors and telemetry endpoints. During rollout, a legacy job tries to reach an unapproved endpoint; the deny alert fires, policy is updated after review, and the job gets a temporary allowlist. Result: lateral movement drastically reduced, auditors see code-reviewed policies, and teams have a runbook for future changes.

Outcome: Zero trust became a habit—policy changes ship via PR, denials surface quickly, and on-call engineers revert by commit instead of hunting iptables rules at 2 a.m.

Example: Minimal Policy Suite

# Default deny in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: billing
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# Allow only app -> db
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-db
  namespace: billing
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: ["Egress"]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: billing
      podSelector:
        matchLabels:
          app: db
    ports:
    - protocol: TCP
      port: 5432

Pitfalls and How to Avoid Them

• Overly permissive policies: Start restrictive; open only what’s needed.
• Certificate sprawl: Centralize issuance and rotation (mesh CA/SPIRE).
• No staging: Validate policies in non‑prod environments first.
• Hidden sidecars: Mesh sidecars can create surprising flows; document and observe.
• Alert fatigue: Deny alerts everywhere with no context. Aggregate by namespace/app and attach trace/log links.
• Ingress drift: “Temporary” ingress changes that stay forever. Use admission checks and automated scans to catch public exposure.
• Break-glass chaos: Emergency manual fixes never codified. Require retroactive PRs within hours and track exceptions.

Tooling Stack (Keep It Consistent)

• Mesh/identity: Istio/Linkerd + SPIFFE/SPIRE for cert issuance and rotation.
• Policy/admission: Kyverno/Gatekeeper, OPA, or mesh authorization policies.
• Secrets: External managers (Vault/KMS) with CSI drivers; sealed secrets for manifests.
• Observability: Prom/Grafana for policy metrics, Loki/ELK for denied flows, trace IDs from mesh to logs.
• CI/CD: Policy-as-code checks, config linting, and dry-run tests before rollout.

Pick a minimal stack and document paved roads; zero trust fails when every team invents its own tooling.

Governance and Access Control

• RBAC: Restrict who can edit NetworkPolicies, mesh configs, ingress controllers, and egress gateways. Use CODEOWNERS and approvals.
• Change management: Stage policy changes through dev/stage/prod; pin versions; freeze automation during incidents.
• Auditing: Track policy/app changes via Git, admission logs, and mesh telemetry; store them with retention for compliance.
• Break-glass: Provide short-lived emergency roles with auto-expiry; require retroactive PRs and postmortems for every use.

Zero trust dies when implicit trust sneaks back in via permissions; governance keeps the culture honest.

Pre-Deployment Checklist

• Namespace has default-deny ingress/egress.
• Workloads labeled for policies; admission validates labels.
• STRICT mTLS enforced (mesh or SPIRE) with cert rotation.
• Egress rules documented and implemented via gateway.
• Policy changes tested in dev/stage; rollback plan rehearsed.
• Alerts wired to SLOs; dashboards show allow/deny and last policy commit.

Conclusion

Zero‑trust on Kubernetes is achievable without heroic effort. Establish identity, encrypt traffic, default‑deny your network, and adopt guardrails that developers can follow. Over time, tighten policies with strong observability, so security improves continuously with minimal toil.