TL;DR
Kubernetes will happily run almost anything you give it—and then faithfully surface every design mistake you made. Treat it like a platform, not a pile of YAML: strong defaults, clear ownership, Git‑driven changes, and observability wired to real SLOs.
“If your Kubernetes cluster requires a hero on call, you do not have a platform—you have a very complicated demo.”
Introduction
Kubernetes has become the de facto standard for container orchestration. It is powerful, flexible, and extremely unforgiving of fuzzy thinking. For DevOps teams, the goal is not to memorize every CLI flag; it is to design a platform where:
- Most teams never touch raw cluster internals.
- Defaults encode hard‑won experience.
- Changes are driven by Git and pipelines, not one‑off
kubectlcommands.
This guide distills practices that show up in reliable production clusters—focusing on reliability, security, cost efficiency, and developer velocity—so you can run Kubernetes like a product, not a hobby.
Why Kubernetes?
Kubernetes simplifies the deployment, scaling, and management of containerized applications. At its best, it enables:
- Self-healing: Automatically reschedules and restarts failed workloads.
- Horizontal scalability: Scales applications up or down based on demand.
- Declarative operations: Everything is code—repeatable, reviewable, and auditable.
- Strong ecosystem: Rich integrations across networking, storage, security, and observability.
Best Practices for Kubernetes in DevOps
1) Organize with Namespaces
Use namespaces to isolate environments (dev, staging, prod), ownership boundaries, and multi‑tenancy. Namespaces also enable scoping of policies like RBAC, resource quotas, and network constraints.
Example:
kubectl create namespace dev
kubectl create namespace prod
2) Set Requests and Limits (Right‑Sized)
Right‑size CPU and memory to protect node stability and achieve fair scheduling. Requests drive scheduling; limits cap burst. Get them wrong and you either waste money or starve critical workloads.
Example:
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Add safeguards cluster‑wide:
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-a-quota
namespace: team-a
spec:
hard:
pods: "200"
requests.cpu: "150"
requests.memory: 300Gi
limits.cpu: "200"
limits.memory: 400Gi
3) Standardize Deployments with Helm (or Kustomize)
Use Helm for templating, versioning, and reuse. Favor values files per environment and pin chart versions for reproducibility. Alternatively, Kustomize overlays keep manifest diffs clean without templating.
Example:
helm install my-app ./my-app-chart
4) Observe Everything: Metrics, Logs, Traces
Adopt a three‑pillar strategy:
- Metrics: Prometheus + Grafana for SLOs, saturation, errors, latency.
- Logs: Fluent Bit/Fluentd + OpenSearch/Loki for searchable logs.
- Traces: OpenTelemetry + Tempo/Jaeger for request context across services.
Instrument readiness/liveness probes and alert on golden signals (errors, latency, traffic, saturation).
5) Automate Releases via CI/CD + GitOps
Integrate image builds, security scans, and manifest updates in CI. Use Argo CD or Flux for declarative drift detection and pull‑based delivery. Treat the Git repo as the source of truth and enable progressive delivery (canary/blue‑green) via service mesh or ingress.
Example GitOps bootstrap (Flux):
flux bootstrap github \
--owner=acme \
--repository=platform-config \
--branch=main \
--path=./clusters/prod
6) Lock Down RBAC and Service Accounts
Apply least privilege. Bind roles to namespaces, use dedicated service accounts per workload, and disable default token mounts where not needed.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-only
namespace: prod
rules:
- apiGroups: [""]
resources: ["pods", "services"]
verbs: ["get", "list", "watch"]
7) Enforce Network Policies
Default‑deny east–west traffic, allow only required flows. This reduces lateral movement and blast radius when something goes wrong.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: prod
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
8) Harden Pods: SecurityContext & PodSecurity
Run as non-root, drop capabilities, read-only root filesystem, and enforce via Pod Security Admission (or Kyverno/OPA Gatekeeper).
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
9) Plan for Disruption: PDBs and Topology
Use PodDisruptionBudgets to protect availability during node upgrades and autoscaling. Spread replicas across zones/nodes with topology constraints so a single failure domain cannot take out all replicas.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: api
10) Autoscale Smartly: HPA + CA
Leverage Horizontal Pod Autoscaler (HPA) with meaningful signals (CPU, custom metrics) and Cluster Autoscaler to add capacity. Validate scaling behaviors under load.
11) Optimize Images and Supply Chain Security
Use minimal base images (distroless), pin digests, sign artifacts (Cosign), and scan images in CI/CD. Enforce policies that block vulnerable images before they ever hit the cluster.
12) Cost Controls: Requests, Bin Packing, and Spot
Right-size requests, prefer bin-packing friendly settings, consider spot/preemptible nodes for non-critical workloads, and use Kubecost for visibility.
13) Multi-Cluster Strategy and Disaster Recovery
Operate distinct clusters per environment/region. Backup etcd, store manifests in Git, and rehearse restore procedures. Prefer managed control planes where possible so your team focuses on workloads and guardrails instead of control plane minutiae.
Conclusion
Running Kubernetes well is a platform engineering discipline. By applying strong defaults (security, quotas, policies), automating delivery via GitOps, observing the right signals, and continuously right‑sizing, you will achieve reliability, speed, and cost control—without heroics.
Stay tuned for more DevOps tutorials and best practices.