TL;DR
Carbon‑ and Cost‑Aware Scheduling for Kubernetes without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.
Inputs for Scheduling
- Regional carbon intensity signals and spot/on‑demand prices
- Workload criticality and SLOs; preemptibility
Patterns
- Shift batch to greener/cheaper regions/time windows
- Use taints/tolerations and priority classes to preempt lower‑priority workloads
- Admission policies to route non‑critical jobs to spot pools
Observability and Guardrails
- Dashboards: cost and carbon per team/service
- Alerts on SLO regressions due to scheduling; error budget gates
Conclusion
Sustainability and cost go together when informed by SLOs and good policies.
“Make Carbon‑ and Cost‑Aware Scheduling for Kubernetes boring: repeatable, measurable, and rehearsed.”
Long‑Form Addendum: A Practical Operating Model for Carbon‑ and Cost‑Aware Scheduling for Kubernetes
“Resilience is a behavior, not a topology.”
1) Define Your Non‑Negotiables
Start with the constraints you cannot violate: customer impact, regulated data boundaries, and acceptable recovery windows. Write them down as measurable targets:
- RTO and RPO per system of record
- An SLO for the top user journeys (login, checkout, API success)
- A definition of “tier-1”: what pages the on-call, what can wait
2) Runbook (Repeatable)
- Preflight: verify capacity headroom, DNS and ingress health, controller errors, and that tier‑1 workloads have sane replicas and a PDB.
- Execute: make one change at a time (control plane, then node pools, then add-ons). Publish timed checkpoints to a shared channel.
- Validate: run synthetics per region/cluster, confirm burn-rate alerts are stable, and ensure the control plane (API, scheduler) latency has not regressed.
- Rollback: revert the last change (node pool, add-on, or traffic steering) before you start debugging. Debugging is easier when the blast radius is shrinking.
- Document: update the runbook with the 2–3 pivots that actually worked.
3) Concrete Guardrails
- No manual drift: changes go through Git; break-glass is time-bound and then codified.
- Standardized components: one ingress pattern, one policy stack, one logging/tracing convention per fleet.
- Controlled disruption: test drain behavior in staging; ensure you can drain a node without violating PDBs or taking SLO hits.
- Ownership clarity: every platform component has an on-call and an escalation path.
4) Metrics That Prove This Works
- Change failure rate during maintenance windows
- Median time to complete a safe node pool rotation
- % tier‑1 services with rehearsed runbooks in the last 90 days
- Alert quality: pages that include dashboard links, owners, and a next action
5) A Small Checklist
- One “go/no‑go” dashboard exists (SLO burn + synthetics + platform signals).
- Every tier‑1 service has a tested rollback and a failover decision tree.
- Policies and configs are versioned and enforced (GitOps + RBAC).
- Quarterly drills produce measurable improvements and updated runbooks.
Glossary (Tooltips)
- PDB: A primary guardrail for “safe” node operations.
- RTO: How fast you must recover.
- RPO: How much data you can lose.
- SLO: Your stop/go signal for risky operations.
- RBAC: Prevents dangerous manual changes and bypasses.
- IaC: The sibling discipline to GitOps.
Appendix 1: Checklists, Gates, and a 30/60/90 Plan
A Minimal “Go/No‑Go” Gate
Before you execute a risky operation, confirm:
- You have a clear stop signal (SLO burn + a synthetic journey).
- You can roll back within minutes (node pool revert, traffic revert, or Git revert).
- You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).
30/60/90 (Operating Improvements)
- 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
- 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
- 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).
Common Investigations (What On‑Call Actually Does)
- “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
- “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
- “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.
Checklist
- One owner per platform component and an escalation path.
- One canonical runbook per operation (upgrade, failover, restore).
- One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).
Appendix 2: Checklists, Gates, and a 30/60/90 Plan
A Minimal “Go/No‑Go” Gate
Before you execute a risky operation, confirm:
- You have a clear stop signal (SLO burn + a synthetic journey).
- You can roll back within minutes (node pool revert, traffic revert, or Git revert).
- You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).
30/60/90 (Operating Improvements)
- 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
- 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
- 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).
Common Investigations (What On‑Call Actually Does)
- “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
- “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
- “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.
Checklist
- One owner per platform component and an escalation path.
- One canonical runbook per operation (upgrade, failover, restore).
- One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).
Appendix 3: Checklists, Gates, and a 30/60/90 Plan
A Minimal “Go/No‑Go” Gate
Before you execute a risky operation, confirm:
- You have a clear stop signal (SLO burn + a synthetic journey).
- You can roll back within minutes (node pool revert, traffic revert, or Git revert).
- You have capacity headroom to absorb churn (surge nodes, autoscaler limits, and realistic disruption budgets).
30/60/90 (Operating Improvements)
- 30 days: standardize dashboards and alerts; prove you can drain a node without violating a PDB; document one runbook with owner + links.
- 60 days: automate preflight checks; rehearse a controlled failure (node pool rotation or traffic failover) and publish a short retro.
- 90 days: make the drill routine; track outcomes (maintenance change failure rate, duration, and customer impact).
Common Investigations (What On‑Call Actually Does)
- “Are we failing because of DNS?” Check CoreDNS latency, NXDOMAIN spikes, and node-local cache health.
- “Are we failing because of scheduling?” Check pending pods, webhook timeouts, and priority class preemption.
- “Are we failing because of data?” Confirm which writes are region/cluster pinned and whether replication lag is within RPO.
Checklist
- One owner per platform component and an escalation path.
- One canonical runbook per operation (upgrade, failover, restore).
- One “break-glass” procedure with time-bound access and codification afterward (RBAC + audit logs).