TL;DR
Platform Reliability Scorecards: SLOs for Shared Services and Teams without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.
Why Scorecards?
Because shouting “be reliable” is useless. A scorecard is the scoreboard that forces choices: ship or freeze, refactor or apologize. It makes platform teams own their pipes (build, deploy, auth, mesh, gateways, data planes) and makes tenant teams see their mess in daylight. Less politics, more facts.
Foundations
Anchor SLOs in what users actually touch and tie money and releases to them. Define the SLIs, write the error budget policy, and decide who bleeds when the budget burns. Put the numbers in one brutal dashboard with trends; no spreadsheets, no mystery.
Scope and Rollups
Cover the platform layers and the tenants that ride on them. CI/CD, registry, gateway, mesh, identity, observability, secrets, data plane—each gets a line. Group tenants by product or BU. Weight rollups by traffic or revenue; flat averages are lies that paint everything green while one service is on fire.
SLO Definitions (Examples)
HTTP availability and latency
service: gateway
slos:
- name: availability
target: 99.95
window: 28d
sli:
events:
good: sum(rate(http_requests_total{code=~"2..|3.."}[5m]))
total: sum(rate(http_requests_total[5m]))
- name: p95_latency
target: 99
window: 28d
sli:
latency:
percentile: 0.95
threshold_ms: 300
histogram: http_request_duration_seconds_bucket
Queue processing timeliness
service: async-worker
slos:
- name: time_to_handle
target: 99
window: 28d
sli:
custom:
expr: histogram_quantile(0.99, sum by (le) (rate(job_time_to_handle_seconds_bucket[5m]))) < 5
Sloth/OpenSLO Artifacts
Define SLOs declaratively to generate recording/alerting rules and docs.
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
name: gateway
spec:
service: gateway
slos:
- name: availability
objective: 99.95
sla: 99.9
labels:
tier: platform
alerting:
name: GatewayAvailability
labels: { severity: "page" }
annotations: { summary: "Gateway availability budget burn" }
page_alert:
labels: { severity: "page" }
ticket_alert:
labels: { severity: "ticket" }
slo:
events:
errorQuery: sum(rate(http_requests_total{code=~"5.."}[5m]))
totalQuery: sum(rate(http_requests_total[5m]))
Scorecard Design
Make it a status page with teeth:
- Columns: Service/Team, SLO, Target, Actual, Budget Remaining, Trend, Incidents, Actions
- Status: R/A/G tied to the objective and burn rate—no “yellow-ish” hand waving
- Drill-downs: Links to runbooks and dashboards so on-call can jump straight to fixes
- Cadence: Weekly in platform ops; monthly with execs so budget follows reality
Example (conceptual)
| Entity | SLO | Target | Actual | Budget | Trend | Status |
|------------------|----------------------|--------|--------|--------|-------|--------|
| Gateway | Availability | 99.95 | 99.97 | 84% | ↗ | Green |
| Mesh (control) | API P95 < 200ms | 99% | 98.8% | 65% | ↘ | Amber |
| Build Service | Build success rate | 99% | 98.5% | 40% | ↘ | Amber |
| Payments Team | Checkout availability| 99.9 | 99.6 | 15% | ↘ | Red |
Error Budget Policy (EBP)
Budgets without consequences are wall art. Tie actions to what’s left:
- Green (>70%): Ship normally; experiment if you dare.
- Amber (30–70%): Slow down; add guardrails; harden the obvious weak points.
- Red (<30%): Freeze risky releases; fix the root causes before product managers promise more features.
Example policy gate in CI:
# Pseudo step in pipeline
- name: check-error-budget
run: |
REMAINING=$(curl -s $SLO_API/gateway | jq -r '.budget.remaining')
if (( $(echo "$REMAINING < 0.3" | bc -l) )); then
echo "Budget critical: freezing release"; exit 1
fi
Tenancy and Accountability
- Every shared service has an owning team and a pager; list them on the scorecard.
- Tenants see their own score plus the platform services they ride on.
- Correlate burn with deploys and infra changes so blame sticks to events, not people.
Data Flow and Automation
- OTel/Prometheus generate SLIs → Sloth spits out rules → Alertmanager yells → Scorecard API renders tiles → CI enforces gates.
- Incident timelines link to the SLO lines they broke.
Anti-Patterns to Avoid
- Vanity SLOs that ignore user pain.
- One mega “platform SLO” that hides which layer is failing.
- Pretending planned maintenance doesn’t hurt users.
Adoption Path
- Start with 3–5 platform services and 1–2 app teams.
- Give each one availability + latency SLO; wire Sloth/OpenSLO.
- Stand up the scorecard; review weekly.
- Add budget gates to CI/CD; publish the policy.
- Expand and tune. Ruthlessly.
Conclusion
Scorecards turn SLO talk into forced trade-offs. They end the “it’s fine” meetings and send engineers to the real problem: keeping user experience intact.
“Make Platform Reliability Scorecards: SLOs for Shared Services and Teams boring: repeatable, measurable, and rehearsed.”
Addendum: Turning Platform Reliability Scorecards: SLOs for Shared Services and Teams Into a Repeatable Capability
Runbook (Platform Lens)
- Choose a paved road: define the default template + docs + guardrails.
- Measure outcomes: lead time, ticket volume, and rollback speed.
- Control drift: policies in CI and at deploy time; visible, actionable errors.
- Operate like a product: roadmap, changelog, deprecations, office hours.
Metrics to Track
- Time to first deploy for a new service
- % services on the default path
- Platform support load and median time-to-close
Glossary (Tooltips)
- IDP: Where templates, guardrails, and paved roads live.
- DX: What improves when you remove toil and ambiguity.
- CI/CD: The path from commit to production.
- SLO: How you keep the platform honest.
- OKR: Helps align platform work to outcomes.
Appendix 1: Making the Standard Path the Fast Path
A Practical Product Loop
Platforms win by compounding: every week you make the default path easier and the exception path more expensive.
- Pick 1–2 templates as “blessed defaults.”
- Add guardrails in CI/CD and at deploy time, with actionable errors.
- Publish a changelog and deprecation policy so teams trust upgrades.
- Review adoption and exceptions monthly; prioritize work that reduces repeated tickets.
30/60/90 (Adoption and Reliability)
- 30 days: define platform SLOs; ship one golden path with docs + runbook; measure baseline tickets and lead time.
- 60 days: add self-service for the top ticket categories; introduce an exception registry (owner + expiry).
- 90 days: enforce defaults for new services; migrate the highest-risk legacy patterns; tie outcomes to OKRs.
Checklist
- One obvious default path exists (and it works reliably).
- Exceptions are time-bound and visible.
- Feedback loops exist: office hours, a roadmap, and real metrics.
Appendix 2: Making the Standard Path the Fast Path
A Practical Product Loop
Platforms win by compounding: every week you make the default path easier and the exception path more expensive.
- Pick 1–2 templates as “blessed defaults.”
- Add guardrails in CI/CD and at deploy time, with actionable errors.
- Publish a changelog and deprecation policy so teams trust upgrades.
- Review adoption and exceptions monthly; prioritize work that reduces repeated tickets.
30/60/90 (Adoption and Reliability)
- 30 days: define platform SLOs; ship one golden path with docs + runbook; measure baseline tickets and lead time.
- 60 days: add self-service for the top ticket categories; introduce an exception registry (owner + expiry).
- 90 days: enforce defaults for new services; migrate the highest-risk legacy patterns; tie outcomes to OKRs.
Checklist
- One obvious default path exists (and it works reliably).
- Exceptions are time-bound and visible.
- Feedback loops exist: office hours, a roadmap, and real metrics.