Managed vs DIY Observability: Cost, Retention, and Signal Quality

TL;DR

Pick based on pain, not vendor slides. Managed buys you time and fewer 3 a.m. pages. DIY (Prometheus/Mimir, Loki, Tempo, Grafana) buys you control, data residency, and lower bills if you know what you’re doing. If you can’t say how you’ll cap cardinality and tier data on day one, you’re not ready for DIY.

Evaluation Criteria

Judge by outcomes: ingest/storage/query cost predictability, signal quality (real histograms, exemplars, span richness), retention design (hot vs warm + downsampling), tenancy/governance (budgets, SLO boards, isolation), and operational burden (upgrades, migrations, multi-region). If your answer to “how do we stop label explosions?” is “we’ll be careful,” choose managed.

Reference Options

Managed: Datadog, New Relic, Grafana Cloud, AWS AMP/AMG + OTEL, Google Cloud Ops—fast start, sane UX, someone else runs storage. DIY: Prometheus + Mimir for metrics, Loki for logs, Tempo for traces, Grafana for eyes; add OpenSearch/ClickHouse when logs get spicy. More knobs, more pager duty.

Architecture Sketches

Managed (push): Apps → OTel SDK/Collector → SaaS ingest → UI & Alerts
DIY (pull+push): Apps → OTel → Edge Prom/Loki/Tempo → Remote write/object storage → Grafana UI
Hybrid: Critical SLO metrics in DIY (low cost); richer APM traces/logs in managed

Retention and Tiering

Metrics (Prometheus/Mimir)

Keep a short hot window (7–14 days) for speed, remote-write to Mimir/Thanos for months+ in object storage, and downsample (5m → 1h) for long-range SLI charts so queries stay cheap.

# Prometheus TSDB retention (local hot)
--storage.tsdb.retention.time=14d
--storage.tsdb.wal-compression

# Recording rules to pre-aggregate
- record: http:latency_ms:p95:rate5m
  expr: histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

Logs (Loki)

Set retention per tenant, ship chunks to S3/GCS via boltdb-shipper, and keep labels lean—high-cardinality fields belong in the log body, not as labels.

limits_config:
  retention_period: 14d
  per_stream_rate_limit: 4MB
  ingestion_rate_mb: 20
  max_global_streams_per_user: 10000
schema_config:
  configs:
    - from: 2025-01-01
      store: boltdb-shipper
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Traces (Tempo)

• Trace retention short (2–7 days) + span metrics long
• Use exemplars to bridge metrics and traces

metrics_generator:
  registry:
    external_labels: { cluster: prod }
  storage:
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true

Sampling and Cardinality Controls

Tail-based sampling in OTel Collector

processors:
  tail_sampling:
    decision_wait: 2s
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: high_latency
        type: latency
        latency: { threshold_ms: 600 }
      - name: service_critical
        type: probabilistic
        probabilistic: { sampling_percentage: 25 }
      - name: default
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Metrics cardinality

• Avoid unbounded user_id, request_id as labels
• Cap resource and span attributes; pre-aggregate via recording rules
• Use native histograms where supported to reduce bucket explosion

Cost Management Tactics

Observability isn’t a feelings budget. Drop noisy logs; park long-tail logs in cheap object storage and hydrate only when needed. Right-size scrape intervals. Use exemplars sparingly. Put per-team ingest/cardinality budgets in writing and page when they blow past limits—before finance pages you.

Governance and Tenancy

Give teams their own tenant and a hard budget cap. Standardize SLO dashboards with burn alerts. Enforce residency in-region and scrub PII at ingest—compliance shouldn’t be a side quest.

Decision Matrix (Guidance)

• Small teams, high velocity: Start managed. Reconsider when the bill shocks you.
• Regulated or heavy data gravity: DIY/hybrid with strict residency and cost caps.
• Platform teams with storage chops: DIY metrics, maybe managed tracing; hybrid usually wins.

Example: Hybrid Implementation Steps

1. Ship OTel from apps to local collectors
2. Metrics: Prometheus → remote write to Mimir; 14d hot + 12m cold with downsampling
3. Logs: Loki with 7d hot, 30d warm; drop debug/trace logs at edge
4. Traces: Tail-sample; 2d retention in Tempo; exemplar links to metrics
5. Grafana: SLO dashboards; per-team folders; budget panels

Conclusion

Pick observability like you pick databases: match the workload and your tolerance for pain. Managed buys speed; DIY buys control and cheaper scale. Hybrid often wins: precision where it matters, thrift where it doesn’t. Whatever you choose, cap cardinality, sample smart, tier retention, and tie it all to SLOs and budgets—or enjoy surprise outages and surprise bills.

“Make Managed vs DIY Observability: Cost, Retention, and Signal Quality boring: repeatable, measurable, and rehearsed.”

Addendum: Turning Managed vs DIY Observability: Cost, Retention, and Signal Quality Into a Repeatable Capability

Runbook (Platform Lens)

1. Choose a paved road: define the default template + docs + guardrails.
2. Measure outcomes: lead time, ticket volume, and rollback speed.
3. Control drift: policies in CI and at deploy time; visible, actionable errors.
4. Operate like a product: roadmap, changelog, deprecations, office hours.

Metrics to Track

• Time to first deploy for a new service
• % services on the default path
• Platform support load and median time-to-close

Glossary (Tooltips)

• IDP: Where templates, guardrails, and paved roads live.
• DX: What improves when you remove toil and ambiguity.
• CI/CD: The path from commit to production.
• SLO: How you keep the platform honest.
• OKR: Helps align platform work to outcomes.

Appendix 1: Making the Standard Path the Fast Path

A Practical Product Loop

Platforms win by compounding: every week you make the default path easier and the exception path more expensive.

1. Pick 1–2 templates as “blessed defaults.”
2. Add guardrails in CI/CD and at deploy time, with actionable errors.
3. Publish a changelog and deprecation policy so teams trust upgrades.
4. Review adoption and exceptions monthly; prioritize work that reduces repeated tickets.

30/60/90 (Adoption and Reliability)

• 30 days: define platform SLOs; ship one golden path with docs + runbook; measure baseline tickets and lead time.
• 60 days: add self-service for the top ticket categories; introduce an exception registry (owner + expiry).
• 90 days: enforce defaults for new services; migrate the highest-risk legacy patterns; tie outcomes to OKRs.

Checklist

• One obvious default path exists (and it works reliably).
• Exceptions are time-bound and visible.
• Feedback loops exist: office hours, a roadmap, and real metrics.

Appendix 2: Making the Standard Path the Fast Path

A Practical Product Loop

Platforms win by compounding: every week you make the default path easier and the exception path more expensive.

1. Pick 1–2 templates as “blessed defaults.”
2. Add guardrails in CI/CD and at deploy time, with actionable errors.
3. Publish a changelog and deprecation policy so teams trust upgrades.
4. Review adoption and exceptions monthly; prioritize work that reduces repeated tickets.

30/60/90 (Adoption and Reliability)

• 30 days: define platform SLOs; ship one golden path with docs + runbook; measure baseline tickets and lead time.
• 60 days: add self-service for the top ticket categories; introduce an exception registry (owner + expiry).
• 90 days: enforce defaults for new services; migrate the highest-risk legacy patterns; tie outcomes to OKRs.

Checklist

• One obvious default path exists (and it works reliably).
• Exceptions are time-bound and visible.
• Feedback loops exist: office hours, a roadmap, and real metrics.