Observability with OpenTelemetry: Unifying Metrics, Logs, and Traces

TL;DR

Observability with OpenTelemetry: Unifying Metrics, Logs, and Traces without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“You do not need more dashboards; you need signals that agree on what is happening.”

Why OpenTelemetry?

Observability succeeds when instrumentation is consistent. OpenTelemetry provides vendor‑neutral libraries and agents for capturing metrics, logs, and traces—standardizing how services emit signals so analysis is faster and less brittle. Instead of bespoke middleware per language, you use a shared SDK that speaks W3C Trace Context and semantic conventions.

Core Concepts

• SDKs and exporters: Language‑specific libraries emit telemetry to backends.
• Resource attributes: Identify services, versions, environments consistently (service.name, deployment.environment).
• Context propagation: Carry trace context across HTTP, gRPC, queues, and scheduled jobs.
• Semantic conventions: Standard attribute names for HTTP, DB, messaging; metrics instruments (Counters, Histograms).

Foundations

• Collector: Receives (OTLP, Jaeger, Prometheus), processes (batch, sampling), and exports telemetry. Decouples SDKs from backend vendors.
• Pipelines: Separate trace/metric/log pipelines with tailored processors.
• Deployment: Agent collectors as DaemonSets + gateway collectors for central processing; secure with TLS and auth.

Instrumentation Strategy

• Start with auto‑instrumentation for common frameworks (HTTP, gRPC, DB). Then add manual spans for critical business flows (checkout, search).
• Capture high‑value metrics (request counts, latency, errors), map them to SLOs, and attach exemplars to link metrics → traces.
• Correlate logs with trace IDs (trace_id, span_id) to connect symptoms to causes. Update logging libraries to include context automatically.
• Control cardinality: avoid user IDs, random IDs, or high-card labels; use baggage for passing structured data when necessary.

Runbook: Rolling Out OpenTelemetry

1. Inventory: List services, language stacks, existing telemetry. Identify top incident-prone journeys.
2. Deploy collector: Install OTel Collector (agent + gateway) with OTLP receivers, Prometheus receivers, tail-sampling processor, and exporters (Tempo/Jaeger, Prometheus, Loki).
3. Auto-instrument: Enable language auto-instrumentation (Java agent, Python instrumentation). Start in staging, verify spans/log correlation.
4. Manual spans + metrics: Instrument critical paths with custom spans (tracer.startSpan). Add application metrics via OTel Meter API.
5. Correlate logs: Configure loggers to fetch trace_id from context (otel.getSpan().spanContext), push to Loki/ELK.
6. Dashboards + SLOs: Build golden signal dashboards (latency, error rate) per service; add trace exemplars; define SLOs with burn-rate alerts.
7. Train teams: Document instrumentation patterns, run tracing workshops, and embed runbook links in alerts.

“You don’t need another vendor agent. You need one instrumentation story everyone understands.”

Example: HTTP Server Tracing (Node.js)

const opentelemetry = require('@opentelemetry/api');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(new OTLPTraceExporter({ url: 'http://otel-collector:4318/v1/traces' })));
provider.register();

const tracer = opentelemetry.trace.getTracer('api');

function handler(req, res) {
  const span = tracer.startSpan('handle_request');
  try {
    // work
    res.end('ok');
    span.setAttribute('http.status_code', 200);
  } catch (e) {
    span.recordException(e);
    span.setStatus({ code: 2 });
  } finally {
    span.end();
  }
}

Backends

• Metrics: Prometheus/Mimir via OTel Collector; Grafana dashboards with exemplars linking to traces.
• Logs: Loki/OpenSearch/Elastic; ingest via Fluent Bit → OTel. Include trace_id for correlation.
• Traces: Tempo/Jaeger/Zipkin or vendor APM via OTLP.
• APM integrations: Datadog/New Relic accept OTLP; retain OTel instrumentation for portability.
• Storage planning: Set retention per signal; plan for cost (trace sampling, log retention).

SLOs and Alerting

• Define SLOs on latency/error rates per service; attach error budgets. Use burn-rate alerts (e.g., multi-window, multi-burn) to catch fast regressions.
• Include trace exemplars on latency/error charts so on-call can jump directly to a trace showing the slow path.
• Alerts should embed runbook links, Grafana dashboards, and trace_id/span_id for immediate debugging.

Operational Practices

• Sampling: Use head sampling for high-volume services; tail-based sampling to keep all error/slow traces. Fine-tune rates to balance fidelity and cost.
• Version tagging: Use deployment.environment and service.version resource attributes to correlate rollouts to telemetry changes.
• Runbooks: Document standard queries (top slow spans, error trace search). Keep them in Git; link in alerts.
• Cardinality control: Evaluate metric label cardinality; limit high-card tags (user_id, session_id). Use histograms for latency (OTel histograms with exemplars).
• Security: Redact sensitive data at SDK or collector level; ensure telemetry channels are encrypted.

Pitfalls

• Inconsistent attributes break aggregations; enforce naming via linting and code reviews.
• Too many custom metrics create cardinality explosions; audit metrics monthly.
• No tracing in async flows hides problems in queue/cron workflows; propagate context manually through message attributes.
• Collector bottlenecks: Under-provisioned collectors drop data; monitor CPU/memory, queue length, and dropped spans.
• Ignoring teams: Without developer training, instrumentation stagnates. Embed OTel champions in feature teams.

Adoption Plan

1. Deploy OTel Collector; integrate Prometheus/Loki/Tempo.
2. Auto-instrument services; add manual spans for critical paths.
3. Correlate logs with trace IDs; update dashboards + runbooks.
4. Define SLOs and alerts; rehearse incident response with OTel data.
5. Add tail sampling, exemplars, and standard semantic conventions.
6. Measure outcomes (MTTR, alert quality) and optimize costs.

Tooling Stack

• Collector: OTel Collector via Helm/operator; separate agent and gateway deployments.
• Instrumentation: Official OTel SDKs + auto-instrumentation packages.
• Metrics: Prometheus/Mimir, Grafana Cloud, or vendor; metric exporters configured via collector.
• Traces: Tempo, Jaeger, Zipkin, or SaaS APM.
• Logs: Fluent Bit → OTel → Loki/ELK.
• Alerting: Alertmanager, PagerDuty, Opsgenie with SLO-based policies.
• CI: Tests that ensure instrumentation compiled (use oteltest libs, unit tests for critical spans).

Governance and Enablement

• Define an instrumentation style guide (resource attributes, naming, metrics units). Enforce via code reviews and static analysis (e.g., linters checking service.name).
• Provide starter libraries or shared middlewares (HTTP interceptors, gRPC interceptors) pre-wired with OTel.
• Embed telemetry champions in each product team; hold office hours and instrumentation clinics.
• Track onboarding progress per team; goal: 100% services emitting OTel signals within quarter.
• Tie observability adoption to reliability OKRs (e.g., reduce MTTR 30% by quarter end).

Common Queries and Runbooks

• Create Grafana dashboards with templated queries: “Top N slowest spans,” “Error traces by service,” “Log search by trace ID.”
• Write step-by-step runbooks: “Slow request diagnosis,” “Spike in 5xx errors,” with links to dashboards and traces.
• Store runbooks in Git; update after every incident.

Metrics That Matter

• Instrumentation coverage: % services with OTel SDK; % requests with trace/log correlation.
• MTTR/MTTD: Compare before/after OTel adoption; target improvements.
• Sampling efficiency: % error/slow traces captured vs total. Ensure no critical traces dropped.
• Cardinality incidents: Number of metric label explosions or backend failures due to cardinality.
• Telemetry cost: Track data egress and storage costs; adjust sampling and retention.
• Alert quality: Number of noisy alerts vs actionable ones after OTel adoption.
• Runbook adherence: How often incidents follow documented runbooks; aim to increase.

A Quick Win: Make Trace IDs Ubiquitous

If you do only one thing beyond “install the collector,” make sure every log line for request-handling code includes trace_id (and span_id where practical), and every dashboard panel has a direct link into a filtered trace view. That turns triage into a repeatable click-path: alert → chart → exemplar → trace → code owner. It’s also a forcing function for consistent attribute naming because correlation breaks loudly when teams diverge.

Conclusion

OpenTelemetry unifies telemetry and reduces observability toil. Standardize signals, correlate quickly, and manage SLOs with confidence across your microservices. Build paved roads (collector, SDK configs, dashboards) so teams can focus on insights, not plumbing. When telemetry agrees, incidents become faster to diagnose, SLOs stay healthy, and engineers trust the signals they see.