Monitoring Microservices with Prometheus and Grafana

TL;DR

Monitoring microservices with Prometheus and Grafana is about more than pretty graphs. You want a small set of metrics that tell you: Are users getting a good experience? Are we burning our error budget? Where is the bottleneck? Wire those in, build dashboards your teams actually look at, and let the rest go.

“If nobody knows which chart to open during an incident, you don’t have observability—you have wall art.”

Introduction

Monitoring is a critical aspect of managing microservices. Prometheus and Grafana are two popular open-source tools that provide robust monitoring and visualization capabilities. Used well, they let you move from “it feels slow” to “p95 latency for checkout is over 800ms and our cart abandonment rate is climbing.” Used badly, they become a blinking light show nobody trusts.

This guide focuses on the basics done right: the right metrics, simple dashboards, and a feedback loop between incidents and instrumentation.

Why Monitoring Matters

Effective monitoring helps you:

• Detect and resolve issues quickly—especially when MTTR matters more than perfect uptime.
• Understand system performance and behavior across dozens of small services.
• Protect SLOs and error budgets, rather than chasing random alerts.

Without good monitoring, microservices are just a complicated way to be surprised at scale.

Step-by-Step Guide (Lab-Style)

1. Install Prometheus (Lab Setup)

1. Download Prometheus from the official website.
2. Extract the archive and navigate to the Prometheus directory.
3. Start Prometheus:

./prometheus --config.file=prometheus.yml

2. Configure Prometheus (Scraping Services)

Edit the prometheus.yml file to scrape metrics from your microservices. For example:

scrape_configs:
  - job_name: 'microservices'
    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']

3. Install Grafana

1. Download Grafana from the official website.
2. Start Grafana:

./bin/grafana-server

1. Open Grafana in your browser (default: http://localhost:3000).

4. Add Prometheus as a Data Source

1. Log in to Grafana (default credentials: admin/admin).
2. Go to Configuration > Data Sources.
3. Click Add data source and select Prometheus.
4. Enter the Prometheus URL (e.g., http://localhost:9090) and click Save & Test.

5. Create Dashboards

1. Go to Dashboards > New Dashboard.
2. Add a new panel and configure it to display metrics from Prometheus.
3. Save the dashboard.

6. Instrument Your Microservices

Add Prometheus client libraries to your microservices to expose metrics. For example, in a Node.js service:

const client = require('prom-client');
const express = require('express');
const app = express();

const counter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
});

app.get('/', (req, res) => {
  counter.inc();
  res.send('Hello, World!');
});

app.listen(8080, () => {
  console.log('Server is running on port 8080');
});

From here, start exposing the classic “four golden signals”: latency, traffic, errors, and saturation. For HTTP services, that means request counts, durations, and error codes. For queues, it means backlog depth and processing rates. Don’t instrument everything; instrument the paths users actually touch.

“If every service has 200 metrics but nobody can tell you the SLO for checkout, you instrumented sideways.”

Making Dashboards Useful (Not Just Pretty)

Some practical rules:

• Put user-impacting metrics (availability, latency) at the top; technical metrics (CPU, memory) below them.
• Build one high‑level service dashboard per domain with links into “drill‑down” panels.
• Add annotations for deployments and incidents so graphs tell stories, not just numbers.
• Highlight SLO thresholds directly on graphs; don’t make people remember targets.

When someone gets paged, they should know exactly which dashboard to open first and which graph to stare at second.

Alerts and SLOs

Monitoring without alerts is a scrapbook; alerts without SLOs are a firehose.

• Define a small set of SLOs per service (e.g., “99.5% of requests under 300ms over 30 days”).
• Use Prometheus alert rules that consider burn rate over multiple windows, not just “latency > X once.”
• Page on SLO-impacting issues; route lower-severity alerts to tickets.

“Alerts should be expensive. If they’re cheap, your team will treat them that way.”

Conclusion

Prometheus and Grafana provide powerful tools for monitoring microservices. By following this tutorial, you can set up a robust monitoring system and gain valuable insights into your applications. Focus on a handful of meaningful metrics, tie them to user experience and SLOs, and use the data to change behavior—not just to decorate dashboards.

Stay tuned for more DevOps tutorials and best practices.