Monitoring with Prometheus and Grafana
Monitoring with Prometheus and Grafana
Three signals
| Signal | Answers | Typical tools |
|---|---|---|
| Metrics | how much, how fast, how often - over time | Prometheus, Grafana |
| Logs | what exactly happened | Loki, Elasticsearch |
| Traces | where did the time go across services | OpenTelemetry, Tempo, Jaeger |
Prometheus in Kubernetes
graph LR
subgraph targets
KS[kube-state-metrics<br/>object state]
NE[node-exporter<br/>DaemonSet: CPU, disk, net]
KL[kubelet / cAdvisor<br/>container usage]
APP[your app /metrics]
end
P[Prometheus] -->|scrape every 30 s| KS & NE & KL & APP
P -->|rules fire| AM[Alertmanager] --> N[mail / Teams / PagerDuty]
G[Grafana] -->|PromQL| P
Prometheus pulls metrics over HTTP and stores them as time series:
metric_name{label="value"} number.
kube-prometheus-stack (Helm) installs all of it plus the Prometheus Operator and its CRDs:
| CRD | Purpose |
|---|---|
ServiceMonitor / PodMonitor | “scrape the Pods behind this Service” - owned by the app team |
PrometheusRule | recording and alerting rules |
Prometheus, Alertmanager | the servers themselves |
PromQL in five lines
kube_pod_container_status_restarts_total # a counter
rate(container_cpu_usage_seconds_total{namespace="shop"}[5m]) # per-second rate
sum by (pod) (container_memory_working_set_bytes{namespace="shop"})
increase(kube_pod_container_status_restarts_total[1h]) > 3
kube_deployment_status_replicas_available / kube_deployment_spec_replicas < 1
What to watch
- RED for services: Rate, Errors, Duration.
- USE for resources: Utilization, Saturation (e.g. CPU throttling), Errors.
- Kubernetes specifics: restarts, OOM kills, Pending Pods, replicas available < desired, PVC usage, certificate expiry.
Alert on symptoms users feel, not on every cause - an alert nobody acts on trains people to ignore alerts.
Note
On kubeadm, etcd, scheduler and controller-manager listen on localhost only. The stack then reports them “down”. The lab disables those scrapes; in production expose their metrics endpoints instead.