Lab 38 - Production incident analysis

Lab 38 - Production incident analysis

Infodiscover

Goal: three tickets from users. Find the root cause with metrics, events and logs, mitigate, and write a short postmortem. Chapter: Monitoring with Prometheus and Grafana

Requires the monitoring stack from Lab 37.

Set the scene

cd ~
cat > incidents.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata: { name: prod }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout, namespace: prod }
spec:
  replicas: 2
  selector: { matchLabels: { app: checkout } }
  template:
    metadata: { labels: { app: checkout } }
    spec:
      containers:
        - name: checkout
          image: registry.k8s.io/hpa-example
          resources:
            requests: { cpu: "50m" }
            limits:   { cpu: "50m" }
---
apiVersion: v1
kind: Service
metadata: { name: checkout, namespace: prod }
spec:
  selector: { app: checkout }
  ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: reports, namespace: prod }
spec:
  replicas: 1
  selector: { matchLabels: { app: reports } }
  template:
    metadata: { labels: { app: reports } }
    spec:
      containers:
        - name: reports
          image: busybox:1.36
          command: ["sh", "-c", "x=; while true; do x=\"$x$(head -c 1000000 /dev/zero | tr '\\0' a)\"; sleep 2; done"]
          resources:
            limits: { memory: "64Mi" }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog, namespace: prod }
spec:
  replicas: 2
  selector: { matchLabels: { app: catalog, track: stable } }
  template:
    metadata: { labels: { app: catalog, track: stable } }
    spec:
      containers: [{ name: web, image: "nginx:1.27" }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog-test, namespace: prod }
spec:
  replicas: 1
  selector: { matchLabels: { app: catalog, track: test } }
  template:
    metadata: { labels: { app: catalog, track: test } }
    spec:
      containers:
        - name: web
          image: nginx:1.27
          command: ["sh", "-c", "rm /usr/share/nginx/html/index.html; nginx -g 'daemon off;'"]
---
apiVersion: v1
kind: Service
metadata: { name: catalog, namespace: prod }
spec:
  selector: { app: catalog }
  ports: [{ port: 80 }]
---
apiVersion: v1
kind: Pod
metadata: { name: traffic, namespace: prod }
spec:
  containers:
    - name: traffic
      image: busybox:1.36
      command: ["sh", "-c", "while true; do wget -qO- http://checkout >/dev/null; wget -qO- http://catalog >/dev/null 2>&1; done"]
YAML
kubectl apply -f incidents.yaml

Wait 5 minutes for the metrics to build up. Do not read the manifest - work from the symptoms.

The tickets

INC-101 - “Checkout is very slow since this morning.” - Sales

INC-102 - “The reports service keeps crashing, we lose half our reports.” - Finance

INC-103 - “About every third catalog page shows an error.” - Web team

For each ticket:

  1. Find the cause - Grafana, PromQL, kubectl events, describe, logs.
  2. Mitigate - restore service.
  3. Postmortem - fill in the template below.
Incident:     INC-xxx
Impact:       who noticed what, since when
Detection:    which metric / event shows it (the query!)
Root cause:
Mitigation:   what you changed
Prevention:   alert, policy, test that would have caught it earlier

Hints

Slow but not failing: saturation. Grafana Kubernetes / Compute Resources / Pod has a CPU Throttling panel. PromQL: rate(container_cpu_cfs_throttled_periods_total{namespace="prod"}[5m]) / rate(container_cpu_cfs_periods_total{namespace="prod"}[5m])

Look at the restart count and the last termination reason: kube_pod_container_status_last_terminated_reason{namespace="prod"} and memory over time: container_memory_working_set_bytes{namespace="prod",container="reports"}.

All Pods are Running and Ready. Which Pods are behind the Service? kubectl get endpointslices -n prod -l kubernetes.io/service-name=catalog -o wide, and where do they come from?

Solution

TicketRoot causeMitigationPrevention
INC-101CPU limit 50m on a CPU-heavy app: throttled > 80 % of periodsraise limit / remove CPU limit, add HPAalert on throttling ratio; load test before go-live
INC-102memory grows ~0.5 MB/s until the 64Mi limit: OOMKilled loopshort term raise limit; real fix is the leak in the appalert increase(kube_pod_container_status_restarts_total[1h]) > 3; memory trend panel
INC-103a forgotten catalog-test Deployment carries app: catalog - the Service load-balances to a broken Pod (403/404)delete catalog-test or make the Service selector track: stableselectors specific enough; review labels; error-rate alert per Service
kubectl set resources deploy/checkout -n prod --limits=cpu=500m --requests=cpu=200m
kubectl set resources deploy/reports -n prod --limits=memory=256Mi   # buys time only
kubectl patch svc catalog -n prod -p '{"spec":{"selector":{"app":"catalog","track":"stable"}}}'

Mitigation came before root-cause fixing in all three: restore service first, repair properly afterwards.

Clean up

kubectl delete ns prod