Lab 22 - Health checks

Lab 22 - Health checks

Goal: see what each probe does when it fails - and how readiness protects a rollout. Chapter: Startup, readiness and liveness probes

Readiness: no traffic, no restart

# 1 - three replicas behind a Service
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 3
kubectl expose deploy web --port 80
kubectl rollout status deploy/web

# 2 - add a readiness probe on a file we control
kubectl patch deploy web -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","readinessProbe":{"httpGet":{"path":"/ready.html","port":80},"periodSeconds":3}}]}}}}'
kubectl get pods -l app=web -w       # Ctrl-C: new Pods Running but 0/1 READY

The rollout is stuck, safely: new Pods never become ready, so the old ones keep serving.

# 3 - make them ready by creating the file
for p in $(kubectl get pods -l app=web -o name); do
  kubectl exec $p -- sh -c 'echo ok > /usr/share/nginx/html/ready.html' 2>/dev/null
done
kubectl rollout status deploy/web
kubectl get endpointslices -l kubernetes.io/service-name=web -o wide

# 4 - take one Pod out of the Service by hand
P=$(kubectl get pods -l app=web -o name | head -1)
kubectl exec $P -- rm /usr/share/nginx/html/ready.html
sleep 10
kubectl get pods -l app=web                  # one 0/1, RESTARTS still 0
kubectl get endpointslices -l kubernetes.io/service-name=web -o yaml | grep -B2 -A2 'ready: false'

Not ready = out of the load balancer. Nothing is restarted.

Liveness: restart

# 5 - a Pod whose health file disappears after 30 s
cat > live.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: live
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "touch /tmp/healthy; sleep 30; rm /tmp/healthy; sleep 600"]
      livenessProbe:
        exec:
          command: ["cat", "/tmp/healthy"]
        periodSeconds: 5
        failureThreshold: 2
YAML
kubectl apply -f live.yaml
kubectl get pod live -w                     # RESTARTS 1 after ~40 s; Ctrl-C
kubectl events --for pod/live | tail -5

Liveness probe failed ... Container app failed liveness probe, will be restarted.

Startup: protect slow starters

# 6 - an app that needs 40 s to start, with a strict liveness probe
cat > slow.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: slow
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 40; touch /tmp/started; sleep 3600"]
      livenessProbe:
        exec: { command: ["cat", "/tmp/started"] }
        periodSeconds: 5
        failureThreshold: 2
YAML
kubectl apply -f slow.yaml
kubectl get pod slow -w                     # restarted every ~10 s - never starts

Liveness kills it before it ever finishes booting - a restart loop.

# 7 - the fix: a startup probe holds liveness back
kubectl delete pod slow
cat >> slow.yaml <<'YAML'
      startupProbe:
        exec: { command: ["cat", "/tmp/started"] }
        periodSeconds: 5
        failureThreshold: 12        # up to 60 s to start
YAML
kubectl apply -f slow.yaml
kubectl get pod slow -w                     # Running, 0 restarts after ~45 s

Discovery

Infodiscover

Task A. Give the web Deployment a liveness probe (tcpSocket port 80) and a readiness probe (httpGet / port 80). Expected: rollout completes, all Pods 1/1.

kubectl patch deploy web -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","readinessProbe":{"httpGet":{"path":"/","port":80}},"livenessProbe":{"tcpSocket":{"port":80},"periodSeconds":10}}]}}}}'
kubectl rollout status deploy/web

Task B. Why is it a bad idea for the liveness probe of web to check that the database is reachable?

If the database goes down, every web Pod fails liveness and is restarted at the same time - the restarts fix nothing and add an outage. Dependency checks belong in readiness (or nowhere).

Clean up

kubectl delete deploy web; kubectl delete svc web
kubectl delete -f live.yaml -f slow.yaml