Lab 23 - Logs, events and troubleshooting

Lab 23 - Logs, events and troubleshooting

Goal: read logs and events fluently, then diagnose five broken Pods from cluster output alone. Chapters: Logs and events · Troubleshooting

Warm-up: logs and events

# setup - a chatty Deployment with two containers per Pod
cd ~
cat > chatty.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatty
spec:
  replicas: 2
  selector:
    matchLabels: { app: chatty }
  template:
    metadata:
      labels: { app: chatty }
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "i=0; while true; do i=$((i+1)); echo \"app line $i\"; sleep 2; done"]
        - name: sidecar
          image: busybox:1.36
          command: ["sh", "-c", "while true; do echo sidecar alive; sleep 5; done"]
YAML
kubectl apply -f chatty.yaml

Task L1. Show the last 5 lines of the app container of all chatty Pods, with the Pod name in front of each line.

kubectl logs -l app=chatty -c app --tail=5 --prefix

Task L2. Follow the sidecar of one Pod via the Deployment, with timestamps, only the last minute.

kubectl logs -f deploy/chatty -c sidecar --timestamps --since=1m

Task L3. Where do these logs physically live? Find the directory on the node.

kubectl get pods -l app=chatty -o wide              # pick the node
ssh student@worker01 'sudo ls /var/log/pods | grep chatty'

Task L4. List only Warning events in the whole cluster, then all events of one chatty Pod.

kubectl events -A --types=Warning
kubectl events --for pod/$(kubectl get pods -l app=chatty -o name | head -1 | cut -d/ -f2)

kubectl delete -f chatty.yaml

For each case below: create the mess with the command given, then answer the question before you open the box. The routine is always get -> describe -> logs.

Case 1 - it will not pull

kubectl run broken1 --image=nginx:doesnotexist
kubectl get pod broken1

Question. What state is it in, and where exactly does the cluster tell you why?

kubectl describe pod broken1 | tail -15

kubectl delete pod broken1

Case 2 - it keeps dying

kubectl run broken2 --image=busybox:1.36 --restart=Always -- sh -c 'echo starting; exit 1'
sleep 30
kubectl get pod broken2

Question. The RESTARTS counter is climbing. Get the output of the attempt that already failed, not the one running now.

kubectl logs broken2 --previous
kubectl describe pod broken2 | grep -A6 'Last State'

kubectl delete pod broken2

Case 3 - it never gets placed

kubectl run broken3 --image=nginx:1.27 \
  --overrides='{"spec":{"containers":[{"name":"broken3","image":"nginx:1.27","resources":{"requests":{"memory":"900Gi"}}}]}}'
kubectl get pod broken3

Question. It stays Pending forever. Which component is complaining, and what is the exact reason?

kubectl describe pod broken3 | tail -10
kubectl get events --sort-by=.lastTimestamp | tail -5

Follow-up. Prove it is about requests and not real memory use: how much memory is actually free on worker01?

kubectl top nodes
kubectl describe node worker01 | grep -A6 'Allocated resources'

kubectl delete pod broken3

Case 4 - it runs but answers nothing

kubectl run web --image=nginx:1.27
kubectl expose pod web --port 80 --name web-svc
kubectl label pod web run-                     # remove the label the Service selects

Question. A client gets a timeout. The Pod is Running and healthy. Find the break without looking at any logs.

kubectl get endpointslices -l kubernetes.io/service-name=web-svc
kubectl describe svc web-svc | grep -i selector
kubectl get pod web --show-labels

Fix it, then confirm the endpoint comes back.

kubectl label pod web run=web
kubectl get endpointslices -l kubernetes.io/service-name=web-svc
kubectl run tmp --image=busybox:1.36 --restart=Never -it --rm -- \
  wget -qO- --timeout=3 web-svc | head -3

kubectl delete pod web; kubectl delete svc web-svc

Case 5 - it will not even start

kubectl create deploy broken5 --image nginx:1.27
kubectl patch deploy broken5 -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","envFrom":[{"configMapRef":{"name":"app-settings"}}]}]}}}}'
kubectl get pods -l app=broken5

Question. What state, which event, and what is the smallest fix?

kubectl describe pod -l app=broken5 | tail -5
kubectl create cm app-settings --from-literal=MODE=prod
kubectl get pods -l app=broken5 -w

kubectl delete deploy broken5; kubectl delete cm app-settings

The routine, one more time

kubectl get pod <name> -o wide                 state, node
kubectl describe pod <name>                    Events at the bottom - read first
kubectl logs <name> [--previous] [-c ctr]      what the app said
kubectl get events --sort-by=.lastTimestamp    the wider picture
kubectl exec -it <name> -- sh                  only when it is running