Lab 08 - Pod troubleshooting

Lab 08 - Pod troubleshooting

Infodiscovery

Goal: diagnose four broken Pods from cluster output alone. Chapter: Troubleshooting routine

For each case: create the mess with the command given, then answer the question before you open the box. The routine is always get -> describe -> logs.

Case 1 - it will not pull

kubectl run broken1 --image=nginx:doesnotexist
kubectl get pod broken1

Question. What state is it in, and where exactly does the cluster tell you why?

kubectl describe pod broken1 | tail -15

ErrImagePull, then ImagePullBackOff as the kubelet backs off. The Events block says Failed to pull image ... manifest unknown. There are no logs - the container never started, so kubectl logs has nothing to show.

kubectl delete pod broken1

Case 2 - it keeps dying

kubectl run broken2 --image=busybox --restart=Always -- sh -c 'echo starting; exit 1'
sleep 30
kubectl get pod broken2

Question. The RESTARTS counter is climbing. Get the output of the attempt that already failed, not the one running now.

kubectl logs broken2 --previous
kubectl describe pod broken2 | grep -A6 'Last State'

--previous reads the log of the previous container instance. Last State: Terminated, Exit Code: 1 confirms the application chose to exit - this is an application bug, not an infrastructure problem. The BackOff delay grows to a maximum of five minutes.

kubectl delete pod broken2

Case 3 - it never gets placed

kubectl run broken3 --image=nginx:1.27 \
  --overrides='{"spec":{"containers":[{"name":"broken3","image":"nginx:1.27","resources":{"requests":{"memory":"900Gi"}}}]}}'
kubectl get pod broken3

Question. It stays Pending forever. Which component is complaining, and what is the exact reason?

kubectl describe pod broken3 | tail -10
kubectl get events --sort-by=.lastTimestamp | tail -5

FailedScheduling from the scheduler: 0/4 nodes are available: 4 Insufficient memory. Pending always means the scheduler could not place it - capacity, a taint, or an unbound volume.

Follow-up. Prove it is about requests and not real memory use: how much memory is actually free on worker01?

kubectl top nodes
kubectl describe node worker01 | grep -A6 'Allocated resources'

The node is nearly idle. The scheduler does its arithmetic on requests only.

kubectl delete pod broken3

Case 4 - it runs but answers nothing

kubectl run web --image=nginx:1.27
kubectl expose pod web --port 80 --name web-svc
kubectl label pod web run-                     # remove the label the Service selects

Question. A client gets a timeout. The Pod is Running and healthy. Find the break without looking at any logs.

kubectl get endpointslices -l kubernetes.io/service-name=web-svc
kubectl describe svc web-svc | grep -i selector
kubectl get pod web --show-labels

The EndpointSlice has no addresses: the Service selects run=web, and the Pod no longer carries that label. No endpoints = no traffic, and nothing anywhere logs an error. This is the most common Service failure there is.

Fix it, then confirm the endpoint comes back.

kubectl label pod web run=web
kubectl get endpointslices -l kubernetes.io/service-name=web-svc
kubectl run tmp --image=busybox --restart=Never -it --rm -- \
  wget -qO- --timeout=3 web-svc | head -3

kubectl delete pod web; kubectl delete svc web-svc

The routine, one more time

kubectl get pod <name> -o wide                 state, node
kubectl describe pod <name>                    Events at the bottom - read first
kubectl logs <name> [--previous] [-c ctr]      what the app said
kubectl get events --sort-by=.lastTimestamp    the wider picture
kubectl exec -it <name> -- sh                  only when it is running