Lab 24 - Ephemeral containers and kubectl debug

Lab 24 - Ephemeral containers and kubectl debug

Goal: debug a Pod that has no shell, a Pod that crashes at start, and a node. Chapter: Troubleshooting and kubectl debug

A Pod without tools

# 1 - a minimal image: no shell, nothing
kubectl run minimal --image registry.k8s.io/pause:3.10
kubectl exec -it minimal -- sh              # fails: no sh in the image

# 2 - bring your own tools as an ephemeral container
kubectl debug -it minimal --image busybox:1.36 --target minimal

Inside the debug container:

ps aux                  # you see /pause - the app's process namespace is shared
ls /proc/1/root/        # the app container's filesystem, through /proc
exit

# 3 - it is recorded in the Pod
kubectl get pod minimal -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
kubectl describe pod minimal | grep -A3 'Ephemeral Containers'

Ephemeral containers cannot be removed or restarted - they die with the Pod.

A Pod that crashes at start

# 4 - crashes immediately, so exec and debug --target have nothing to attach to
kubectl run crasher --image busybox:1.36 -- sh -c 'cat /config/app.conf && sleep 3600'
kubectl get pod crasher                     # CrashLoopBackOff
kubectl logs crasher --previous

# 5 - start a copy with a shell instead of the failing command
kubectl debug crasher -it --copy-to crasher-dbg --container crasher -- sh

Inside the copy: ls /config - it does not exist. The app expected a mounted config file. Exit, then:

kubectl delete pod crasher crasher-dbg

The original stays untouched; --copy-to is for experiments. Other useful flags: --set-image=crasher=busybox:1.37, --share-processes.

A node

# 6 - a privileged Pod on worker01 with the host filesystem at /host
kubectl debug node/worker01 -it --image busybox:1.36

Inside:

chroot /host
systemctl status kubelet --no-pager | head -5
ls /var/log/pods | head
crictl ps | head -5
exit; exit

# 7 - the node debugger leaves a Pod behind
kubectl get pods | grep node-debugger
kubectl delete pod $(kubectl get pods -o name | grep node-debugger | cut -d/ -f2)

Discovery

Infodiscover

Task A. From an ephemeral container attached to a nginx:1.27 Pod, list the listening TCP ports of nginx.

kubectl run web --image nginx:1.27
kubectl debug -it web --image busybox:1.36 --target web -- netstat -tlnp
kubectl delete pod web

The debug container shares the Pod network (always) and, with --target, the process namespace.

Task B. Which profile do you need to run tcpdump from a debug container, and why?

--profile=netadmin - it adds the NET_ADMIN and NET_RAW capabilities. Use an image that contains tcpdump, e.g. nicolaka/netshoot:

kubectl debug -it <pod> --image nicolaka/netshoot --profile netadmin --target <ctr> -- tcpdump -i eth0 -c 5

Clean up

kubectl delete pod minimal --ignore-not-found