Troubleshooting and kubectl debug
Troubleshooting and kubectl debug
The routine
graph TB
A["kubectl get pod -o wide<br/>state? node?"] --> B["kubectl describe pod<br/>Events at the bottom"]
B --> C{"container started?"}
C -->|no| D["fix from the event:<br/>image, scheduling, config, volume"]
C -->|yes| E["kubectl logs [--previous]"]
E --> F["exec / debug<br/>look inside"]
| State | Look at |
|---|---|
Pending | events: FailedScheduling - capacity, taint, affinity, PVC |
ImagePullBackOff | events: image name, tag, registry credentials |
CreateContainerConfigError | events: missing ConfigMap / Secret / key |
CrashLoopBackOff | logs --previous, describe -> Last State, exit code |
OOMKilled (exit 137) | memory limit too low, or a leak |
Running, not Ready | readiness probe - no traffic from Services |
| no response via Service | EndpointSlice empty -> selector or readiness |
Terminating forever | finalizer, or a node that is gone |
Work outwards: Pod ready? -> Service has endpoints? -> DNS resolves? -> Gateway route accepted?
Ephemeral containers and kubectl debug
Modern images are minimal or distroless - no shell, no ps, no curl.
kubectl exec has nothing to run. kubectl debug brings the tools with it.
graph LR
subgraph Pod
A[app container<br/>distroless]
D[ephemeral debug container<br/>busybox / netshoot]
end
D -. "--target: same process namespace" .-> A
# add a debug container to the running Pod
kubectl debug -it web --image=busybox:1.36 --target=web
# copy the Pod and change it (the original keeps running)
kubectl debug web -it --copy-to=web-dbg --container=web -- sh
# a Pod on the node with the host filesystem at /host
kubectl debug node/worker01 -it --image=busybox:1.36
| Mode | Use when |
|---|---|
ephemeral container (--target) | Pod runs, image has no tools |
--copy-to (+ --set-image, new command) | Pod crashes at start - start it with a shell instead |
node/<name> | node problem: kubelet logs, disk, network |
--profile (general, netadmin, sysadmin, restricted) grants the
debug container the capabilities it needs, e.g. netadmin for tcpdump.
Ephemeral containers cannot be removed - they end when their process ends and disappear with the Pod.
Other helpers
kubectl top pods --sort-by=memory
kubectl port-forward svc/web 8080:80
kubectl auth can-i create deployments --as=system:serviceaccount:dev:app