Logs and events

Logs and events

Container logs

Applications write to stdout/stderr. The runtime stores it on the node in /var/log/pods/<ns>_<pod>_<uid>/<container>/, the kubelet rotates it (default 10 MiB x 5 files).

kubectl logs web                        # one Pod
kubectl logs web -c sidecar             # one container
kubectl logs web --previous             # the crashed instance before this one
kubectl logs -f deploy/web              # follow, via the controller
kubectl logs -l app=web --prefix --tail=20   # many Pods, name in front
kubectl logs web --since=10m --timestamps
kubectl logs web --all-containers

Logs disappear with the Pod. For history you need a logging stack:

graph LR P[Pods stdout] --> F["/var/log/pods on node"] F --> A["agent DaemonSet<br/>Fluent Bit / Promtail / Vector"] A --> B[(Loki / Elasticsearch / cloud)] B --> G[Grafana / Kibana]

Events

Events are API objects written by controllers, the scheduler and the kubelet: what happened to an object and why. They expire after 1 hour.

kubectl describe pod web                          # events at the bottom
kubectl events --for pod/web
kubectl events -A --types=Warning
kubectl events --watch
kubectl get events --sort-by=.lastTimestamp | tail
Event reasonTypical cause
FailedSchedulingno capacity, taint, affinity, unbound PVC
Failed / BackOff (pull)wrong image or registry credentials
BackOff (restart)container keeps crashing
Unhealthya probe failed
FailedCreate (on ReplicaSet)quota exceeded, admission rejected
FailedMountSecret / ConfigMap / volume missing
Killingliveness failure, eviction, deletion

Rule: describe / events first, logs second. A container that never started has no logs.

References