Day 2 on Eriks training corner

Troubleshooting routine

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting routine

One order of operations, every time. It is short because it works.

graph TB A["kubectl get pod -o wide what state? which node?"] --> B["kubectl describe pod read the Events at the bottom"] B --> C{"did the container start?"} C -->|no| D["fix from the event: image, scheduling, volume, probe"] C -->|yes| E["kubectl logs [--previous] what did the app say?"] E --> F["kubectl exec -it -- sh only if it is running"]

The states and what they mean

State	Where the answer is
`Pending`	`describe` -> FailedScheduling: no capacity, a taint, an unbound PVC
`ContainerCreating`	`describe` -> image pull or volume mount in progress, or stuck
`ImagePullBackOff` / `ErrImagePull`	wrong image name or tag, or registry credentials
`CrashLoopBackOff`	`logs --previous` - the app started and exited
`Running` but not `Ready`	the readiness probe is failing; the Service will not send traffic
`OOMKilled`	`describe` -> Last State; memory limit too low
`Terminating` forever	a finalizer, or a container ignoring SIGTERM

Wider than one Pod

kubectl get events --sort-by=.lastTimestamp | tail -20
kubectl get events -A --field-selector type=Warning
kubectl get pods -A -o wide | grep -v Running
kubectl top nodes ; kubectl top pods

When the application answers nothing

Work outwards from the Pod: is the Pod ready, does the Service have endpoints, does DNS resolve, does the Gateway route.

Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

Namespaces

A virtual cluster inside the cluster: a name scope for objects, and a place to attach quotas and policies.

graph TB subgraph cluster subgraph ns1 [namespace: dev] P1[pod web] --- P2[pod db] end subgraph ns2 [namespace: prod] P3[pod web] --- P4[pod db] end N[nodes, PVs - not namespaced] end

kubectl get ns
kubectl create ns dev
kubectl get pods -n dev
kubectl get pods -A
kubectl config set-context --current --namespace=dev
kubectl api-resources --namespaced=false # what lives outside namespaces

Namespaced: Pods, Deployments, Services, ConfigMaps, Secrets, PVCs, HTTPRoutes. Not namespaced: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CRDs.

ReplicaSets

Mon, 01 Jan 0001 00:00:00 +0000

ReplicaSets

A ReplicaSet keeps N copies of a Pod running. That is the whole job.

graph LR RS[ReplicaSet replicas=3] --> P1[Pod] RS --> P2[Pod] RS --> P3[Pod] P2 -.node dies.-> X((gone)) RS -->|reconcile| P4[new Pod]

The loop never stops: count Pods matching the selector, compare with spec.replicas, create or delete the difference. Self-healing and scaling are the same mechanism.

spec:
 replicas: 3
 selector:
 matchLabels:
 app: web
 template:
 metadata:
 labels:
 app: web  # must match the selector

Pods are bound by the selector, not by name: a loose Pod carrying a matching label gets adopted and counted.

Deployments

Mon, 01 Jan 0001 00:00:00 +0000

Deployments

A Deployment owns ReplicaSets and each ReplicaSet owns Pods. That extra level is what makes zero-downtime updates and rollbacks possible.

graph TB D[Deployment nginx] --> RS1[ReplicaSet v1 - replicas 0] D --> RS2[ReplicaSet v2 - replicas 3] RS2 --> P1[Pod] RS2 --> P2[Pod] RS2 --> P3[Pod]

Changing the Pod template creates a new ReplicaSet and scales the old one down in steps. The old ReplicaSet stays at 0 replicas - that is your rollback.

DaemonSets

Mon, 01 Jan 0001 00:00:00 +0000

DaemonSets

One Pod per node - automatically, including nodes that join later.

graph TB DS[DaemonSet log-agent] --> P1[Pod on control01] DS --> P2[Pod on worker01] DS --> P3[Pod on worker02] DS --> P4[Pod on worker03] N[new node joins] -.-> P5[Pod created automatically]

There is no replicas field. The node list is the replica count.

Typical users: CNI agents, kube-proxy, log shippers, node exporters, storage drivers. Your cluster is already running several - kubectl get ds -A.

Scheduling: taints, tolerations, affinity

Mon, 01 Jan 0001 00:00:00 +0000

Scheduling: taints, tolerations, affinity

By default the scheduler puts a Pod wherever it fits. Three mechanisms let you change that, and they work in opposite directions.

Mechanism	Lives on	Says
Taint	the node	“keep Pods away unless they tolerate this”
Toleration	the Pod	“I can put up with that taint”
nodeSelector / node affinity	the Pod	“I want to run on a node like this”

The pair is asymmetric on purpose. A toleration permits but does not attract: a Pod that tolerates a taint may still land anywhere else. If you want it on those nodes, you need affinity as well.

Services

Mon, 01 Jan 0001 00:00:00 +0000

Services

Pods come and go and their IPs change. A Service is the stable address in front of them, with load balancing built in.

graph LR C[Client] --> S[Service nginx-svc ClusterIP 10.96.x.x] S --> P1[Pod 10.244.1.5] S --> P2[Pod 10.244.2.7] S --> P3[Pod 10.244.3.9]

The Service selects Pods by label. Matching, ready Pod IPs land in an EndpointSlice, and kube-proxy programs each node accordingly.

Types

Type	Reachable from	Note
`ClusterIP`	inside the cluster only	the default
`NodePort`	`<any-node-ip>:30000-32767`	opens the port on every node
`LoadBalancer`	outside	needs a cloud provider or MetalLB - not in this cluster
`ExternalName`	-	just a DNS CNAME

spec:
 type: NodePort
 selector:
 app: nginx  # must match the Pod labels
 ports:
 - port: 8080 # the Service port
 targetPort: 80 # the container port
 nodePort: 30080 # the port on every node

DNS

nginx-svc # same namespace
nginx-svc.dev # other namespace
nginx-svc.dev.svc.cluster.local # fully qualified

Headless Services

clusterIP: None gives no virtual IP; DNS returns the Pod IPs directly. This is how StatefulSets give each replica a stable name - see StatefulSets.

Gateway API

Mon, 01 Jan 0001 00:00:00 +0000

Gateway API

A NodePort per application does not scale: you end up remembering that the shop is on 30080 and the wiki on 30081. You want one HTTP entry point that routes by host name and path.

Kubernetes has had two answers to that. Ingress is the old one - stable, but frozen: every controller extended it with its own annotations, so a rewrite rule for nginx means nothing to Traefik. Gateway API is the replacement: the same job expressed as real, typed resources, and it is where all the development now happens.

Nodes

Mon, 01 Jan 0001 00:00:00 +0000

Nodes

A node is a machine that runs Pods. Cluster-scoped - not in a namespace.

Task	Command
List	`kubectl get nodes -o wide`
Detail, conditions, taints, capacity	`kubectl describe node worker01`
Stop scheduling new Pods	`kubectl cordon worker01`
Cordon and evict	`kubectl drain worker01 --ignore-daemonsets --delete-emptydir-data`
Allow scheduling again	`kubectl uncordon worker01`
Remove from the cluster	`kubectl delete node worker01`

stateDiagram-v2 [*] --> Ready Ready --> SchedulingDisabled: cordon SchedulingDisabled --> Ready: uncordon SchedulingDisabled --> Drained: drain Drained --> Ready: uncordon

Drain is the maintenance command. What it will not move: