Day 3 on Eriks training corner

Rolling updates and rollbacks

Mon, 01 Jan 0001 00:00:00 +0000

Rolling updates and rollbacks

Deployment lifecycle

stateDiagram-v2 [*] --> Progressing: new template (image, env, ...) Progressing --> Complete: all new Pods available Progressing --> Failed: progressDeadlineSeconds exceeded Complete --> Progressing: next change Failed --> Progressing: fix or rollout undo

Only a change of the Pod template starts a rollout. replicas does not.

kubectl rollout status deploy/web # waits; exit code != 0 on failure
kubectl get deploy web -o jsonpath='{.status.conditions[*].type}'

Condition	Meaning
`Progressing=True`	rollout running or finished
`Progressing=False` reason `ProgressDeadlineExceeded`	stuck longer than `progressDeadlineSeconds` (600 s)
`Available=True`	at least `replicas - maxUnavailable` Pods ready

Strategies

graph LR subgraph RollingUpdate A1[v1 v1 v1] --> A2[v1 v1 v2] --> A3[v1 v2 v2] --> A4[v2 v2 v2] end subgraph Recreate B1[v1 v1 v1] --> B2[- - -] --> B3[v2 v2 v2] end

spec:
 minReadySeconds: 5 # Pod must stay ready this long to count
 revisionHistoryLimit: 5 # old ReplicaSets kept for rollback
 strategy:
 type: RollingUpdate  # or Recreate (downtime, never two versions)
 rollingUpdate:
 maxSurge: 1 # extra Pods allowed during the rollout
 maxUnavailable: 0 # Pods allowed to be missing

Setting	Effect
`maxUnavailable: 0, maxSurge: 1`	zero downtime, needs room for one more Pod
`maxUnavailable: 1, maxSurge: 0`	no extra capacity, one Pod less during rollout
`Recreate`	all old Pods go first - for apps that cannot run two versions

A broken image stalls a rolling update - the old Pods keep serving.

Requests, limits and quotas

Mon, 01 Jan 0001 00:00:00 +0000

Requests, limits and quotas

resources:
 requests: { cpu: "250m", memory: "128Mi" } # guaranteed, used for scheduling
 limits: { cpu: "500m", memory: "256Mi" } # hard ceiling, enforced by cgroups

Units: 1 CPU = 1 core = 1000m. Memory Mi/Gi (binary), M/G (decimal).

	Request	Limit
Used by	scheduler (node must have it free)	kubelet / kernel at runtime
CPU over it	fine, if the node has spare	throttled - slower, not killed
Memory over it	fine, until the node is short	OOMKilled - container restarts

graph LR N["Node allocatable 4 CPU"] --> A["Pod A request 1"] N --> B["Pod B request 2"] N --> F["free for scheduling: 1"]

The scheduler counts requests, not real usage.

Autoscaling with HPA

Mon, 01 Jan 0001 00:00:00 +0000

Autoscaling with HPA

The HorizontalPodAutoscaler changes replicas of a Deployment (or StatefulSet) based on metrics.

graph LR K[kubelets] --> MS[metrics-server] MS -->|metrics.k8s.io| HPA[HPA controller every 15 s] HPA -->|scale| D[Deployment] D --> P[Pods]

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: web
spec:
 scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
 minReplicas: 1
 maxReplicas: 6
 metrics:
 - type: Resource
 resource:
 name: cpu
 target: { type: Utilization, averageUtilization: 50 } # % of the REQUEST
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 300 # default: wait 5 min before shrinking

desired = ceil(current replicas x current usage / target) Example: 2 Pods at 150 % CPU, target 50 % -> ceil(2 x 150 / 50) = 6.

Scheduling and node operations

Mon, 01 Jan 0001 00:00:00 +0000

Scheduling and node operations

How the scheduler decides

graph LR P[new Pod] --> F["Filter requests fit? taints tolerated? selector / affinity match?"] F --> S["Score spread, preferred affinity, least allocated"] S --> B[bind to best node] F -->|no node left| PE[Pending + FailedScheduling event]

The tools

Mechanism	On	Says
`nodeSelector`	Pod	run only on nodes with these labels
node affinity `required...`	Pod	same, with operators (`In`, `NotIn`, `Exists`)
node affinity `preferred...`	Pod	try these nodes, else anywhere
pod anti-affinity	Pod	not next to Pods like these
`topologySpreadConstraints`	Pod	spread evenly over nodes / zones
taint	Node	keep Pods away…
toleration	Pod	…unless they tolerate it

A toleration permits, it does not attract. To reserve nodes for a workload: taint the nodes + toleration + selector on the Pod.

Startup, readiness and liveness probes

Mon, 01 Jan 0001 00:00:00 +0000

Startup, readiness and liveness probes

Without probes Kubernetes knows only “the process runs”. Probes tell it whether the application works.

Probe	Question	On failure
startupProbe	has it finished booting?	the other probes wait; after `failureThreshold` -> restart
readinessProbe	can it take traffic now?	removed from Service endpoints - no restart
livenessProbe	is it hung?	container restarted

graph LR S[container starts] --> SP{startup OK?} SP -->|no, too long| R[restart] SP -->|yes| L[liveness + readiness run] L -->|readiness fails| NR[not Ready - no traffic] L -->|liveness fails| R

containers:
 - name: app
 image: nginx:1.27
 startupProbe:
 httpGet: { path: /, port: 80 }
 failureThreshold: 30 # 30 x 2 s = up to 60 s to start
 periodSeconds: 2
 readinessProbe:
 httpGet: { path: /ready, port: 80 }
 periodSeconds: 5
 livenessProbe:
 httpGet: { path: /healthz, port: 80 }
 periodSeconds: 10
 failureThreshold: 3

Handlers: httpGet (2xx/3xx = OK), tcpSocket, exec (exit 0 = OK), grpc. Tuning: initialDelaySeconds, periodSeconds, timeoutSeconds (1 s!), failureThreshold, successThreshold.

Logs and events

Mon, 01 Jan 0001 00:00:00 +0000

Logs and events

Container logs

Applications write to stdout/stderr. The runtime stores it on the node in /var/log/pods/<ns>_<pod>_<uid>/<container>/, the kubelet rotates it (default 10 MiB x 5 files).

kubectl logs web # one Pod
kubectl logs web -c sidecar # one container
kubectl logs web --previous # the crashed instance before this one
kubectl logs -f deploy/web # follow, via the controller
kubectl logs -l app=web --prefix --tail=20 # many Pods, name in front
kubectl logs web --since=10m --timestamps
kubectl logs web --all-containers

Logs disappear with the Pod. For history you need a logging stack:

Troubleshooting and kubectl debug

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting and kubectl debug

The routine

graph TB A["kubectl get pod -o wide state? node?"] --> B["kubectl describe pod Events at the bottom"] B --> C{"container started?"} C -->|no| D["fix from the event: image, scheduling, config, volume"] C -->|yes| E["kubectl logs [--previous]"] E --> F["exec / debug look inside"]

State	Look at
`Pending`	events: `FailedScheduling` - capacity, taint, affinity, PVC
`ImagePullBackOff`	events: image name, tag, registry credentials
`CreateContainerConfigError`	events: missing ConfigMap / Secret / key
`CrashLoopBackOff`	`logs --previous`, `describe` -> Last State, exit code
`OOMKilled` (exit 137)	memory limit too low, or a leak
`Running`, not Ready	readiness probe - no traffic from Services
no response via Service	EndpointSlice empty -> selector or readiness
`Terminating` forever	finalizer, or a node that is gone

Work outwards: Pod ready? -> Service has endpoints? -> DNS resolves? -> Gateway route accepted?