Lab 21 - Scheduling and node maintenance

Goal: reserve a node for a workload, require specific hardware, and take a node out for maintenance without downtime. Chapter: Scheduling and node operations

Tolerations permit

# 1 - reserve worker03 for "database" workloads
kubectl taint node worker03 workload=db:NoSchedule

cd ~
cat > tolerant.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: tolerant
spec:
  tolerations:
    - key: workload
      operator: Equal
      value: db
      effect: NoSchedule
  containers:
    - name: box
      image: busybox:1.36
      command: ["sleep", "3600"]
YAML
kubectl apply -f tolerant.yaml
kubectl get pod tolerant -o wide

Look at which node it picked. It probably is not worker03. The toleration said “I can live with that taint”, not “put me there” - the scheduler was free to choose any node and the empty one was not necessarily the best fit.

This is the single most misunderstood thing about taints. Repel and attract are separate decisions.

# 2 - now attract it as well
kubectl label node worker03 workload=db
kubectl delete -f tolerant.yaml
cat > pinned.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: pinned
spec:
  tolerations:
    - key: workload
      operator: Equal
      value: db
      effect: NoSchedule
  nodeSelector:
    workload: db
  containers:
    - name: box
      image: busybox:1.36
      command: ["sleep", "3600"]
YAML
kubectl apply -f pinned.yaml
kubectl get pod pinned -o wide          # worker03, guaranteed

Taint + toleration + selector is the complete “reserve these nodes” recipe: the taint keeps everyone else out, the toleration lets this Pod in, the selector puts it there.

Node affinity

# 3 - label two nodes and require one of them
kubectl label node worker01 disktype=ssd
kubectl label node worker02 disktype=hdd
kubectl get nodes -L disktype

cat > affinity.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: needs-ssd
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]
  containers:
    - name: box
      image: busybox:1.36
      command: ["sleep", "3600"]
YAML
kubectl apply -f affinity.yaml
kubectl get pod needs-ssd -o wide       # worker01

# 4 - what happens when nothing matches?
kubectl label node worker01 disktype=hdd --overwrite
kubectl delete pod needs-ssd
kubectl apply -f affinity.yaml
kubectl get pod needs-ssd               # Pending
kubectl describe pod needs-ssd | tail -6

didn't match Pod's node affinity/selector. A required rule that matches nothing means the Pod never runs - it does not fall back.

Node maintenance

# 5 - tidy up the scheduling experiments first
kubectl delete pod pinned needs-ssd --ignore-not-found
kubectl taint node worker03 workload=db:NoSchedule-
kubectl label node worker03 workload-
kubectl label node worker01 disktype- ; kubectl label node worker02 disktype-

# 6 - something to move around, protected by a PodDisruptionBudget
kubectl create deploy filler --image nginx:1.27 --replicas 6
kubectl create pdb filler --selector app=filler --min-available 5
kubectl get pods -l app=filler -o wide

# 7 - cordon, drain
kubectl cordon worker02
kubectl get nodes                                    # SchedulingDisabled
kubectl drain worker02 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -l app=filler -o wide               # none on worker02

Watch the drain output: with minAvailable: 5, drain evicts one Pod at a time and waits until its replacement is ready. The application never drops below 5.

# 8 - back in service - and rebalance
kubectl uncordon worker02
kubectl get pods -l app=filler -o wide               # still nothing on worker02
kubectl rollout restart deploy/filler
kubectl get pods -l app=filler -o wide

uncordon does not move Pods back. Kubernetes never moves a running Pod - replacing them does.

Discovery

Infodiscover

Task A. Spread the filler replicas evenly across the workers (max difference 1).

Show one answer

kubectl patch deploy filler -p '{"spec":{"template":{"spec":{"topologySpreadConstraints":[{"maxSkew":1,"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"DoNotSchedule","labelSelector":{"matchLabels":{"app":"filler"}}}]}}}}'
kubectl get pods -l app=filler -o wide | awk '{print $7}' | sort | uniq -c

Task B. Set the PDB to minAvailable: 6 and drain worker03. What happens?

Show one answer

kubectl patch pdb filler -p '{"spec":{"minAvailable":6}}'
kubectl drain worker03 --ignore-daemonsets --timeout=30s
kubectl uncordon worker03

Cannot evict pod as it would violate the pod's disruption budget - drain retries until timeout. A PDB that allows zero disruptions blocks every maintenance.

Task C. Set the PDB back to minAvailable: 5. Then put a bare Pod on worker03 - what does drain do?

Show one answer

kubectl patch pdb filler -p '{"spec":{"minAvailable":5}}'
kubectl run orphan --image nginx:1.27 --overrides='{"spec":{"nodeName":"worker03"}}'
kubectl drain worker03 --ignore-daemonsets        # refuses
kubectl drain worker03 --ignore-daemonsets --force --timeout=60s
kubectl get pod orphan                            # gone, not rescheduled
kubectl uncordon worker03

Nothing recreates a bare Pod - --force deletes it for good.

Clean up

kubectl delete deploy filler; kubectl delete pdb filler
kubectl delete pod orphan --ignore-not-found
kubectl get nodes          # all Ready, none SchedulingDisabled